Impact of 5G on Customer Experience [Part2]

Measuring the 5G QoE

My previous blog entry (here) looked at the major shifts happening in mobile networks from 5G technology and architectures, and how these changes are making the network more complex and challenging to the network operators and how they are impacting Quality of Experience {QoE}.

This blog entry will now look at different methods currently used for measuring Quality of Service (QoS) and Quality of Experience (QoE) for mobile subscribers, and how many of these methods will struggle to cope with the complexities of 5G networks.

There are several types of measurement solutions typically used to measure QoS and QoE for mobile networks. The most popular methods are:

  • Core Network Probes

  • RAN Probes

  • Network Equipment Stats

  • Drive Testing

  • Static Test Probes

  • Device-Based Measurement

Core Network Probes

Core network probes consist of additional hardware and software installed alongside Mobility Management Entities (MMEs), the Packet Data Network Gateways (PDN GWs), or other core network elements. The probes monitor the communications between network elements, decode the protocol messages and then calculate network and service performance measurements by various dimensions. Core probes are relatively expensive to buy, install and operate, especially when monitoring the performance of User Plane data (i.e. internet connection data), due to the huge growth rate (45% CAGR, according to Ericsson). Some network operators have stopped monitoring User plane data altogether, as the cost of User Plane probing often exceeds the cost of the associated network elements themselves (e.g. Serving Gateways and PDN Gateways).

With cost reductions and scalability driving the architectural shift to virtualised network elements and connectivity using Network Function Virtualisation (NFV) and Software-Defined networks (SDN), in one way it becomes easier to monitor network elements. Software-based virtual probes can be used instead of dedicated hardware probes, and these can be logically connected to the monitored elements through virtualised network connectivity. As long as a physical data centre infrastructure is already setup for NFV and SDN, then this infrastructure can also be used for virtual probes and their connectivity.

Aside from the usual benefits of virtualisation, probing actually becomes significantly more complex for networks deploying NFV and SDN. Network elements are no longer in a fixed known location, instead they are virtual software entities running in a VM or container somewhere in a telco datacentre. Probes can no longer connect to physical elements, and now must be instantiated as virtual software probes and controlled alongside (and in synchronisation with) the network elements they are monitoring. This requires common management interfaces to be supported by the software probes, and for them to comply with emerging standards for orchestration and management.

With NFV and SDN, probes are no longer physically and logically separate from the network to be monitored, they are actually now part of the network. The Network Orchestration processes that instantiate and manage virtual network elements and connectivity also now have to simultaneously instantiate and manage the associated virtual probes and connectivity. If the virtual probing configuration does not exactly match the network configuration then the service and network performance measurements calculated by the probing system will be partial and/or incorrect. If the virtual network configuration is incorrect in any way, then the virtual probing configuration will mirror the errors and will also be incorrect. Many types of problem that would have been detected with independent hardware probes and physical connectivity to the network links will now be missed when virtualised software probes and connectivity are used to monitor the core network.

Core network slicing (see this article for a more detailed explanation), where new instances of core networks are created dynamically on-demand, for example a 5G Ultra-Reliable Low Latency (URLL) core network, also greatly increases the complexity of core network monitoring.

5g network slicing.png

[5G Network Slices, source: NGMN Alliance 5G White Paper ]

New slices will appear and disappear dynamically as the set of supported services and associated scaling for demand varies over time. Corresponding virtual probes will need to be dynamically instantiated and managed through orchestration in order to monitor each slice’s virtual core elements.

In summary, while NFV and SDN enable more rapid and flexible probing, the overall complexity of the combined core network and probing solution has greatly increased. Probes are no longer independent of network configuration, and successful probing of virtualised core networks requires complex orchestration and management of virtual probes and connectivity across a dynamically changing network and set of services being measured.

Any QoS or QoE solution requiring core probing is now no longer independent from the network, and inherently less accurate and objective when compared with hardware-based core network probes.

RAN Probes

Like the modern core network, RAN networks are also becoming virtualised and software-defined. An example of the additional complexities this involves can be seen in the diagram below from Ericsson

cloud RAN control.png

Monitoring virtualised RAN architectures requires virtual software probes. The network flexibility enabled through cRAN, vRAN and xRAN to dynamically reconfigure RAN on demand means RAN network elements and configurations will change more frequently than non-virtualised RAN networks.

The same Orchestration systems that instantiate and manage the virtualised RAN network will now also have to instantiate and manage virtualised RAN probes, with the same network independence issue mentioned above for the core network probes. Many types of QoS or QoE problems will no longer be visible with virtualised probing. Virtual probe configurations may not match the RAN configuration exactly, or RAN configuration errors may be mirrored in the virtual probing configuration.

Network Equipment Stats

Extracting network or service performance counters from virtualised network elements is in one way easier than collecting from physical elements. Instead of connecting to each physical network element to extract the relevant data, data can be collected from all the virtual elements using instantiated virtualised data collectors.

However, because the 5G network is significantly more complex there are many more instances of network elements to collect data from. Dynamic network slicing requires data collectors to be dynamically instantiated and to adapt to new network element types and instances. Additionally, Load-balancing and dynamic routing of user-plane traffic means that stats collected from specific devices (e.g. gateway traffic routers) can no longer be assumed to be related to specific services. The relationships between specific network elements (physical or virtual) and the services they deliver are now no longer simple nor deterministic. Mobile Edge Computing (MEC) and Control/User Plane Separation (CUPS) also introduce many more network element types and instances to be monitored and many more sets of performance data to be collected.

In summary, an increasingly complex and dynamic network means network equipment stats collection is now significantly more complex and therefore significantly more expensive.

Drive Testing

Against a background of increasingly complex and dynamic Core and RAN networks, drive testing has a distinct advantage over network probing. The size, complexity and dynamic nature of Core and RAN doesn’t need to be reflected in the complexity of a drivetest solution measuring QoS or QoE. Drivetest probes generally measure from the device/subscriber’s perspective of a service, and thus treat the underlying network as simply a “black box” delivering the services.

However, drive testing covers only a very small percentage of each network, and requires expensive equipment and/or professional services to be used on a regular basis. So while it is isolated from 5G complexities, drive testing is not scalable to cover the whole network.

Static Test Probes

As with drivetest probes, static test probes also take the device/subscriber’s perspective when measuring QoS. Network complexity, scale and dynamic nature will not impact the complexity of a static test probe-based solution.

However, as with drivetest solutions, static test probes only cover a very small percentage of the network, and are also not scalable to cover the whole network.

Device-Based Measurement

From a device’s perspective, the network is simply a “black box” delivering services. Virtualisation, Slicing, MEC, and CUPS will determine exactly how the Core and RAN networks are configured to deliver services, but these network changes do not change how we measure the quality of the service at the end-device. Like drivetest and static test probes, the complexity of any device-based QoS/QoE solutions is not impacted by the complexity of the network used to deliver the measured services.

Key service measures such as latency, throughput and success rate are all likely to be affected by how the network is configured, but the algorithms used to calculate those performance measurements on the device do not change. No matter how complex or dynamic the underlying network is, the measurement methodology on the end device is the same.

QoS and QoE Measurement: The Ideal Solution

In an ideal world, a solution to measure the QoS and QoE for services delivered from a network should have the following attributes:

  • Excellent geographical coverage, in terms of the % network covered

  • Matches actual Subscriber Behaviour, where measurements are made while moving around and from locations visited by real subscribers

  • Delivers enough data for measurements to have statistical significance

  • Delivers a single set of homogeneous measurements, and not multiple mismatched sets of vendor-specific KPIs

  • Agnostic to the dynamic and complex configuration of the network being monitored

  • Measurements are done from actual Subscriber Devices, delivering very accurate QoE and QoS measurements

  • Provides good value, where enough QoS/QoE visibility is delivered at an acceptable price point

5G Solution Summary.png

Summary

Against the background of several key technological and architectural shifts in today’s networks, 5G consolidates many of these changes and adds additional complexity to the mix. Traditional probe-based QoS solutions are significantly impacted, and huge investment is already being made by probe vendors to develop virtualised probes, management interfaces and tools in order to try and maintain their monitoring capabilities. Network operators will have to spend heavily to replace legacy hardware-based probe solutions with virtualised software probes, and to integrate them with network orchestration systems.

Based on the various strengths and weaknesses of different types of QoS/QoE solution available for measuring 5G Networks, a Device-based solution has many advantages over the other QoS/QoE solutions, and offers better value to the network operator at a lower price point.

About the Ciqual On-Device QoE Solution

Ciqual Insight consists of a low-footprint SDK embedded into one or more smartphone applications. Deployed across a wide population of consumer devices, the solution measures QoS and QoE in the background, transparently to end users, and with minimal impact on battery life and data usage. Measurements are reported anonymously, ensuring end-user privacy is maintained. For more information, see https://www.ciqual.com/ciqual-client-and-sdk/.