Quality of Experience – how four colors make sense of terabytes of metrics

What is QoE?

In its most general sense, QoE is a measure of a customer's satisfaction with a service as a whole. This is also closely related to the Net Promoter Score (NPS) concept – how likely a customer is to recommend the service to others.

There are many aspects of a video service that can affect overall customer satisfaction. These include available content, price, and customer service. The Agama solution helps operators excel in these areas by measuring engagement, streamlining operations and providing insights to customer-facing teams.

But when it comes to measuring QoE, the Agama solution is focused on another key driver of customer satisfaction: technical quality. With this focus, QoE is a holistic measure of how well services are encoded, delivered and played back.

How is QoE different from QoS or MOS?

Quality of Service (QoS) is not a single metric but rather a class of metrics which measure the technical performance of a service. QoS measurements usually differ between technology domains. Examples of QoS measurements in an IPTV network are packet loss and jitter. For a streaming service, examples include buffering ratio and Exits Before Video Start. An experienced engineer may know what level of jitter or buffering ratio is problematic in a given ecosystem, but QoS metrics don't directly convey how the end user perceives the service.

Mean Opinion Score (MOS) does take the end user's experience into account, but unlike QoE, it typically focuses on a single aspect of a service, such as video encoding quality. As such, it may not only be domain specific, but can even be codec specific.

Video MOS can help understand, tune and monitor the output of encoders, but it is only one piece of a much larger puzzle. The Agama solution measures thousands of QoS metrics – and MOS happens to be one of them. Together, these thousands of metrics are the basis for the Agama QoE, which brings us to our next point.

How is QoE calculated in the Agama solution?

Some vendors of video analysis probes use the term QoE to refer to MOS or other video encoding assessments, typically focused on the video head-end or playout. The Agama solution takes a more holistic approach.

In the Agama solution, the overall quality of every stream and every viewing session is continuously assessed. Simply put, we calculate the combined impact of all detected errors and collected measurements. The quality is then ranked in a color-coded scale from green (perfect quality) to red (unavailable). This simple yet powerful four-color scale is the Agama QoE.

Aggregated top-level QoE charts in the Agama solution

As we've seen, the QoS metrics that go into the QoE calculation differ between, for example, a head-end Analyzer probe and a streaming app with Client Device Monitoring. But the approach is similar for all technologies, and the resulting QoE classification means the same across all of them.

What is it good for?

Even within a single technology domain, the Agama QoE creates situational awareness without having to focus on specific QoS metrics, which saves precious time when incidents occur. This is true even if you're a domain expert who does understand the underlying QoS metrics – which you can always drill down to in order to understand the root cause.

But a typical video service covers multiple technology domains. Streams may be received in the head-end over SDI, transcoded and delivered over multicast or cable to set-top boxes. Or transrated, segmented and delivered over a CDN to streaming apps. A consistent, unified top-level quality classification is crucial to understanding if one technology domain is performing worse than others.

This is even more important when multiple teams are involved. The Agama QoE helps the whole organization speak the same language when it comes to quality. Some Agama customers take this so far as to use QoE to set SLA levels, internal benchmarks and continuous improvement goals.

                                                                                       ***