A famous saying attributed to Peter Drucker goes, “You can’t improve what you can’t measure,” which is apt for the video streaming industry.
The video industry is very competitive and has multiple content providers fighting for a small cohort of users. In such situations, engineering and product teams must do everything possible to improve their products and ensure an excellent experience for the end user.
One area that has a huge impact on the user experience is video streaming. The quality and consistency of the streaming experience can make or break a product’s success.
In order to measure the performance of the video streaming ecosystem, companies need to gather information from several different points of the video delivery pipeline. They then need to gather all this data in a database and make sense of it.
Some companies choose to do this in-house, while others buy a solution from vendors who build such specialized products. In this article, we focus on the latter and explain some of the important criteria on judging a vendor and choosing a solution.
But, first, lets start off with an understanding of QoE and QoS.
What are QoE and QoS Metrics?
Generally, measurement is divided into QoE (Quality of Experience) and QoS (Quality of Service) metrics.
In layman’s terms,
- Quality of Service refers to the quality of the infrastructure used to deliver the experience to the end user. This involves the video streaming pipeline, including the CDN, caches, APIs, etc.
- Quality of Experience refers to the quality of the service as perceived by the end user. This includes app boot time, startup delays, buffering, crashes, etc.
To measure the QoE and QoS for a streaming service, you must have several touch-points and measurement sensors across the entire pipeline.
Everything for the user begins and ends at the video player, and the player’s performance determines a lot of the user’s perception of the app.
The time it takes for the video player to start playing a video, the time for the app to open, the amount of buffering a user sees, etc., significantly impact the user’s experience.
Let’s take a look at one of these metrics – Video Startup Time: this is the time a video player takes to display the first video frame after the user presses the play button. This is a critical metric that companies must track because it is the first touchpoint for a customer and can significantly influence the rest of the experience.
For example, a long startup time can indicate several things –
* the CDN might be cold, and it’s entirely possible that the CDN had to fetch the first segment of video from the origin and incurred a long round trip delay.
* it could indicate problems in the pipeline and servers being slow to respond.
* if an ad plays before the video, and if the ad does not load, that will further increase the startup delay, which can hurt the user experience quite a lot.
As you can see, there are multiple possible causes for every single event or error in a video pipeline and it is important to have a lot of data, the right dimensions, and the tools to slice & dice your data to arrive at the root cause.
What should you expect out of a QoE Analytics product
While choosing a QoE analytics vendor to integrate into your video pipeline, you need to consider a lot of important factors that can affect the integration, usage, and ultimately, the ability for you to catch and diagnose issues.
Metrics: the essential metrics that are needed for a video streaming company to deliver an extraordinary experience should be available out of the box. These include metrics such as start-up delay, buffering, re-buffering percentages, exit before video starts, errors (fatal and non-fatal), and dimensions/filters such as Country, City, CDN, ISP, Player, Player Version, etc. that allow you to deep-dive into the problems and figure out the root cause.
Support for Playback Platforms
Another important aspect is the support for video playback platforms where your service is deployed. To ensure a holistic collection of data to identify the hot spots for problems, we should ensure that the video analytics platform chosen supports the devices which we want to monitor.
Commonly supported platforms are Web (HTML5), Android, iOS, Roku, Chromecast, AppleTV, Android TV, FireTV, SmartTV (Samsung, LG, Panasonic) and also gaming consoles (XBox, etc.)
Data Retention Duration
You also need to consider the amount of time that the video analytics platform retains your data. This is critical if you want to go back in time to analyze problems that occurred and understand how your platform has changed or evolved over time.
Typically, platforms retain data for up to 13 months or 25 months that allow for a year-on-year comparison.
Filters and Dimensions
When you look at a data point such as latency, it does not mean anything in isolation. But, when you look at latency under the lens of city, country, CDN, device, operating system, application version, then it begins to reveal a lot of information. For example, the latency in Mexico and Android phones could be high, whereas, the latency in Brazil and Android phones could be low.
It is important to be able to mix and match the different dimensions available in order to narrow down on the root cause.
Support for Custom Data or Metadata
Another point to look out for is the ability to send custom data/metadata to the data analytics platform and use that for further analysis.
This is important because vendors typically build platforms that work for a majority of their target audience and might not be capturing or sending the data that you need to run your platform. In such situations, it becomes important to be able to send custom events or data to the platform that can help in deep dives and debugging.
For example, while the vendor captures all the errors thrown by the video player (by default), you might have tuned the played using custom logic and consequently, you would want to send custom data to track the performance of your logic.
Conclusion
In this article, we touched upon some of the points that a company should consider when choosing a QoE / QoS vendor. Buying a video analytics software is a long and costly process, and it makes sense to carefully consider the pros and cons of each vendor in the process.