Over the last few years, video streaming consumption has accelerated rapidly. Watching video anytime, anywhere, on any screen has become the preferred way to consume entertainment content in many households. As a result, bandwidth usage and costs have skyrocketed for video service providers. To boost their video profitability, service providers are deploying advanced technologies such as content-aware encoding (CAE).

CAE has seen many incarnations over the years. Designed to reduce bitrates and improve video quality, CAE is an essential tool for video service providers. This article will briefly examine the history behind CAE, highlight the latest innovations in content-aware encoding and forecast the evolutions we can expect in the near future.

The Origins of Content-Aware Encoding

CAE was introduced in 2015 with Netflix announcing a per-title encoding technology that looked at the entire file and determined the best video file parameters to use, such as resolution, frame rate, and encoding settings. (See Figure 1.)

 

Figure 1. Per-title bitrate ladder, per-title encoding blog. Courtesy of Netflix, December 2015.

Per-title encoding was followed by the launch of per-segment encoding where decisions are made at the segment level as opposed to the entire file. Per-segment encoding provides a gain of about 40% to 50% compared with traditional CBR encoding with a fixed ladder. CAE for VOD has been deployed by all major OTT service providers, including Netflix, YouTube, Disney+, HBO, Discovery, and Paramount+. During the start of the COVID-19 global health crisis, the EU asked OTT service providers to drop UHD and to save 25% on the encoding ladder. Overall, viewers did not complain about the impact on video quality but some were frustrated to no longer have UHD available despite paying for it. 

CAE was initiated from a vendor perspective on the file side, followed by a Live implantation, the processing is done in line at the encoder level, and the encoded output is a VBR packaged in either HLS or DASH. 

This technique can also be applied to enable ultra-fast file transcoding, which is critical when video service providers need a quick turnaround of 4K VOD content. The performance of such an approach is roughly 40% average bandwidth savings with a limited CPU overhead and no additional delay vs. CBR encoding. Other companies took a similar approach with CAE being a feature of their encoding platforms, whether as an appliance or in the cloud. The drawback of this type of CAE technology is that it’s tied to the encoder and can only work with the encoder’s vendor. The technology cannot be used by streaming companies who have developed their own encoding platform.

An alternative is to develop a CAE solution that can be bolted on to an existing encoding solution, being proprietary or Open source.

The Next Evolution of CAE

The next innovation for CAE is content-adaptive streaming (CAS), which involves end-to-end streaming orchestration, as opposed to CAE which is based on content characteristics. CAS takes into account any feedback from the network (i.e., client, CDN, physical network) during the content preparation stage. 

The industry is aiming to reduce the bitrate while keeping the same quality and increasing the bitrate for less watched channels. This is a topic of interest for streaming at scale for millions of concurrent viewers, especially for live sports events.  

Conclusion

The video industry has made significant progress since the introduction of per-title encoding, evolving from a per-segment approach to CAE schemes, universal content-adaptive encoding, and finally to a system where the network traffic guides the bitrate savings in real-time. All of those techniques have been instrumental to extending the life of AVC. Now we are seeing new codecs like HEVC and AV1 replace AVC, with the possibility of CAE and CAS techniques being adopted by video service providers to further boost bandwidth savings and improve video quality for streaming applications.