Video Software Encoding: 2023 State-Of-The-Art and Future Trends

In this opinion piece, Zoe Liu, the Co-Founder and CTO of Visionular, talks to OTTVerse about her predictions and opinions on how the encoding world will evolve in 2023 and beyond. Zoe shares her thoughts on energy-efficient video transcoding, the growth of HEVC and AV1, real-time transcoding, the role of AI in transcoding, and other topics.

Energy Efficient Video Transcoding

We expect that energy efficiency optimization for HEVC and other software encoding solutions will draw more attention during 2023.

“Energy efficiency” will grow in importance as a selection criterion for software transcoders in addition to the traditional BD-Rate performance criterion (mainly focused on delivering more excellent visual quality at lower bitrates).

This is primarily due to the exponential growth of videos belonging to

• UGC (user-generated content)
• PUGC (professional UGC), such as those video content hosted over Vimeo)

ShareChat’s data shows their platform receives an astounding 75+ million monthly content uploads!

Such a high volume of uploaded videos will result in larger transcoding costs (computation resources and energy). In these circumstances, increases in energy efficiency can result in significant reductions in computational costs while still assuring adequate coding efficiency. 

The solution might be to upgrade to a newer generation codec. However, newer video coding standards use more complex coding algorithms than their predecessors, making encoding less energy efficient.

This brings us to a classic stand-off.

• If new codecs are not energy efficient, video solution providers will not migrate to them.
• But, if providers stay with old codecs, they will lose out on newer encoding tools & methods and will incur high network distribution costs, especially for UGC and PUGC content. 

In this regard, we expect that encoders should balance and trade off the following in the search for energy efficiency.

• Visual quality 
• Bitrate consumption  
• CPU usage 
• Encoding speed 
• Processing delay 

H.264/AVC will Dominate; HEVC & AV1 Will Grow

H.264/AVC

We expect that H.264/AVC will continue to dominate due to its unanimous support across all platforms. Nevertheless, it will also keep losing market share. 

H.265/HEVC

We expect continuing growth for the deployment of H.265/HEVC, particularly for HDR and UHD (2K/4K/8K) video compression.

We saw a dramatic increase worldwide in the deployment of HEVC/H.265 in recent years, especially after Chrome quietly enabled its support for HEVC playback with underlying hardware HEVC decoders.

We have received explicit market feedback that some cloud sports analysis and streaming platforms potentially see extraordinary opportunities in HEVC across all users and platforms. This is because more than 65% of their user base can now play back HEVC streams with the newly released Chrome support.  

AV1

We expect to see a growing deployment of AV1, the open-source royalty-free video codec format finalized by AOMedia.

We have already seen YouTube and Vimeo (the top video-sharing platforms worldwide), Meta/Facebook (the largest social media platform), and Netflix (the leading streaming platform) all roll out AV1 formatted video content to their users.

AV1 will also be expected to take a larger share of the market due to the recent announcement by Qualcomm for its hardware decoding support starting with Snapdragon 8 Gen 2.

We expect Apple to announce its support for AV1 throughout the entire iOS/MacOS/Safari ecosystem in 2023 since we have observed the new AV1 API option in its released AVFoundation framework.

AV1 in RTC (Real Time Communication)

In RTC, there is a need for extremely low latency, and it is here that AV1 for Screen Content Coding (SCC) is expected to grow due to the unique set of coding tools in AV1 dedicated to the effective compression of screen content.

Webex, a leading RTC platform, has already rolled AV1 into production. Fascinatingly, Webex teamed up with NASA and other tech companies and brought AV1 real-time video encoding to the Artemis I mission for significantly higher quality, extremely low bandwidth video calls between the Moon and the Earth.  

Other Codecs

Other codecs will coexist in the market for the year 2023. We expect the market will continue to be shared across H.264, HEVC, AV1, VP8/VP9, AVS2/AVS3, and a few other codecs that currently have smaller market shares, e.g., EVC, VVC, etc.

Beyond the RTC use case, we expect VOD and live streaming use cases to require a lower delay, higher density, and higher speed.

This will be the dominant trending demand out of encoders in 2023.

Architecture Support: x86, ARM, and M1

We believe software and hardware encoder solutions will coexist, addressing different use cases with their respective advantages. In general, software encoders can deliver superior coding efficiency, i.e., better visual quality while consuming less bitrate (bandwidth).

At the same time, they maintain sufficient flexibility to be deployed across a large variety of platforms and devices.  

Software encoders need to consider supporting both x86 and ARM architectures. Specifically, there will be a spurt in the support requests for ARM processors owing to the prevalence of Apple M1 machines, edge, mobile devices, and ARM-equipped servers.

These processors feature significantly lower power consumption and pricing than their x86 equivalents (e.g., deployment of the AWS Graviton processor on Amazon EC2).

Our preliminary evaluation shows that Graviton2 ARM processors may perform 30% slower than fellow x86 processors but offer a significant pricing advantage.  

New Codecs and Improvements

We expect new codec standards will continue to be developed. Specifically,

• AV2 is under development by AOMedia and will provide new coding tools beyond AV1.
• Efforts to create new coding tools beyond VVC have been conducted over the Enhanced Compression Model (ECM) by the continuous joint effort between ITU-T and MPEG.
• The use of AI in video codecs will further be explored, aiming to break the constraint of the traditional 2D Transform + Motion Compensation framework.

Apart from transcoding, technologies that combine video processing and encoding will be developed and deployed. For instance, the deployment of super-resolution has been highlighted by leading technicians in the field to facilitate video streaming.

Per-title ABR will still take the lead to support multiresolution, multi-bitrate video sharing and distribution over many network conditions.  

AI in Transcoding

We collaborated with several universities worldwide to co-author a paper titled “Advances in Video Compression System Using Deep Neural Network: A Review and Case Studies” [1], published in the Proceedings of the IEEE in September 2021, which is one of the early review papers addressing the theme of AI in Video Transcoding.

Apart from this, the 2023 WACV (IEEE/CVF Winter Conference on Applications of Computer Vision) has elected a paper featuring AI in Transcoding as its Best Algorithm Paper [2].

AI provides and shows promise, but it still needs efficient and effective implementation approaches to finally be applied to video codec applications.  

References 

[1] Dandan Ding, Zhan Ma, Di Chen, Qingshuang Chen, Zoe Liu, and Fengqing Zhu, “Advances in Video Compression System Using Deep Neural Network: A Review and Case Studies,” Proceedings of the IEEE, Vol. 109, No. 9, September 2021 pp. 1494-1520. [arXiv.org Online]

[2] Zhihao Duan, Ming Lu, Zhan Ma, and Fengqing Zhu, “Lossy Image Compression with Quantized Hierarchical VAEs.” [WACV 2023 PAPER AWARDS] [arxiv.org online link]