What's a Video Codec? Comprehensive Guide with Examples

A video codec is a software or a hardware device that is used to compress and decompress digital video files. In today’s world, where the internet has become a basic need, video streaming has become an integral part of our daily lives. We watch videos on various platforms, and every platform has its own video format.

But have you ever wondered how these videos are compressed and decompressed to be streamed smoothly over the internet? Well, the answer is – by using a “Video Codec.”

Video codecs are an essential part of digital video production and distribution. They enable high-quality video to be compressed into smaller file sizes, which are easier to store, transmit, and playback.

In this technical article, we will explain what a video codec is, its importance in video production and distribution, and how it works.

Let’s get started!

Table of Contents

What is a Video Codec?

A video codec is a software or hardware that compresses and decompresses digital video data. The term “codec” is a portmanteau of “coder-decoder”.

It uses a series of algorithms and mathematical transforms (such as the Discrete Cosine Transform) to reduce the amount of data in a video file by removing redundant and unnecessary information, while preserving the essential visual details. This compressed video data can be stored, transmitted, and played back on various devices with different computing power and storage capacity.

Some common video codecs include H.264/AVC, H.265/HEVC, VP9, LCEVC, AV1, VVC, and so many others in the research and development stages. Use these articles to learn more about these video codecs.

Types of Video Codecs

There are two primary types of video codecs: lossy and lossless. Lossy codecs achieve higher compression ratios by removing more data than lossless codecs. However, this can result in lower video quality due to the loss of information. Lossless codecs, on the other hand, preserve all the original data and produce a higher quality output but at the cost of larger file sizes.

Let’s go a bit deeper into each of these concepts.

Lossless Video Codecs:

Lossless video codecs, as their name suggests, are video codecs that compress video files without losing any data or quality. These codecs are commonly used for professional video editing and archival purposes, where the highest possible quality is required.

As a consequence of removing very little data, lossless video codecs tend to produce very large file sizes, making them unsuitable for streaming or sharing over the internet. They are preferred for video archival or video editing purposes.

Lossy Video Codecs:

Lossy video codecs are those that compress video files by removing some data or quality from the video. These codecs are commonly used for video streaming and sharing over the internet because they produce smaller file sizes while still maintaining a decent level of quality.

Lossy codecs are able to achieve smaller file sizes by discarding data that is not easily visible to the human eye. For example, they may reduce the color depth of the video or remove some of the finer details. Some examples of lossy video codecs include H.264, H.265, VVC, AV1, and VP9.

Block Diagram of the EVC video Codec — How do codecs work? Block Diagram of the EVC Codec Presented at IBC2019.

How do Video Codecs Work?

Now that we’ve covered some basic concepts, let’s dive into how video codecs work.

Encoding Process

The encoding process involves compressing video data into a smaller file size.

The video codec uses a mathematical algorithm to remove redundant and unnecessary information, while preserving the essential visual details. This algorithm is a combination of a mathematical transform called the Discrete Cosine Transform that converts the data from the pixel domain to the transform domain. Read this explanation of the DCT to a 5 year old!

The encoding process starts with the video codec analyzing the video data to determine the most efficient way to compress it.

The codec then divides the video into small blocks, called macroblocks, and analyzes each macroblock to identify redundant and unnecessary information. Such codecs are called block-based codecs.

The codec then applies a compression algorithm to each macroblock, which removes the redundant and unnecessary information, resulting in a smaller file size. The codec also uses a process called quantization to further reduce the file size. Quantization involves reducing the accuracy of the video data to a level that is imperceptible to the human eye.

The compression process also involves several complex algorithms such as Motion Estimation and Compensation, Mode Decisioning, Deblocking Filters, Mode Prediction, and others to complete the compression process.

Finally, the compressed video data is stored in a container format and can be played back on various devices with different computing power and storage capacity.

For more details, please read our tutorial on video transcoding and encoding.

Decoding Process

The decoding process involves decompressing the compressed video data into a format that can be displayed on a screen. The decoding process reverses the encoding process and is both computationally and mathematically light-weight and not complex.

Decoding is kept simple to ensure that a video can be decoded, decompressed, and displayed at the required frame rates (24/50/60 fps).

Common Video Codecs

There are many different video codecs available, but some are more popular than others. Let’s take a look at some of the most common video codecs.

H.264/AVC

H.264/AVC is one of the most popular video codecs used today. It is a lossy codec that provides a good balance between video quality and file size. H.264/AVC is used for video streaming, video conferencing, and video surveillance systems.

H.265/HEVC

H.265, also known as High-Efficiency Video Coding (HEVC), is a newer video codec that offers better compression than H.264. It can achieve similar video quality with smaller file sizes, making it ideal for 4K and HDR video streaming. H.265 is supported by newer devices and platforms, but it is not as widely used as H.264.

VP9

VP9 is a free and open-source video codec that was developed by Google. It is similar to H.265 in terms of compression efficiency but is not as widely supported as H.264 or H.265. VP9 is commonly used for video streaming on YouTube.

AV1

AV1 is a newer video codec that offers even better compression than H.265 and VP9. It is a free and open-source codec that was developed by the Alliance for Open Media (AOMedia). AV1 is not yet widely supported but is expected to become more popular in the coming years.

LCEVC

LCEVC (Low Complexity Enhancement Video Coding) is a video codec that combines a base codec with an enhancement codec to provide better compression and video quality. LCEVC is designed to work with existing video codecs, such as H.264/AVC and H.265/HEVC, to provide better video quality and compression.

Which Video Codec Should I Use?

While there is no magic bullet while choosing a video codec, here’s a checklist that can help you go in the right direction.

Analyze Your Requirements: Start by analyzing your video production and distribution requirements. Consider factors such as video quality, file size, compatibility, and licensing.
Test Different Codecs: Test different codecs with various devices and applications to see which ones provide the best balance of video quality and file size for your specific requirements.
Compression Efficiency: Look for codecs that provide efficient compression without sacrificing much video quality. Do these tests on the same content at the same parameters (bitrate, framerate, and resolution).
Check for Compatibility:
- Ensure that the codec is compatible with your target devices and applications.
- Check for compatibility with various operating systems, web browsers, and hardware platforms.
- Also, ensure that the DRM you want to use is compatible with the video codecs you’re evaluating.
Check for Licensing: Ensure that the codec is licensed for commercial use. Check for any licensing fees or restrictions that may apply to the codec. Remember, the HEVC codec has been slow off the blocks thanks to licensing and patent issues.

By following this checklist, you can choose the right video codec that meets your requirements and provides the best balance of video quality and file size for your specific needs.

FAQ: How is a Codec Created?

A video codec is typically created by a committee (backed by hundreds of researchers and engineers) and the process is typically like this –

the committee announces that there is a need for a new codec and specify the requirements (example: 50% better compression than the previous codec, ability to compress 8K video, new color formats, etc.)
they ask for contributions, proposals, and suggestions and evaluate their feasibility.
they coordinate all the documentation, testing, experiments that are necessary to decide on the rules and tools for the codec.
and, finally, the committee publish the codec’s specifications for the world to use.

This is the same process which every video codec out there (H.264/AVC, HEVC, AV1, VP9, etc.) has gone through and there are international bodies that regulate the creation, maintenance, and licensing of a video codec.

FAQ: Why is a Codec Specification Important?

Fundamentally, a video codec specification is a document that describes the structure of an encoded bitstream so that anyone in the world can write a decoder for it.

A video codec specification doesn’t tell you how to create that bitstream – it only specifies the structure of the bitstream.

Why is this important?

Let’s say you read online that “Apple has added support for HEVC in their next release”. This means, that an engineering team at Apple downloaded a copy of the HEVC spec and read it cover-to-cover, and wrote a program that can decode a bitstream that complies with the HEVC specification

And why were they able to do this?

Because, the video codec committee wrote the HEVC video codec specification, spelling out every step of the codec, and it’s bitstream – so that anyone in the world can write a decoder for it.

FAQ: Do Video Codec Specifications Define the Encoders?

No, but, there is a bit of word-jugglery going on here. Let me explain.

A codec specification tells you the structure of the bitstream and how a decoder will parse it.

This means that you can create an encoder however you want to – as long as the bitstream that is produced by the encoder conforms to the codec’s specification. That is the most critical point I am trying to make here.

FAQ: What are Test Models and Reference Codecs?

When you create an encoder or a decoder, how do you know that it works correctly? And how do you know if it will work with every encoder and decoder in the world? In other words, where is the gold standard or the reference encoder / decoder for you to compare your code against?

The “need” for a gold standard or a reference is why codec committees and working groups release “test models” and reference encoders and decoders.

These implementations are not optimized for speed or performance. They only contain an implementation of the encoder and decoder as per the specification and you can use them to verify the bitstream that your encoder implementation produced.

In addition, committees often release encoded bitstreams that adhere to the standard. Decoder manufacturers can use these bitstreams to test their software or hardware for codec compliance.

Conclusion

In conclusion, video codecs are essential for video production and distribution. They enable high-quality video to be compressed into smaller file sizes, which are easier to store, transmit, and playback. Video codecs use mathematical algorithms to remove redundant and unnecessary information while preserving the essential visual details.

Understanding the different types of codecs and their features is important for content creators who want to optimize their videos for different platforms.

As technology continues to advance, we can expect to see even more efficient and effective video codecs in the future.

PS: If you want to learn more about video codecs, then check out these articles on OTTVerse.

Krishna Rao Vijayanagar

Website

Krishna Rao Vijayanagar, Ph.D., is the Editor-in-Chief of OTTVerse, a news portal covering tech and business news in the OTT industry.

With extensive experience in video encoding, streaming, analytics, monetization, end-to-end streaming, and more, Krishna has held multiple leadership roles in R&D, Engineering, and Product at companies such as Harmonic Inc., MediaMelon, and Airtel Digital. Krishna has published numerous articles and research papers and speaks at industry events to share his insights and perspectives on the fundamentals and the future of OTT streaming.

What’s a Video Codec? Comprehensive Guide with Examples