Preserving Pristine Live Content through Frame Rate Conversion

Video is not, of course, a moving image. It is a set of still images, updated fast enough to fool the brain and the eye that it is moving. Thomas Edison, an early movie pioneer, said that 46 frames per second was the minimum needed for the eye to perceive motion.

In the early days of television, that was beyond the capabilities of electronics at the time, so a cheat was introduced: the interlaced delivery. This sent half a picture – the odd-numbered lines – first, followed by the second half, the even-numbered lines. This way, you could appear to achieve better than Mr. Edison’s 46 frames a second while delivering half that number of frames.

For all sorts of historical reasons, there is no standard for the rate at which pictures are updated. In many parts of the world, it is 50 times a second, 50 Hz.

• In interlaced delivery, that means 50 fields (half pictures per second).
• Progressive delivery gives the full 50 frames per second.

In North America and other parts of the world, it was initially set at 60 Hz. But when color came along, it was found that the American NTSC system creates a very distracting artifact when shown on black and white screens, which could be cured by shifting the frequency by as little as 0.1%. So American broadcasters standardized on 29.97 Hz interlaced and 59.94 Hz progressive.

Producers who want to make content available to audiences worldwide must convert between production frame rates: 50 Hz in Europe and much of the world, 59.94 or 60 in Hollywood and the rest of North America.

Technically, this is a big challenge.

How does Frame Rate Conversion work?

You obviously cannot just speed the content up or slow it down. Running something at 80% of its original speed would look ridiculous. But if you imagine converting from 60 Hz to 50 Hz, you have to create five in the new version from every six frames in the original.

Simply dropping frames (or duplicating frames if you are moving from 50 to 60) does not work. You end up with a very visibly jerky motion. You can often see this in online videos where camera moves appear to stutter due to poor-quality conversion.

Linear conversion – trying to derive the new frame from the original frames on either side of it – results in blurry video, ghosting of moving objects, judder and strobing. Again, acceptable if there is no alternative but not suitable for prestige programming.

The leading experts in video processing looked at this issue and developed a motion-compensated frame rate conversion technique. As the name suggests, it analyses the movement in the sequence and determines where each moving object should be on the precise time point of the frame to be synthesized.

That calls for two processes.

• First, estimate the motion of the key objects, and
• Second, apply that motion to create the new frames at the required temporal position.

These processes are not as simple as they sound.

A typical picture does not consist of a single moving object on a still background. There will be objects moving in different directions and crossing paths, sharp delineations in lighting across the image, and the camera may be moving, too. So there is much to process in calculating the motion and subsequent picture building.

Frame Rate Conversion is Expensive

Frame rate conversion in real-time, as needed for sports coverage, required a dedicated appliance based on custom-designed hardware.

These standalone frame rate converters had a price tag of $100k or more. Broadcast engineers would never rely on a single point of failure, so main and backup on a single conversion channel would mean a capital investment of around a quarter of a million dollars.

That is a very large sum of money for a single-function piece of electronics.

The practical result is that live international broadcasting was limited to very few events where that investment could be justified. It also meant that today’s sports coverage, where we expect multiple feeds, not just one, was technically and commercially impossible.

Software-Based Frame Rate Conversion is the Way Ahead

The solution lies in distilling all the processing required for live motion-compensated frame rate conversion into software that can run on standard hardware. Further, that software should be capable of running in the cloud.

This is now possible, and we offer it today. The highest quality frame rate conversion can be spun up in a cloud – Azure, GVP, AWS, and others – as needed. It can be offered as a standalone software-as-a-service or sold as a software product in a Docker container to be integrated into a wider delivery workflow.

Cloud media delivery workflows depend upon the video being a compressed video stream, using a format like MPEG, JPEG-XS, or NDI. The software frame rate converter can work within these encoded domains, so there is no need to convert back to baseband – another layer of processing that adds latency – then encode again after conversion – which introduces the risk of compression artifacts.

The critical point is that there is no longer a need for huge capital investment in hardware, which will have relatively low utilization and may need to be shipped worldwide between assignments. Now you can get the same image quality but pay just a few dollars per hour in cloud processing charges. That means live intercontinental broadcasts have become available to many, not just a few of the most prestigious events.

Sport is a major driver here, not least because it has to be shown live, with any latency reducing its value. Cloud processing can be initiated from anywhere in the world, in just a few minutes and with as many instances as you have signals to feed.

Sport is also the most challenging for frame rate conversion, with several camera movements, players, and the ball. Low-quality conversion is instantly identifiable as stuttering and jerky and is unwatchable over a few moments. But with affordable cloud conversion, more sports, productions, and signals can be distributed globally.

This is transformative: democratizing access to all sports coverage by all audiences and confirming the business case through a new charging model tied directly to the time on air.