Live streaming — protocols, latencies & technology options

Live streaming of videos has many applications in live sports & event broadcasting, streaming live classrooms, surgical videos, and many more. RTMP has been a popular protocol for streaming live video from encoders and mobile apps. Encoder support for RTMP, along with libraries for streaming video in RTMP format for Android and iOS, has helped to stream live easily.

Many live streaming services, such as Twitch, YouTube live, and Facebook live, supports RTMP. However, RTMP is based on the flash video, which has seen its end of life. Recently, WebRTC, which is primarily used for real-time video communication, has made streaming directly from browsers possible.

In this article, let us look at various options available for live streaming, their latencies, and review some of the open source projects and AWS-managed services for live streaming.

RTMP live streaming

RTMP to HLS streaming

Live streaming options range from sub-second low latency streaming to latencies of several seconds. Each of these has different use cases, consume different resources, and has a varied cost of operations. For live streaming of video, the possibilities are dependent on the capabilities of clients/players.

HLS has been a universal streaming format with player support across browsers, mobile (Android, iOS), and TV platforms. However, HLS inherently has higher latency because it is chunk-based using HTTP and is being served from a playlist of segments. Low-latency HLS and DASH format are defined, but player support for these formats are still evolving. RTMP has been a de-facto standard for publishing live streams from encoders and mobile apps. However, conversion of RTMP to formats such as HLS & DASH is necessary since players no longer support flash video. 

There are libraries available for browser playback of FLV (over RTMP or websocket) and MPEG-2 TS streams (over HTTP/websocket) which work by transmuxing FLV/MPEG2-TS to ISO BMFF MP4 and feed it to Media Source Extensions (HTML5 MSE). These offer lower latency compared to HLS since they can play non chunked source fromats. They use the same MSE as browser based HLS players use and hence support same range of codec formats only limited by what its container format supports. flv.js and mpegts.js are some examples of these libraries. Even though such options exist, RTMP to HLS/DASH is a more commonly seen option for broadcasting live with several second latencies. 

Many commercial and open-source encoders, such as FFmpeg, support RTMP. OBS Studio, a popular open-source broadcast software, supports streaming over RTMP to several pre-defined services.

WebRTC live streaming

WebRTC is a standards based option for low latency live streaming. It was primarily developed for video conferencing. It is not chunk based and also has option for streaming video & audio packets over UDP. WebRTC also supports peer to peer video streaming but in case of live streaming to several audiences, peer to peer option will not be useful and hence require a server that is capable of relaying WebRTC to users.

RTMP/WebRTC to WebRTC live streaming

Streaming videos to end users using WebRTC has many limitations. WebRTC stack in browsers lacks the same support that is available in HLS streaming. Since WebRTC was primarily used for real-time video communication, many browsers support baseline profile H.264. Hence higher bandwidth is needed to stream video at better quality levels, or users have to be satisfied with lower quality streaming at moderate bandwidth levels. Also, WebRTC needs more server resources and doesn’t scale as easily as HLS, which is HTTP based. Since a websocket connection to the server is needed to be kept open for data transfer, each server can only serve limited end users. Nevertheless, WebRTC-based streaming is gaining popularity, and there are many commercial solutions emerging providing this as a service.

WebRTC to HLS live streaming

For broadcast applications that are needed to be played on a variety of devices and can tolerate some latency, HLS can be used. There are open-source and commercial projects which aim to convert WebRTC to HLS. There is no option to set the I frame interval in the browser WebRTC, and the I frame interval is browser dependent.

Latency can be reduced by keeping segment size minimal with a shorter playlist. However, the small segment size and the short playlist are not handled well by most players and easily lead to stalling of buffers. Hence some experimentation and tuning have to be done to ensure players handle the stream and can tolerate the small chunk size and playlist duration. 

HTML5 MediaRecorder to HLS video streaming

HTML5 MediaRecorder API is primarily used for recording videos on browsers. However, this can also be used in live streaming by recording small duration chunks, uploading to the backend, and converting them to HLS segments with optional transcoding to be streamed to users. This option offers a cost-effective and scalable path that can also be implemented in a serverless fashion with no need of WebSockets. There are some caveats here that the MediaRecorder chunks encoded by different browsers are of different codecs, and all are not suitable for directly converting to HLS segments and need transcoding of audio, video, or both to be able to play on a wide range of HLS supported devices. 

AWS managed live streaming

AWS offers interactive video service which supports RTMP to HLS live streaming. AWS has two options — one which supports only SD and just restreams from RTMP to HLS, whereas another which also supports HD and transcodes live stream making multiple qualities available to users.

AWS also has an Elemental stack — MediaConnect, MediaLive & MediaPackage which can ingest RTMP, transcode to single/multiple qualities and package to HLS/DASH/CMAF with DRM and stream to users. It is a comprehensive pipeline with good features for broadcast video but comes with significant costs.

AWS Kinesis Video Streams with WebRTC is a WebRTC-managed service provided by AWS. 

In this article, we discussed RTMP-based live video streaming with its delivery options and emerging WebRTC-based live streaming and also reviewed technology options in open source and AWS-managed services that can be used for live streaming.