Akamai Closed Captioning Fix

Challenge

Nearly all TV channels host videos on their websites, e.g. up-to-the-minute news broadcasts that give visitors to their sites quick and convenient access to what’s happening in the world as well as in their towns. In order to prepare these videos for the user, they are first uploaded as video files through the backend.

The real challenge though is a live stream that is often offered on TV channel websites. It’s being captured directly from the video source and re-streamed to the visitor’s web browser. In this particular case, videos also had to contain closed captions (CC).

Closed captioning is a process of displaying text as an overlay to the video, in order to provide additional or interpretive information for the viewer. A typical use of closed captioning in real life is the live transcription of the audio portion of a program, which may also include certain non-verbal elements.

This technology was not supported by the streaming solution used by Worldnow. It was therefore up to Redwerk to create an efficient and modern solution which would enable the display of closed captions together with the streamed video files.

Solution

The technology used by Worldnow for re-streaming and displaying live streams on their clients’ websites was provided by Akamai Technologies, a U.S. network which specializes in Internet content delivery. While this is one of the largest distributed computing platforms in the world and currently serves between 15 and 20 percent of all web traffic, the display of closed captions proved to be a problem as they just disappeared once a video signal containing CC data was uploaded to their servers. Our software experts quickly identified the source of the problem: Flash media live encoder (FMLE) actually deleted all CC data when encoding the video signal to AAC/H264.

In order to make this work in a way that would both satisfy our client and deliver a great end-user experience, two possible roads were identified for our project team to take:

1. Find a solution to the issues caused by the existing Workflow and FMLE

During the analysis of the existing solution, the following shortcomings of FMLE were detected:

• Unable to read closed caption data from a device’s pin
• Unable to copy streams without modifying it
• CC data was deleted during h264 encoding
• No configuration option available to remedy this behavior
• No open source solution for modifying this information available

2. Find alternative encoder(s) capable of retaining CC data during encoding

We tried and tested various encoding software solutions, like Afritelly encoder, Wirecast, and Sorenson Squeeze. However, things proved difficult as all of these either had similar problems with CC data or were incompatible with streaming to Akamai. The reason for this was the secure connection used by Akamai, which is protected by usernames and passwords.

In the end and after much deliberation, the winner was… ffmpeg, which is an open source project with a highly flexible configuration and supports tons of formats for encoding and streaming. In order to adapt this solution to our needs, however, a lot of research was still needed:

• Repeat the current workflow process of streaming video and audio to Akamai via ffmpeg. This proved successful as streaming ffmpeg to Akamai through basic auth worked faultlessly.
• Encode CC data as a live stream and send this stream to Akamai via ffmpeg. Here, it was important to make sure that no CC data would be lost. In this stage, development was hampered due to some bugs in ffmpeg, which made it impossible to implement the functionality as required:
  1. The program is unable to successfully read the pins of the device, so CC data cannot be read from it
  2. FFmpeg deletes CC data when encoding to H264, just like FMLE; even if we can read closed captions and encode them to the video stream, ffmpeg will still delete CC data during encoding

This proved to be a complex project, due to the above-mentioned reasons, but the happy ending arrived when we found out that if the step of video encoding is skipped and ffmpeg is used only for transmitting the stream to Akamai, CC data is not removed.

After this piece of good news, the remaining task was much simplified: finding a way to encode video to H264 without losing CC data and to upload it to Akamai via ffmpeg. In order to overcome this final hurdle, we settled on MainConcept H264 encoder, which is able to use video and CC byte-streams as input data and produce a signal with the closed captions encoded to the video stream (a so-called elementary stream).