Technology + Creativity at the �� Feed

Streaming Euro 2020 live at a record new scale

2021-09-21T13:38:10+00:00

Summer 2021 offered a rich selection of events to sports enthusiasts: Euro 2020, Wimbledon and the Tokyo Olympics, to name but a few. An ever-increasing proportion of the ��’s audience has been moving from traditional broadcast to online consumption - the Euro 2020 Final, where Italy played England, set a new record for �� iPlayer, with 7.1 million viewers streaming the match online.

In comparison, the 2018 World Cup Quarter Final, where Sweden opposed England, only attracted 3.1 million online viewers, yet our systems at the time became overwhelmed, leading to a major outage. Luckily enough, this happened at the very end of the match when the outcome was already settled. However, this incident did not go unnoticed by our engineers and triggered a significant redesign of our workflows.

How did we scale our live streaming architecture to reliably deliver media and allow audience figures to reach new heights?

From pull to push

To answer that question, let’s first review what happened in 2018. Many streaming platforms, including iPlayer at the time, operate with what is commonly described as a pull model. The player on your TV, phone or computer requests the media from a third-party Content Distribution Network (CDN). This forwards the request to the ��’s routing and caching servers, which in turn send it on to the packager. The role of this last system is to transform encoded media by wrapping the raw video and audio into container files that are suitable for distribution to a variety of client devices.

You can view things as a funnel: multiple layers of caching and routing are present along the way to collapse millions of player requests every second down to a trickle of requests back to the origin media packager, each layer pulling the media from the next layer up. All of this occurs while trying to keep as close to the live time of the event as possible.

The 2018 pull model. Arrows represent the flow of media requests, from iPlayer on user devices all the way up to the packager.

Even though widely used, this pattern has some significant drawbacks, the best illustration being the major incident that hit the �� during the 2018 World Cup quarter-final. Our routing component experienced a temporary wobble which had a knock-on effect and caused the CDN to fail to pull one piece of media content from our packager on time. The CDN increased its request load as part of its retry strategy, making the problem worse, and eventually disabled its internal caches, meaning that instead of collapsing player requests, it started forwarding millions of them directly to our packager. It wasn’t designed to serve several terabits of video data every second and was completely overwhelmed. Although we used more than one CDN, they all connected to the same packager servers, which led to us also being unable to serve the other CDNs. A couple of minutes into extra time, all our streams went down, and angry football fans were cursing the �� across the country.

�� iPlayer outage during the Euro 2018 quarter-final. Photo: Steve Hy

We did fix and mitigate the immediate performance issue. After running several long and deep root cause analysis sessions, we concluded that having our media servers exposed to a request load that was ultimately not in our control was not a safe position to be in.

We decided to re-architect the system by switching from a pull to a push model. The idea is quite simple: produce each piece of media once and publish it to a dedicated live origin storage service per CDN. From where we stand, the CDN acts as a pure ingest point and has no visibility on any of our servers. In other words, we alleviate any risk of the CDN overwhelming our systems: as more people view our streams, the load remains constant. We end up with a much cleaner separation of concerns, with the �� accountable for producing all media in time and the CDN now solely responsible for caching it and making it available for download at scale.

The 2021 push model. �� iPlayer on user devices simply downloads media that was previously published to a CDN.

Drilling down on the packager

Before they arrive at our packager, video and audio signals are encoded into formats appropriate for distribution to our audiences. The cloud-based encoder is also responsible for generating an adaptive bitrate set of outputs, in other words, a range of picture and sound qualities to cater for different network speeds and device capabilities.

The resulting encoded media is then sent to the packager. Its role is to prepare all the files which will be requested by devices to play a stream over an HTTP connection. The packager produces two types of streams:

- HLS streams: in our case, the encoded media is packaged in fragmented transport stream (TS) containers. Apple devices and some older TVs use these.
- DASH streams: in our case, the encoded media is packaged in fragmented ISO BMFF containers (MPEG-4 part 12), commonly referred to as fragmented MP4. All other devices use these.

At a high level, the revised workflow is composed of four main components, the repackager, the distributor, the manifest generator, and the conductor:

A high-level architecture of the new push packager workflow.

All four components are designed as load-balanced fleets of microservices to support the load of dozens of �� streams continuously flowing through them. These streams correspond to all our linear TV channels and their many regional variants, half a dozen World Service streams, and a variable number of event streams. The workflow is run in two independent cloud regions to allow for additional resiliency. Let’s delve into the role of each component.

The repackager

The upstream encoder produces video using the H.264 standard and audio using the AAC standard. Both the video and audio are wrapped in TS containers, and the encoder continuously sends small container files which hold roughly 4s of content each. The repackager component receives these chunks of content via HTTP PUT requests. Conveniently, their format is suitable for HLS streaming: to make HLS available, the repackager can simply forward them untouched to the next component in the workflow, the distributor.

However, it additionally needs to transform the TS chunks in a format suitable for our DASH streams. The repackager’s task is to parse the TS container and extract the raw H.264/AAC data. Using information gathered by reading through this media data, it rewraps the pictures and sound in an MP4 container with all required metadata. The generated MP4 files are sent to the distributor alongside their TS counterparts.

Structured metadata from one of our generated MP4s, displayed in the mp4box.js viewer.

The distributor

The distributor component receives the MP4 and TS chunks sent by the repackager over HTTP, and in turn, publishes them to CDN storage locations so that our audiences can access them. Depending on the stream, one file may be published to several CDN endpoints for additional redundancy. The distributor can essentially be viewed as a one-to-many mapper, handling all authentication and retry behaviour when interacting with a CDN.

Additionally, for every TS chunk it receives, the distributor sends an HTTP request to the manifest generator, triggering the generation of a manifest corresponding to that chunk.

The manifest generator

Manifests are text documents. Their primary goal is to guide the player to download the different media chunks within a given stream. When the play button is pressed, the player first downloads the manifest, and using the information it contains, keeps on requesting small media files containing 4s of content. The downloaded files are rendered one after the other, allowing viewers to have a continuous and seamless playback experience. HLS and DASH define different types of manifests, and the manifest generator component is responsible for creating these variants. Generated manifests are sent to the distributor component over HTTP to be published to the CDN alongside the media chunks.

The conductor

The conductor provides several REST API endpoints to configure the behaviour of the other components in the workflow. For example, it is responsible for driving the automatic scaling of the microservices when additional streams are started, adjusting the monitoring, keeping track of start and end time of events, filtering encoder inputs, and a variety of other management tasks.

Paving the way to the future

Driven by a cross-departmental effort involving several teams, our push packager workflow has been successfully running in production since February 2021. Initially trialled with the �� Two HD channel, it now continuously carries over 40 streams, with additional event streams soon to be migrated. Over a million new files are produced every hour, and billions of media chunks have successfully been uploaded to CDN storages since its inception. Across all summer 2021 sporting events, the system allowed us to reach a record 253 million play requests on �� iPlayer and �� Sport.

Even though the numbers shared here fit well within the realm of hyper-scale when it comes to data produced and network traffic, it is worth noting that online streaming is a small proportion of the ��’s audience coverage. Linear broadcast TV still makes up the lion’s share. However, online consumption is steadily increasing, reaching about a quarter of all viewers during the summer events, and an IP-only future is fast approaching. With these efforts, we will be better placed to move forward: our new packager gives us the flexibility to adapt our content to take full advantage of improvements in specifications and allows us to reliably and cost-effectively serve millions more online viewers in the coming years.

Want to see our system in action? Open �� iPlayer and simply press play on one of our live streams!

Refreshed interface, extra features: new �� Media Player launched for web browsers

2021-09-02T16:04:34+00:00

We are pleased to announce the launch of our new web media player, codenamed Project Toucan. We have been working on the player over the last year, and it brings many benefits for audiences, including a refreshed user interface and additional features. The new player is already live on �� Weather, �� Food and �� Motorsport for most browsers as it begins to incrementally roll out over the coming months – Safari and iPhones will be supported in due course.

New playback and skip forward and back controls in the new version of the media player.

Toucan is a significant update to our existing Standard Media Player (SMP) - which has been around in one form or another since the 2012 Olympics. The existing player has been continuously updated since then and is still used across the whole of the �� website, from �� iPlayer to �� Sounds and from the World Service to Weather. Over the last twelve months, the player delivered over three billion streams to web browsers on desktop, tablet, and mobile phones.

Why introduce a new player?

The SMP dates from 2012 and has accumulated support for many older browsers and technologies, which are increasingly being deprecated or even no longer exist. One example is the use of iframes which is used to separate the player code from the embedding page code. This technique still works but is no longer the preferred method in the web development community. Additionally, we wanted to completely redesign the user interface (UI) and take a mobile-first approach. Building a brand new UI on top of an old codebase didn’t make sense due to the incompatible technologies we wanted to use. The advantage of a new player is that we can drop all the old code, rebuild more efficiently and be more able to add up to date features and technologies in the future.

Toucan, therefore, supports many modern features that audiences expect. These include 20-second skip forward/back buttons, variable speed playback, slicker animations, and a cleaner design with mobile use in mind. Accessibility is also vitally important and has been considered from the start of our development. The player now has keyboard control – press space to play and pause, arrow keys to seek forward and back, and F to go full screen.

The new volume slider in the latest �� media player.

One major advantage we’ve brought to the new player is a componentised approach. This means that the UI is loaded separately from the main player and enables us to produce different UIs more easily if required and load the most appropriate one. For example, a child-friendly version with larger buttons or additional features could be created and used where necessary (and save download costs by not loading any UI which is not needed). Another advantage of the componentised UI is that we can build-once, use-everywhere, meaning we don’t waste effort building the same UI for multiple use-cases. The new UI that comes with the new player is already being used with the new Chromecast receiver, which has prevented exactly that kind of duplication. This UI can be seen now on your TV if you cast from iPlayer mobile (and shortly, iPlayer web and Sounds mobile).

New Chromecast receiver controls used in the iPlayer mobile app version of our updated player.

We have also updated to a more modern version of CSS and JavaScript, and the player continues to be built on Dash.js, the open-source media playback component, which we contribute back to for the benefit of the Dash.js community. New releases of Dash.js are, of course, always fully tested before making it live in production.

There are performance gains from the smaller size and more efficient build practices of Toucan and benefits from not needing to support as many older browsers as the current player does - and will continue to do so.

It’s built using Web Components, which was recently released in browsers as a native way to produce a component that can easily be integrated in diverse situations. We’ve also dropped our use of iframes in favour of Web Components’ Shadow DOM – this is leading to a significant decrease in player load times.

Now that we have launched the first version, we will be adding all the other features that audiences need in a modern player. This includes audio on-demand, podcasts, live video, live audio, variable speed playback and much more. This will take time, but keep your eyes peeled as we incrementally develop the player across the ��’s digital services.

You will soon see the new player on all our web and mobile services that playback video and audio, including �� Sport, �� News, �� iPlayer, �� Sounds, World Service and many more over the coming months as the features that these services require are built and launched into the new player.

While Toucan is exciting and will eventually replace the current player, we won’t rush to switch off the SMP. It will be needed by the audience where they are using browsers that don’t support the necessary features that Toucan requires, and we are committed to making sure that people can enjoy the �� wherever they happen to be and on whichever platform they choose to use.

But for browsers that do support Toucan - we hope you enjoy that experience. Do send feedback to us at mediaplayer@bbc.co.uk.

Update on ICY streams for manufacturers

2020-12-16T12:15:34+00:00

This blog post is intended to ensure that we have reached all of the vendors necessary for an upcoming change to a legacy way the �� distributes live radio online. There is no action for listeners to take at this time and we do not anticipate any impact to your service.

We’re making some changes to our ICY (sometimes referred to as “shoutcast” or “icecast”) streams, and some vendors may need to update.

We are not turning off these streams, or making any changes to the technical requirements of the devices that receive them. However, the URLs that some devices need to connect to in order to receive them will be changing as part of us moving the delivery of these streams in-house.

This also enables us to offer these streams securely over TLS, something which is increasingly requested by radio manufacturers and aggregators.

When will the changes happen?

We will be spending the rest of the calendar year setting up the new streams and aim to be ready to move devices to the new URLs in January, starting with devices that the �� is able to directly control.

After that we will be reaching out to the organisations we know about asking them to update. Once the majority of devices are using the new system, we will add in-stream messaging on the old streams warning of the move in early to mid-March if there is still significant legitimate usage on the old platform.

The old URLs will then stop working completely by 31st March 2021.

What do I need to do?

If you use an Internet Radio to listen to �� Radio as it is being broadcast, there is no action required and we are confident at this time that your service will continue uninterrupted.

If you are a device manufacturer, station aggregator or other vendor that offers access to our live streams using the ICY protocol, this depends on how your system is configured. By looking at the playback URLs that you connect to you will be able to confirm if any action is required.

Systems that use our Media Selector APIs on the domain 'open.live.bbc.co.uk', we will be updating these APIs and so no action is required by you.

If you directly use URLs from either of the following hostnames:

1. bbcmedia.ic.llnwd.net
2. bbcwssc.ic.llnwd.net

Then you will need to update to prevent impact to your customers as these endpoints will not work after March 2021.

Many vendors should have already heard from our Distribution team about this change. If you have not and need more information on this change, businesses can get in touch by emailing icy-changes@bbc.co.uk.

Going forward, the �� considers its use of the ICY protocol deprecated and will keep it operating as long as legitimate usage levels remain high enough to justify doing so. We will continue to track usage and review our strategy regularly.

Mixing Audio Description for �� iPlayer

2019-08-20T13:09:02+00:00

In 2009, iPlayer became the first video on-demand service to offer Audio Described content. Audio description makes video content more accessible to blind and visually impaired people by explaining what is happening on screen. This helps viewers with visual impairments to follow what is going on.

This external content is available at its source: iPlayer AD guide

Back then, iPlayer was a catch up service for the ��: programmes would only be available on-demand for a limited period after they had been broadcast and recorded from terrestrial TV streams. Since then, iPlayer has evolved, allowing programmes to be made available before they are broadcast. This is made possible through our file-based delivery workflow which has been in place since 2013 and enables pre-recorded shows to become available on iPlayer as soon as or before they are broadcast on TV.

Unfortunately, as of 2018, audio described content was still living in the catch-up paradigm, where content was only made available on iPlayer after it had been broadcast. This meant that old shows, Doctor Who for example, made available again as box-sets on iPlayer, but not scheduled for the TV, did not have their audio described version made available.

This was a frustration for our viewers who could regularly access audio described content on iPlayer, but would find some programmes missing their audio described version even if it had been available in the past, or was to be broadcast in the near future.

With Killing Eve season 2 released on iPlayer as a box-set with all episodes available with an audio described version, I felt it was a good time to tell our side of the story. This blog post will cover some of the tech we used to make audio described content available on iPlayer.

The media files

To create audio described programmes we need two types of media.

Video file of a scheduled programme (MXF file)

We receive a video MXF file, from the production team who made the programme, in a very high quality, high-bitrate AS-11 DPP format. This is a format developed in conjunction with other UK broadcasters for file-based delivery workflows. The MXF file contains the video and the original programme audio without audio description and is used to create all the different formats needed to stream or download our content. It may be in stereo or surround sound, standard definition or HD. You will see that the audio described version of the programme is also created in the AS-11 DPP format with the same quality and features as the regular programme but with audio description dubbed in.

Audio Description Studio Signal (AD WAV file)

An audio description studio signal is produced and delivered to us by our partner Red Bee Media in a stereo WAV format. The left channel contains the audio described voice track, with no original programme audio and the right channel is control data, not something you’d want to listen to unless you find a fax machine weirdly soothing. The control channel is used to dip the original programme audio so as to ensure the audio description is audible over any background audio in the programme. This paper from �� R&D has more detail on the audio description studio signal.

The media tools

To do the mixing we use the following tools.

Gstreamer

Gstreamer is an open source multimedia framework that we use with an audio description plugin developed by David Holroyd. The plugin only works on audio so the original programme audio must be extracted. It mixes the original programme audio and the audio described voice track, adjusting the gain of the original programme audio as signalled by the control track so that the audio description can be clearly heard. The output is a single mono track so this has to be run twice, once on the left audio channel and once on the right audio channel, for a stereo programme.

Principles of fading original programme audio during description passages. Source: WHP051

BMX

BMX is an open-source project from �� R&D to read and write MXF file formats in common use at the ��, written by Philip de Nier. It has had many uses in the �� over the years, most recently as a widely-used engine for generating DPP files. We use the following utilities from BMX to create our audio described content:

bmxtranswrap to extract the stereo audio channels from the MXF file and convert them to uncompressed WAV format.
mxf2raw to extract the MXF metadata and raw essences, which are the video and audio elements.
raw2bmx to assemble an MXF from the metadata and raw essences

The process

The following steps outline how we use the media files and tools to create an audio described version of a programme. These are the instructions implemented in our code that we deploy on EC2 instances in AWS.

Preparing the AD WAV file

We receive AD files in a mixture of 16 and 24 bits, so we normalise them all with Gstreamer to 16 bit. We now have:

Extracting the audio from the MXF file

First, we extract the left and right audio channels from the MXF and convert them into separate mono WAV formatted files.

Each audio channel starts with what is called a “handle” — a term for a sequence of tones used for broadcast timing. The video also includes a ticking clock during this handle period. We need to remove the handle from the audio before we can mix, as the AD file does not contain the handle stage.

So we extract the handle and store it for later. Finally, we extract the audio after the handle to give us just the original programme audio. The audio is now ready to be mixed.

Mixing the audio description with the original programme audio

The Gstreamer plugin then mixes the original programme’s left and right audio channels with the audio described track. The handle is then prepended to both channels so that they are now in sync with the video. This creates our final mixed audio described tracks ready for reassembling.

Extracting essences

We need to extract all the essences from the original MXF file to then insert the mixed audio described tracks. The essences will consist of the AS-11 metadata, one video essence — the main bulk of the MXF — and the audio essences. The essences look like:

Metadata: as11-core.txt, as11-dpp.txt and as11-segments.txt
Video: essence_v0.raw
Audio: essence_a0.raw essence_a1.raw

Assembling the final MXF file

The mixed audio described tracks are used in place of the audio essences, but all other original extracted essences, video and metadata are used to construct the new audio described MXF. We now have our audio described MXF that is used in our file-based delivery workflow to create the compressed assets that can be played in iPlayer.

The quirks

Performance

Downloading, uploading, extracting and assembling media files is a slow process. This is due to the size of an MXF file. An hour-long HD MXF file will be around 60GB. Processing a file of that size takes considerable time. We took some steps to reduce the processing time by using an i3.large instance for its high network performance and high I/O performance with SSD storage.

With these features, an instance can still take over 40 minutes to complete one mix. This raises a problem for auto-scaling instances.

Auto Scaling

It is common for our components to scale out when work piles up in the form of messages on our instance’s input queue, similarly scaling in when work is low. We want to prevent scaling in from terminating an instance that is currently processing one of the messages and doing work.

Instance protection allows an instance to protect itself from a scale in event by requesting protection from the AutoScalingGroup API. When one of our components will have a long running process like creating audio described content it will protect itself when it starts processing a message, then removing that protection after the message and work has been completed.

Long-term storage means long-term costs

To reduce storage costs for the ��, once a programme’s availability has expired on iPlayer, only the original MXF file is kept for long-term storage. All other versions of that programme are removed, including the audio described version. This is why in the past audio described content that was previously broadcast did not reappear on iPlayer when its original version was made available again.

This is no longer a problem, because the audio description studio signal is now kept in long-term storage. An audio described version of a programme is now recreated when ever the programme is rescheduled for the iPlayer. This is more cost effective than storing the audio described version.

Testing with Docker

The majority of our micro-services don’t manipulate media; that is done with on-site hardware or third party cloud transcoders. The audio description workflow is different. It relies on the tools mentioned above: tools that only a few of our developers have installed locally. We could have mocked the calls and just checked the functionality of the code, but we wanted something a bit more robust.

With Docker we are able to run our code and tools as if they’re running on an EC2 instance, thereby testing the whole audio description workflow locally and in our continuous integration workflow. We do this with a base image of our underlying EC2 instance’s OS and installing our project as an RPM. When testing, we overwrite the compiled jar on the image with a local copy to avoid rebuilding the image.

Interested in learning more?

Visual seeking for �� iPlayer

2019-02-22T10:56:50+00:00

In our commitment to providing quality of service and ease of use, we launched the 'Visual Seeking' feature across the �� websites. This post outlines some of the design decisions that were made to address technical and UX concerns.

A while back we added our visual seeking feature for videos that play across the �� websites, such as News, Sport and iPlayer. This required close collaboration between Media Services (who develop Video Factory, encoding media content for the Web), �� Research and Development (R&D), the �� Online Technology Group (OTG) and Media Playout (who develop SMP - the Standard Media Player used across the �� website).

Visual seeking (aka Thumbnail Scrubbing) is where a small preview picture of the video appears when you move your mouse over the scrub bar. For example, let’s say you wanted to skip to a particular celebrity in Strictly: you can now quickly find the right part by hovering over the scrub bar.

We turned on the back end services a month before the feature went visible to the public, so most programmes had thumbnails available at launch. We later ran a backfill job to generate thumbnails for extended availability content.

We initially launched the feature on desktop and tablets that use a responsive web player. We added the feature later to Sports Apps for mobile devices and recently IPTVs. Other Apps for mobile devices will follow.

The challenges

There were some challenges to address and consider when delivering the visual seeking feature to achieve high perceptible quality, low latency, efficient broadband use and supporting wide range of playback devices with different requirements.

The rest of the blog post will give you details of the solutions and the trade-offs that were considered.

Faster

One way to enable thumbnail scrubbing could be to download the entire video in low quality, then generate the correct frame by seeking the low quality video to the correct place. Decoding video in this way is a particularly CPU intensive process, so instead we opted to generate thumbnails as a collection of JPEGs ahead of time for the media player to consume.

Let’s say a video contains 500 thumbnails. To download these images, the player could make 500 separate requests for one thumbnail at a time. The problem with this is that each request has an overhead, a delay before data starts arriving. Poor network conditions make this delay significant, and browsers will limit the number of simultaneous requests that can be made, resulting in poor performance.

We can improve the efficiency of acquiring the images by packing the individual thumbnails into storyboards. A storyboard is a 5x5 arrangement of thumbnails packed into a single image file. This reduces the number of requests that the player needs to make by a factor of 25 rather than making 500 separate requests for images, the player only has to make 20. The player then treats this storyboard as a sprite sheet, displaying only one thumbnail at a time.

Efficiency

An example storyboard of 25 thumbnails

The player downloads storyboards ‘on demand’ to make efficient use of the broadband connection. When the user hovers over the scrub bar, only its related storyboard and two adjacent neighbours get downloaded. Because the storyboards are downloaded in this way, we can afford to produce a greater number of thumbnails for a video to enable a very fine level of scrubbing. Move your mouse one pixel to the right and you’ll get a new thumbnail. This helps to make seeking more precise, which is important for longer programmes.

Packing thumbnails into storyboards has some pitfalls of its own. The thumbnails themselves compress much better using the JPEG format rather than PNG or GIF. Given this, we decided to use JPEG as we get the same visual quality but for a smaller file. The JPEG format breaks the image into 8x8 pixel blocks, each of which are compressed separately. To provide good quality for the compression used, we needed to pick thumbnail sizes where the edges of the thumbnails fall on the boundaries of these blocks to limit 'bleeding' between adjacent thumbnails. �� R&D recommended three different thumbnail sizes to cater for different screen sizes. The sizes were picked to reduce this bleeding as much as possible: joins between columns line up perfectly, but the joins between rows don't always align with the blocks. Where there is misalignment, the frames will be seen to have a fuzzy edge as seen in the example below.

Compare the fuzzy boundary between thumbnails 2 and 4 with the sharp boundary between thumbnails 3 and 4. The content of adjacent thumbnails affects the image where the 8x8 blocks do not align with the thumbnails.

Standardisation

Another big challenge was the absence of standardisation for media representation of the thumbnails for video in online video industry. �� R&D provided us a library they developed which is based on streaming clips which is strategically better but not supported on TV version of iPlayer. Thus, we couldn’t use it. �� R&D and Media Services collaborated on producing a form of media presentation description for the thumbnails of a video based on the MPEG-DASH industry standard. Being able to describe the thumbnails as media is particularly beneficial when used with the storyboards to structure the description of thousands of thumbnails for video.

Thumbnail scrubbing media presentation description data structure

The extension to the standard also allows us to describe and structure varying number, size and layout of the thumbnails for the same video. To do so, we preferred using separate adaptation sets for each storyboard with different characteristics to reduce the restrictions of the media description to build a future proof foundation.

Thumbnail scrubbing media presentation description file

The thumbnail media presentation description extension was communicated with the MPEG-DASH community, which led to recognition of thumbnail support in media presentation description hence became part of MPEG-DASH industry standard with DASHIF V4.1.

Another Brick in the Wall (of microservices)

Video Factory is the product that deals with the conversion of audio and video for publishing to the �� sites including iPlayer, Sport and News. It is built using microservices - small pieces of software each with a single purpose, one step in the much larger workflow.

Using this architecture pattern, when we want to add a new feature we just need to create a new microservice and add it to the workflow. This is exactly what we've done for Visual Seeking.

We encode very high quality source video and produce a variety of different bitrates to support playback and streaming with over 1000 different devices, including PCs, phones, tablets, games consoles, and smart TVs. These individual bitrate videos are then distributed and published. We included the thumbnail feature by using the encoder output for a specific bitrate and creating thumbnail storyboards and media presentation description manifest file which are then distributed and published.

The thumbnail scrubbing feature in Video Factory

By adding the new feature in this way, we were able to make the change without affecting any of the original services. Thumbnail generation and the publication of videos work independently of each other, meaning that there is no extra delay in the videos being published; and if the thumbnail service were to fail, the videos would still appear on iPlayer.

The �� thumbnails are originated by Radix and distributed by CDNs

The thumbnails are distributed through a high performance and resilient content delivery architectural solution that also provides authentication and content routing. Radix is a web cache that originates the �� thumbnails as well as the �� on-demand content that is distributed by CDNs.

It’s all about compromise

We had a number of design decisions to make, and ultimately they were decided by compromising between quality and efficiency. We made the decisions considering the trade-offs of each option after some measurements based on the following facts:

Increasing the quality of the image leads to an increase in the storyboard file size
The bigger the file, the longer it will take to download
The perceived quality of a small image doesn’t change after a certain quality level.
An increase in the number of thumbnails in a storyboard also increases the file size but reduces the number of requests that the player needs to make to be able to get thumbnails for video.

We made a target to produce storyboards at approximately 100KB each. Storyboards of this size can be downloaded very quickly on most connections, and they are of reasonable quality. Devices on slower connections will benefit from progressive JPEG loading, meaning that the full image may appear in low resolution before it has finished downloading.

Even though the frame on the left has only loaded 25% of the image data, the entire image is visible and the context is still discernible.

The file size requirement meant that there is a noticeable loss in image quality, but the effects have been reduced by picking thumbnail sizes that play nicely with the JPEG format.

Our microservice creates storyboards from an input video. We could have used the highest quality output from the encoder to produce storyboards, but using such a high quality input would require a powerful server and a lot of time. Instead we can produce essentially the same output with one of the lower quality videos from the encoder: this vastly reduces the amount of resources required (therefore smaller operating costs) for an almost imperceptible difference in quality. The bitrate we’ve chosen takes under 2 minutes to produce thumbnails for an hour-long video.

Spoilers?

The introduction of visual seeking could mean that people might accidentally see a key moment of an episode’s plot as they move their mouse towards the full screen button. This is why we have introduced a short delay between the user hovering over the scrub bar and the thumbnails appearing on our media player. We hope you find Visual Seeking useful. We welcome feedback and we’ll try to answer any questions you may have.

We are hiring!

Video Factory is one of the products that the Media Services team are working on at the ��. There are new features and media technologies we use to improve our services. Visual Seeking is merely one of them. If you like the idea of working with video/audio encoding, publication and live and on-demand streaming services and want to play an important part in the future of online media, you can apply for the job openings here when available.

Grinding Java 11 beans into production

2018-10-04T15:33:15+00:00

Pierre-Yves Bigourdan and Sam Lukes are two software engineers from D+E Platform Media Services, where they build highly available Java microservices in the cloud to enable the ��'s Live streaming capability. They give a light-hearted account of their experiences with Java 11.

On September 26th, 2018, the sun was out for the last remnants of the late British summer. It also happened to be �� Media Services team's monthly innovation day. "Hey Sam, let's try out Java 11..." were the fateful words which led us to push the latest version of OpenJDK into production. Java 11 is the designated long-term support version and had been made publicly available just the day before, therefore ending support for Java 10 which younger versions of ourselves had enthusiastically deployed only months before!

Getting the new version of the JDK set up on our laptops was no problem and after upgrading our respective Eclipse and IntelliJ installations to the latest beta versions, we were in business... Well, barring some intriguing CloseFileExceptions:

Getting our project building with the newly installed compiler was more challenging than we first thought it would be, due to yet more exceptions:

java.lang.IllegalStateException: Could not find sun.misc.Unsafeorg.apache.maven.plugin.PluginExecutionException: Execution default-testCompile of goal ...

Some tips to get you going if your build blows up too:

If any of the libraries you use perform bytecode manipulation, pin their transitive dependencies such as byte-buddy or asm to their latest versions.
If you are using Maven as a build tool, update the Compiler and Surefire plugins.
Include any removed Java EE modules as dependencies to your project, for instance, java.xml.bind.
Migrate any code using removed methods or now encapsulated internal APIs. We're talking about you sun.misc.Unsafe.

"How do we actually install this on the server?" was our next troubling question. Unfortunately, this was not as simple as yum install openjdk-11 since no packages were provided at the time of writing. To solve this problem we had to create our own rpm, the preferred way of installing software on our chosen Linux distribution. When built, our rpm downloads then unpacks the binary files from the OpenJDK website and when installed it registers the executables (java, jps and friends) as alternative Linux commands. We also had to do some extra work to make sure the cacerts truststore used by Java included all the certificates that we needed to be able to talk to our internal services and cloud providers.

We use Jenkins as our continuous integration server and for each project we have a pipeline of jobs which build, test and deploy a release to the cloud. Getting Java 11 available for builds was as simple as installing our newly cooked rpm onto the build slaves and tweaking some global configuration so that Jenkins knew how to configure each job's environment.

However, this was not the end of the story as an uninvited guest showed up to the party, our code quality tool SonarQube. Our build pipeline was blocked due to the fact that several of its plugins were not yet quite ready for the latest version of Java. After a few dependency tree tweaks, a submitted pull request and building from source, we had lift-off. Finally, one of our services was happily processing requests using Java 11 in our pre-production environments!

Whilst so far we are still experimenting with some of the more radical features of recent Java releases such as the modular system, we are already enjoying some of the API and syntax updates including:

a modern and flexible HttpClient
the polarising var keyword for local variable type inference
many richer API methods on collections, strings, optionals, streams...
and many more!

If you are wondering whether you should upgrade to Java 11 yet, then from our experience the answer is "yes" it's definitely possible and a microservices architecture which allows you to take one step at a time will certainly help. However, you might want to wait a few weeks for the open source Java ecosystem to catch up!

Broadcasting the World Cup and Wimbledon in UHD - the full story

2018-07-19T10:00:00+00:00

Jim Simmons from D+E Platform Media Services, discusses our experience of delivering live Ultra High Definition at scale for the World Cup and Wimbledon on a round-by-round basis.

Qualification

We started discussing the possibility of bringing the World Cup and Wimbledon to the audience in UHD towards the end of 2017. A cross-divisional team was set up with members from �� Research & Development, Design & Engineering Platform, and Online Technology Group (OTG) who manage internet distribution, as well as the engineering and support teams in Broadcasting House and Salford, Sport Outside Broadcast (OB) who work closely with our commercial OB facility providers, our TV Platform team who test all the devices that use iPlayer, not to mention the iPlayer and Sport teams themselves.

If you want lots more technical detail of our set up, please check out this blog by our amazing developer, Lloyd Wallis.

In short, the outside broadcast team encodes the UHD at a very high bit rate on site using a contribution encoder and gets it to Broadcasting House in London via IP where it goes in to the video router. This feeds the UHD distribution encoders which generate four bit rates of HEVC video up to 36 Mb/s (iPlayer HD is 5Mb/s for comparison). This is then packaged using the DVB-DASH standard so that 4 second long chunks of audio and video can be requested over the internet via a playlist called a manifest. This is served via an origin server in “the cloud”, which is then cached by servers in a �� data centre to protect the origin. There is a further level of distribution provided by CDNs (content delivery networks) whose job is to serve very large amounts of traffic to very many of people across many ISPs who deliver the chunks to the connected TV. Easy.

We began building this chain in the spring but there were many unknowns. We needed to test contribution and distribution encoders, work out the best bit rates, try and bring all the networking together, organise the operational and support infrastructure - who pressed what button when etc. make sure we had levels of redundancy, test lots of different TVs and devices, ensure there was sufficient internet capacity. There was a long to do list. While we were building, we were also preparing our core services on iPlayer and iPlayer Radio for what we were expecting to be our biggest ever live streaming events.

There weren’t many opportunities to test our workflow. There are not that many UHD capable OB vans in the UK and they are often busy providing sports coverage for BT Sport and Sky so we only had a couple of events to test. We didn’t publicise these because we had no idea if they’d work at all.

We covered the FA Cup game between Southampton and West Brom (which I’ve now seen about fifty times), the Rugby League and some of the FA Cup final. By the end of this testing, we had a good feel for the workflow but we hadn’t put any significant load through the systems. We had only had a couple of thousand viewers.

We did discover that the bit rate for UHD averaged out at around 25Mb/s per viewer. We added this in to the total bandwidth predictions for the World Cup across the �� to see how many concurrent users we could comfortably support and found that we would need to limit access to the trial. The standard HD and mobile streams were our first priority so we needed to ensure the trial didn’t take bandwidth needed for normal iPlayer and �� Sport viewing.

We decided we’d be comfortable with a maximum number of around 60,000 concurrent UHD viewers, but that the number would be continually re-assessed during the tournament depending on overall performance. We also estimated there were maybe 2 million devices that might be capable of playing the UHD streams, hence the warnings about the streams being available on a first come, first served basis.

The Group Stages

The World Cup kicked off for us on the Friday 15th June as Egypt took on Uruguay. Trivia fans will want to know that the first ever UHD World Cup goal on the �� was scored by Gimenez for Uruguay in the 89th minute.

The first couple of games ran relatively smoothly from a production point of view. We were closely monitoring feedback from audiences, via a survey and also through social media. The audience response was generally very positive, especially with regard to the picture quality. There were some comments about stuttering and juddering and speech getting out of sync despite viewers having high speed internet.

In the France v Australia game on the Saturday, some video didn’t arrive in time to be served to the audience. Unfortunately, the audio did arrive as DASH separates the two. This led to the audio becoming one segment out of sync - about four seconds. We failed over to the back-up. If you left the stream and came back, the issue was fixed, but if you stayed on the stream it remained faulty.

In the later game between Germany and Mexico, we had a router switch fail which caused more issues.

The quality of service that UHD demands of network infrastructure quickly roots out any problems, or components that are introducing latency. We replaced some optical connectors and noticed significant improvements in our logs and a reduction in complaints.

While we were fixing switches and networks, we took some other precautionary measures. We removed the lowest bit rate to give the distribution encoders less data to generate in each time period and we increased the wait time on the packagers. This increased our latency by a further 10 seconds. Not ideal but we wanted to keep the audience experience as smooth as possible.

We’d also noticed a number of comments on the size of the “�� UHD” logo in the top right hand corner of the screen. This was set to a DPP (Digital Production Partnership) guideline but we all agreed it did look a bit big, so we reduced the size by 10 percent. We changed that on the Tuesday of the first week.

On the day of the first England game, 18th June, one of our contribution encoders had an issue during the preceding Belgium vs Panama match so we removed it from the chain and switched to the back-up. We used that for the rest of the tournament.

The England vs Tunisia game proved to be our most streamed live event up to that date. We stopped access to UHD in the first half for a short period of time as we re-balanced traffic across our CDNs to optimise performance. Things calmed down a bit and we re-enabled UHD access at half time. We had just over 35,000 UHD viewers at this point but were shipping around 5Tbits of data per second across all services.

We continued to work on our distribution and found and fixed further issues with networks and infrastructure along the distribution chain during the next few matches as we discovered them.

As the technical situation settled down, we felt confident enough to let the presentation team on �� One mention the fact that there was UHD available. This was done by the commentators and an on-screen graphic. Doing so resulted in an immediate spike of 10,000 new viewers within a few seconds.

Two days later (27th of June) we had a serious external networking incident that caused a problems for viewers such as buffering and streams failing. We let the stream run, but there were only a few thousand watching. Things recovered for the second half of the South Korea vs Germany game. We asked Sport not to mention us for a little while just in case.

Round of 16/Wimbledon

As we moved in to the round of 16 in the World Cup, we added Centre Court coverage from Wimbledon. This added a bit of complication as we didn’t have enough UHD encoders to provide 2x resilience for both events. This meant a degree of source swapping for the back-ups.

As the network issues had now been resolved, we put back the lowest bit rate and reduced our packager waiting time by 20 seconds. This got our latency down to around 45 seconds - but it was very dependent on what device you were using.

Audiences started to increase, both for UHD and in general. For the Belgium vs. Japan game, we peaked at 37,300 concurrent UHD viewers, including football and Wimbledon.

Quarter Finals

Things started to hot up now. For the quarter final between Brazil and Belgium on Friday 6th July, we had 41,600 concurrent UHD viewers at the peak (including Wimbledon viewers). We still hadn’t hit our theoretical maximum limit by full time though.

England vs. Sweden was on Saturday 7th June with a 3pm kick off. We hit our UHD cap of just over 60,000 concurrent viewers half way through the first half. We pretty much stayed at this level for the rest of the match.

Semi Finals

We ran UHD for the France vs. Belgium semi final with no issues and just over 48,000 concurrent UHD viewers. The men’s semi-finals at Wimbledon provided a good soak test for our encoders and staff with play continuing until just after 11pm.

The Finals

The tennis final on Saturday 14th, and tennis and football finals on Sunday 15th July ran smoothly. A peak of 44,300 viewers saw Novak Djokovic and France win their respective tournaments, including the first penalty awarded with the help of VAR in a World Cup final. We don’t know if the VAR was in UHD or not.

We stopped the last stream around 7pm and put live UHD to bed for now.

Extra Time

As well as the technical issues we addressed at the time, there were a couple of other common comments that I’d like to address now.

Latency

The elephant in the room for live internet streaming is latency. Well, it’s in the room if you’re watching on Freeview. If you’re online it’ll be a minute or two yet.

Latency is less of an issue for everyday viewing, but it comes in to sharp focus when we look at sport. The usual latency for online streaming is between 30 and 90 seconds depending on your viewing device. For UHD we got latency down to between 45 seconds and a few minutes but it was very variable, again depending on the device. Interestingly, when we asked viewers about latency in our survey, while they wanted it to be as low as possible, most said they wouldn’t trade it off against picture quality.

The problem is one of scale versus speed. It is possible to stream video very quickly over the internet. It’s used all the time for video calling. The problems with low latency streaming are that it generally only supports small numbers of viewers, or it requires very proprietary hardware and software, or there is a significant reduction in quality or resilience. As more and more audiences move online this is becoming a significant issue for the whole industry.

Fortunately, it is also an issue the industry is beginning to address, both through developments by hardware and software vendors, and through the work of standards groups which the �� is contributing to. Ultimately, we need to get the latency of IP delivery comparable to that of linear distribution and we, in �� Platform and OTG have already started working through those issues with our partners in the industry.

It’s too dark

A couple of themes that came out of the user feedback were around the picture being too dark or the colours not being vibrant enough. Having worked with the production teams we think we have managed to improve some of the issues we found e.g. ensuring graphics are inserted at the correct level. For a more detailed assessment of some of the possible causes at the home end then please see this blog written by my R&D colleague Andrew Cotton.

It’s Coming ��

The UHD trial was a brilliant learning experience for us so we definitely consider it a success. It’s clear there’s lots more to learn, work on and improve, and massive events like the World Cup and Wimbledon really put our systems through their paces. We hope those that watched it had a good experience, and sincerest apologies for when it didn’t go so well. Our TV in the office buffered just as France scored in the semi-final and I was livid so I do understand how it can be frustrating at times. Thank you for trying it if you did.

We built the trial in such a way that it can be re-used and improved upon. This will allow the next trial, whatever it may be, to take advantage of all our experiences and fixes we’ve implemented this time around. Live streaming Ultra HD over the internet is at a very early stage, but we’ve taken an important step with this trial. We will continue to build our capability and make sure future audiences have a free-to-air option for live Ultra HD and HDR.

Inside our UHD workflow

2018-06-22T09:52:09+00:00

Last weekend we ran our first streams of the World Cup in Ultra-High Definition and High Dynamic Range, and this week announced that we would be continuing this with our coverage of Wimbledon's main court.

Earlier, our colleagues in �� Research and Development have spoken about some of the deep technical challenges that have been faced.

Meanwhile, in �� Design + Engineering Media Services we’ve tackled some problems of our own around how to take these concepts and build a stable production process with the ultimate goal of handing over the management of our UHD streams to the production teams.

We’ve learned a lot over a first weekend of streams and will be following up with a summary of the issues we’ve encountered during matches and the changes we’ve made to reflect them at the end of the tournament. This post introduces the key components of a live stream and what we ended up building out for UHD.

For Media Services, a live stream at the �� generally consists of the following base components:

Contribution decoding — where the video signal from the outside broadcast site is received back in a �� site and decoded.
Distribution encoding — the signal is then encoded into the final format used for distribution to users, including the full adaptive bit-rate (ABR) set of encodes and structure for chunked-HTTP delivery
Packaging — this encoded media is then packaged into the correct fragmented MP4 containers with appropriate metadata and manifests that can be served to users over HTTP
Distribution origins — the ��’s origin servers that downstream caches use to collect streaming content
Content Delivery Networks — huge third-party networks and caches that provide the scale of delivery needed to handle the ��’s streams to ISPs and other edge networks
Internet Service Providers — the companies our audience pay to receive their internet connection at their home or business, or on their mobile devices
Client Devices — your TV, mobile, or PC

For our existing HD and radio workflows, this is a relatively well defined set of services that we have run for several years now. While there’s plenty we want to improve here, and we always are, it works. For example, two years ago, for the UEFA European Championship, managing ~2.4Tbps was a squeeze at multiple points in the delivery chain, but recently we comfortably managed a record-breaking 2.8Tbps of peak traffic during the Royal Wedding. We broke that on Friday by reaching 3.4Tbps during our first World Cup match, and shortly after broke that one too with just over 5Tbps during the England opening game against Tunisia on Monday.

For UHD however, we’ve had to review every part of this chain. In some areas this involved choosing vendors, for others it involved building new services. After a lot of work across �� R&D, Broadcast Engineering, Media Services, TV Platforms and the Online Technology Group (OTG), we’ve managed to build our first attempt. It’s far from perfect, but here’s how we ended up where we are for our first iteration.

Contribution Decoding

Video services locally generally use a set of cabling and signalling standards called SDI. Since UHD is essentially four times the amount of data, Quad-SDI is just four SDI signals, each carrying one quadrant of the stream.

As a raw signal, this would be tens of gigabits per seconds over the internet from Russia, so would be infeasible. Instead, a ‘contribution’ encode is sent to us. This is an IP stream that is quickly encoded, optimised for low latency rather than very low bit-rates; for the World Cup we’ve opted for a 120Mbps signal. This is then sent back to the UK, where our contribution decoders convert this back into a Quad-SDI signal output over 4 SDI cables.

Distribution encoding

The �� generally prefers to launch new services using cloud services — whether fully managed or running ourselves on cloud infrastructure.

All new ‘regular’ live services we launch, such as World 2020’s Guraji, Telugu, Punjabi and Marathi bulletins, use cloud-based distribution encoding. Meanwhile, we’re hoping to move as much of our legacy on-premise encoding kit onto a cloud product as they reach the end of their useful life.

Unfortunately, for live UHD encoding, most encoding architectures require all ABR representations to be produced on the same instance, and it is very hard to get enough compute resource on one instance in the cloud to achieve this.

Therefore, for this we had to buy physical appliances — each of which (with 4 GPUs) can support only one UHD output. Even then, we had to remove some of the ABR representations we hoped to offer.

Distribution Packaging

After distribution encoding, we normally use a separate distribution packager to package the encoded segments into the formats required by end client devices. For regular HD content, the encoder delivers this to the packager using Microsoft’s Smooth Streaming format over an HTTP POST. Unfortunately, our encoder does not support HEVC video in a Smooth output (understandably so, as how to do so was only defined in March), so we were unable to use our normal packaging component.

Our saviours here, as is often the case, were our colleagues in �� R&D. They for some time have had an internal ‘reference’ packager that is used for internal tests and for some small public trials such as Radio 3 in surround and lossless audio that have been run during the Proms in recent years. This packager operates a ‘pull’ model, preparing chunks for distribution as the encoder writes the encoded output to disk, so removes our Smooth dependency. We’ve worked with the team there to get some more physical, real world servers installed, built and working in a largely production-ready state. The R&D packager then outputs the packaged material into an Amazon Web Services S3 bucket.

Distribution Origin

Our distribution origin services for live serve us well. These are caching/load balancing appliances today. Capacity planning for the initial desire of up to two concurrent UHD streams through them though suggested that we would be reaching the limits of what these devices were capable of, so we’ve looked to replace these too.

For on demand, we have long used our in-house Radix solution. Our OTG colleagues have now set up a new Radix pool for handling UHD content and everything here is now looking pretty good.

Graph of UHD usage patterns for the opening World Cup weekend

Content Delivery Networks & Internet Service Providers

Currently, the ��’s top streaming bit-rate is 5Mbps for HD content. The 36Mbps we’re offering for UHD is therefore proving to be quite a challenge in terms of distribution capacity.

It’s for this reason that we announced a strictly limited number of entries into the UHD trial service at the same time — there just isn’t enough distribution capacity in the UK if every compatible device tries to watch a UHD stream at the same time. We’ll be reviewing this number continually during matches to try and find the right balance between offering the trial to as many users as possible and protecting the stability of internet distribution — both for us and our commercial CDN’s other customers. There’s a few unknowns we have that are resulting in us being conservative initially:

The number of audience members who watch in HD via iPlayer who will now watch in UHD instead
The number of audience members who watch via traditional broadcast methods (Freeview/Freesat etc) that will now watch in UHD instead — this is the first time we are offering higher quality online compared to broadcast so we can’t predict how many people we convert to IP from broadcast
The accuracy of our bit-rate modelling from earlier trials — we have an understanding of how many devices played each of the different bit-rates based on the earlier trials and our live UHD test loop; but didn't know how the model would hold up until we saw our first England match

This ultimately means we’re monitoring very closely and this limit may move up and down as we learn more about how the service is used.

Client Devices

A substantial testing process has been underway to get as many devices as possible on the trial. Additionally, whilst we’ve offered DVB-DASH streams before, UHD is our first DASH-only service so there was a lot of effort around tightening up support and moving more devices over from other streaming formats. The key things we’ve been looking for are stable decoding of a 36Mbps stream and support for either BT.2020 SDR or BT.2100 HLG HDR.

When set-top boxes are involved, we also have to factor in whether the connected display can present the signal — it would be a waste for us to deliver a 36Mbps UHD signal to a panel that can only display a 1280x720 HD image. Therefore, we have had to add code to detect the capability of the connected display and only offer UHD where the connected display is capable. There currently lacks a standard for detecting these capabilities, which made this quite difficult.

Next Steps

There will be lots to learn from this summer’s UHD sport events, and we’ll be going back to look at how to improve the broadcast chain. We’ll also be doing a lot more analysis on how UHD changes when and where the audience consumes content.

Oh, and by the way, Media Services is hiring!

The road to an IP future

2018-05-11T10:03:41+00:00

Yesterday, I gave a speech to the members of the Digital Technology Group (DTG) at their annual meeting in London about one of the ��’s priorities for the future – the road to IP (IP in the sense of delivering programmes and content over the internet). There’s a lot at stake here for our audiences and partners in the UK broadcasting and creative industries. I want the �� to take the lead in telling the story about how we create an IP future that works for us all.

We know that many people already use the internet to consume much of what they watch, listen to and read. In fact, we’ve just published data showing that following a hugely successful January and February, stats for March show �� iPlayer has had its most successful quarter ever – with record-breaking request in the first three months of this year. There are also, however, many people who continue to watch and listen on traditional channels. We at the �� have a duty to serve both audiences.

But when that internet-only world comes (as it inevitably will), we must be at the forefront of change, we must take advantage of the opportunities offered by IP delivery to get more, better, personalised content and services to audiences and to sustain a thriving UK creative economy. In the meantime, we must deliver our traditional services brilliantly – and look at incorporating devices that make the most of both worlds on our journey there.

In my speech yesterday, I outlined the opportunities and challenges of this internet-only world – and how, working with the industry, we must shape this next chapter in the internet’s story, but importantly not leave anyone behind on the journey.

I talked about challenges such as the availability and take-up of high-speed, reliable broadband by everyone. Investment has already been earmarked for this – but more needs to be done.

Another challenge is the threat to the unique media ecology in the UK – which many feel is one of the jewels in the UK’s economic crown. We know that the internet economy tends towards winner-takes all markets – lead by companies that have no particular reason for sustained investment in a broad-range of UK content, or news plurality, or positive outcomes for society. The �� is working hard with others in these areas – providing safe spaces for children online and tackling fake news – but we all need to do more.

And finally, I talked about preserving the qualities and attributes of broadcast in an internet-only world. Qualities such as breadth, universality, even intimacy – in terms of a direct, unmediated relationship between broadcasters and our audience. We must continue to give audiences not just what they think they want – but surprise and delight them with those hidden gems or things that will add value to their lives (not just create more filter bubbles). We want not only to carry these over to an internet world – but use the unique power of the internet to amplify and enhance those attributes for viewers and listeners for the future.

Let’s be clear, the move to an internet-only future won’t happen overnight – and nor would we want it to – we’ve always been clear about the ��’s commitment to serve audiences using broadcast and internet services for the duration of its 11 year Royal Charter. But it’s not that far away. I want to ensure that the �� takes an audience centric approach – working with partners and the wider industry together to help secure the IP future this country needs.

Continuing the ��'s experiments in UHD

2018-04-23T11:31:26+00:00

From the introduction of colour TV on �� Two for Wimbledon in 1966, to championing free-to-air HDTV for terrestrial viewers, we’ve always been working to improve picture quality for audiences.

Lately, our experiments have seen us make select shows available to watch in Ultra-High Definition (UHD) on �� iPlayer. So far, we’ve focused on providing on-demand programmes in UHD, but now we’re taking the next step by streaming live UHD content.

On Sunday, we streamed a full Rugby League match between the York City Knights and Catalans Dragons live in UHD and High Dynamic Range on �� iPlayer Beta. It’s the first time we’ve streamed a live event in such high quality.

For those unfamiliar with the terminology, UHD takes advantage of higher resolution TVs to provide exceptional clarity of picture, while HDR takes this further with greater detail in the shadows and dark areas of the picture, as well as more natural and brighter highlights and mirror-like reflections.

Here at the ��, we use a type of HDR called Hybrid Log-Gamma (HLG), invented by �� Research & Development and Japanese broadcaster, NHK. Due to the way HLG represents light, audiences get the most natural colours possible on their televisions at home. In our last experiment, which saw the whole series of Blue Planet II on �� iPlayer in UHD HDR, this meant that the spectacular animals, vibrant coral reefs and deep blue oceans were brought to life like never before.

The �� chooses to use HLG for �� iPlayer because it fits very well and reliably into existing live production workflows. It also provides a great picture not only to HDR devices, but to standard dynamic range (SDR) devices that support the BT.2020 wide colour gamut as well. Being able to support both HDR and SDR TVs with the same bitstream is cost-effective for the �� and makes the content available to as wide a range of devices as possible.

However, making the move from on-demand UHD programmes to live UHD footage presents significant engineering challenges for our teams. High quality real-time HEVC encoding of UHD is still in its infancy, and is much more demanding for live programmes than for on-demand. That means we need to use higher distribution bitrates, and that people’s TVs will need to work harder to show the best picture.

We’re always looking to provide our audiences with the best possible viewing experience, and live UHD could be the next step in that journey. We’re now looking at the results from this latest experiment to help us build our understanding for how we might be able to provide live events in UHD in the future.

Enabling Secure HTTP for �� Online - audio and video

2017-12-18T10:19:00+00:00

Last year, Paul Tweedy talked about the challenges faced when enabling HTTPS for �� Online. Once our Online Technology Group had paved the way for �� web pages to be served over HTTPS, the next step was to do the same for our Audio/Video (AV) media distribution.

Before we go into more detail, many have asked on other blog posts about where exactly HTTPS is as most of the �� website is still only available on HTTP. Paul’s blog post explained that our platform is now capable of HTTPS, but it is up to individual product teams such as ��page, News and iPlayer to enable it for their websites. Many product teams have investigated enabling HTTPS but either are yet to prioritise the necessary work or have decided that there were too many limitations up to now. The biggest limitation for many sites has been that they play AV content, and this would still be HTTP (or even its predecessor, RTMP) — breaking the “secure” padlock in browser address bars when using the Flash player, and leaving us unable to use the HTML5 player on HTTPS at all.

Enabling HTTPS media delivery, as well as the increasing number of products with personalisation at their core which require you to sign in with a �� iD, will be big drivers products moving over. Since the end of March 2017, we’ve been able to offer HTTPS everywhere it’s needed and over the last few weeks www.bbc.co.uk/iplayer has started upgrading users to HTTPS.

The AV HTTPS project was broken into phases:

Enabling HTTPS at the CDN edge
Rolling out HTTPS where it is preventing use of the HTML5 player (SMP)
Rolling out HTTPS where Apple ATS was going to require it
Enabling HTTPS to our origin AV services
A/B testing HTTPS in our HTML5 player

When we started, our streaming services looked like this:

Enabling HTTPS at the CDN edge

We currently use up to three CDNs to distribute AV content — two 3rd party and our own, BIDI. Our 3rd party providers both “support” HTTPS, so we expected this to be straightforward. BIDI required some development work, but as ultimately it’s nginx under the hood, it also wasn’t expected to be too complex.

Actually being ready to make our first HTTPS request for a media asset took around four months.

We took the decision with BIDI to only use Elliptic Curve Cryptography, and whilst this limits supportable platforms for our in-house CDN, we thought it was not worth the complexity of supporting older technologies, specifically RSA, and to rely on our 3rd party CDN providers instead for these users, which are currently around 5% of our total. Our core media players for most platforms have automatic failover between CDNs when a problem with one is encountered.

The �� had to migrate all of its AV distribution to new hostnames, and in the case of one CDN, an entirely new product adding the additional complexity of needing to rebuild and retest the configuration.

New hostnames and a new platform meant that our CDNs would have cold caches. In normal usage the majority (>97%) of requests for media are served straight from their caches, which is something we rely upon. Once moved to the new configurations, all of the requests would have to come back to our origin cache, Radix, and then potentially to the canonical dynamic AV packaging origin. We couldn’t just flip a switch and have a hundreds of gigabits per second of user traffic suddenly come straight back to our packaging origin; nor did we want to do a big bang move in the case of the new CDN product. So we broke our distribution configuration into 40 chunks, for example DASH iPlayer on desktop for CDN A, and migrated one chunk per deployment window. Including cases where we had to roll back and forward as we found issues, this part took three months.

Rolling out HTTPS

At this stage, we turned on HTTPS for our �� Worldwide commercial offerings — as HTTPS only products we had to serve them using our Flash player until this point. We were also rushing to meet Apple’s deadline of requiring HTTPS for all traffic from apps in its App Store — but when this deadline was extended, we backed off a little.

Enabling HTTPS to our origins

Now we had HTTPS available at the CDN edge, but we then used HTTP between the CDNs and our on-demand (OD) origins, known as “scheme downgrade”. This was a necessary step for us to meet the initial timescales — the same team looks after both BIDI and Radix, with BIDI being the priority enabling HTTPS on Radix was left until later. We moved those over a period of a few weeks, again in small parts to prevent overwhelming our services from cold caches.

A/B Testing in the HTML5 player

By now, HTML5 was our default player for most browsers, so it made up the majority of our desktop AV traffic. We added a new dial we control to the player, enabling it to prefer HTTPS, even when embedded on an HTTP page. This gave us a few benefits:

Starting to warm up the CDNs to HTTPS, so we could see they performed as expected. At the start of this project, HTTPS distribution capacity with our CDNs was a concern.
Analysis of client performance compared to HTTP.

HTTPS preference started at 1%, and we gradually ramped it up so that by the end of March, we were running at preferring HTTPS for 50% of playback sessions, although the actual split of playback sessions was closer to 55% HTTP, 45% HTTPS. A couple of weeks of tweaking, and we saw global MPEG-DASH playback sessions split at around 50%.

Next step was to analyse anonymous performance data we receive from the player. There were a few hypotheses about how HTTPS delivery would perform:

High latency connections would experience elevated error rates due to the multiple TCP round-trips required to establish a TLS (<1.3) connection
Certain UK mobile ISPs could see reduced error rates as they have previously known to intentionally alter the content served, sometimes breaking it
Conversely, these mobile ISPs are also often high-latency connections, and also have transparent carrier-grade proxies, so the reduced cacheability of content could degrade performance
As a result of the increased number of round-trips, average bitrate would reduce slightly and rebuffering events would increase slightly
Also conversely, the increased integrity could result in fewer media download errors, thus reducing rebuffering
Reduced error rates in countries with restricted access to the internet

In summary, we weren’t entirely sure what we to expect.

“High latency connections would experience elevated error rates due to the multiple TCP round-trips required to establish a TLS (<1.3) connection”

Proving this one in itself is difficult. However, CloudFlare have previously talked about the challenges in Australia, which is the country which had the highest HTTPS error rate in our trial.

“Certain UK mobile ISPs could see reduced error rates as they have previously known to intentionally alter the content served, sometimes breaking it”

One UK mobile ISP, which we have observed performing incredibly novel alterations to our content in the past, experienced a 30% reduction in errors for clients using HTTPS. In fact, every major UK mobile ISP had a reduction in errors for HTTPS clients, although most to a lesser degree.

“As a result of the increased round-trips, average bitrate would reduce slightly and rebuffering events would increase slightly. Conversely, the increased integrity could result in fewer media download errors, thus reducing rebuffering”

This has proven true overall — whilst there are fewer media download errors, overall rebuffering has increased and average bitrate has dropped, although only by a small fraction.

“Reduced error rates in countries with restricted access to the internet”

Countries such as China, Russia, Saudi Arabia and Turkey are seeing reduced error rates when using HTTPS.

Other observations

The impact on fixed-line broadband services varies quite substantially from one ISP to another. Some perform better, others worse.
Traffic that looks like it could potentially be proxy, VPN or small “one-man” ISP services has a substantial increase in failures with HTTPS. This is possibly due to poor TLS performance and configuration in these networks.

A note on pre-HTTP streaming technologies

Some of our older AV content is only available on web in a streaming format called RTMP. Unfortunately, this can’t easily be moved to HTTPS, as it’s not an HTTP-based protocol in the first place. To resolve this we are currently in the early stages of a huge undertaking to re-encode as much of our archive that we have source for. In the meantime, on some �� HTTPS sites, some older clips may present an error message when you try to play them, warning that the green ‘secure’ lock in your browser will be broken when you play the clip.

Summary

The �� now uses HTTPS 50% of the time when delivering media to HTML5 responsive web users on HTTP pages. In addition to that:

iPlayer on mobile devices and responsive web is now HTTPS-only
iPlayer Radio on the iOS app is now HTTPS-only
Around 10% of IPTV iPlayer traffic is HTTPS
Other responsive web services such as �� Music have moved to HTTPS, with more to follow
All World 2020 �� News sites (e.g. www.bbc.com/amharic) are launching HTTPS-only

We hope that this work will enable more �� products to finally take the leap into enabling HTTPS for their parts of �� Online. However there are still other concerns, most notably that where we do use HTTPS, we use modern cipher suites and protocols to ensure that when we say it’s secure, it actually is. Large amounts of traffic, especially when you consider World Service news sites such as www.bbc.com/pashto need to balance access to information on cheap or old devices and browsers with the desire for the security HTTPS offers. For example, 5% of our Pashto users currently wouldn’t be able to support TLS 1.2, a standard that has existed since 2008.

Next, we’re looking towards making HTTPS AV perform as well or better, in the average case, when compared to HTTP; or to explain which market area is performing worse and why. We’re currently testing HTTP/2 for media distribution as well. Whilst previously �� R&D’s HTTP/2 experiment suggested h2 performed worse for AV, the technology has changed and we believe that at the very least changes to the minimum size HPACK (compression of HTTP headers in HTTP/2) operates on could reduce compression/decompression overhead and have some beneficial impact on performance. So far our testing is in line with this.

Testing disks: Lessons from our odyssey selecting replacement SSDs

2017-10-19T09:58:00+00:00

Elliot Thomas is a Software Engineer in the �� Design & Engineering - Media Distribution team. He explains the journey his team undertook towards understanding the detail of what's important when evaluating SSDs for stability and performance.

As part of the media distribution chain, we operate a rather large HTTP cache known as Radix. Its purpose is to act as a uniform origin (or root, hence radix) for client-facing HTTP distribution systems. With new streaming formats and ever increasing bitrates putting strain on the existing storage capacity, these systems do require upgrades from time to time.

Radix servers are grouped together in 'pools', with requests for certain media sets sharded across them, for instance, streaming audio may go to pool 3, while progressive downloads may be serviced by pool 1.

Each of these disks will be written to and read from almost constantly, so it's critical that the disks have good I/O performance lest they become a bottleneck in our system.

With these requirements in mind, we do a fair amount of research into the disks we buy. Once we find a promising disk, we run a fairly comprehensive set of block-level tests using fio, consisting of both sequential and random asynchronous reads and writes straight to the disk. Then we throw a few timed runs of the venerable dd program at it. If the results of these tests look good, we'll acquire enough disks to do an application load test with Gatling against Radix, which will typically be run for an hour or so.

For one of these candidates, the results of these initial tests seemed good - little regression on our current disks but with double the capacity. So we selected what will be referred to as disk 'B', a 1TB disk to replace the current 500GB disk 'A'.

Having settled on B as a viable replacement, we went ahead and ordered a bunch for Radix. Before fitting all our servers with the new disks however, we do 'canary' deployments - a trial on a subset of our servers - as our last line of defence against bad decisions. We did a single server, and then a whole Radix 'pool', and left it running for about a week without any observable negative impact.

With canary deployments appearing fine, we proceeded to deploy to another pool.

When we deployed to the second pool, everything looked great initially - increased cache efficiency was really taking the strain off our machines.

Unfortunately, within 4 days we had massive latencies accompanied by sporadic reboots.

So, what went wrong?

Well, for starters, there were a few flaws in our selection tests. Notably, block-level tests don't measure performance when running on a filesystem (the performance of which is not necessarily independent of the disk), and our application load test was configured to simulate a 95% cache hit rate (close to what we observe on BIDI - a similar, audience-facing cache) which is far higher than what we see on Radix - meaning that the test is heavily biased towards disk reads. This isn't a bad thing for a representative load test - it's somewhat realistic behaviour, particularly for a cache, but it does mean the test needs to run a while to actually write a substantial amount of data.

Furthermore, our application load test goes through a layer of read/write caching provided by the operating system. Most systems have a notion of a page cache, which is where parts of a file are stored in otherwise unused memory 'pages' (typically 4KiB) after being fetched from the disk. When these pages are written to, alterations accumulate and persist in memory for some time before being committed back to the disk. This is great - it reduces latency and number of I/O operations, but in this case, I/O operations are exactly what we want!

However, the bigger problem was arguably our canary deployment. We had deployed to pool 1, a fairly low traffic pool serving legacy content (and some other bits, like thumbnails), but the second pool (which we imaginatively call 'pool 2') at the time served all forms of HTTP streaming video and audio, which accounts for an awful lot more of our traffic.

After a few days of progressively worsening latencies and reboots on pool 2, we put into action our alternative plan, which was to reinstate disk A and to use two of our staging machines to build a 'pool 3', which now takes on some of pool 2's working set. This resolves our immediate problem, but we still have a set of ageing disks on machines that need more storage capacity.

So what really went wrong?

The above mentions flaws in the evaluation process, and a flaw in the way we do canary deployments. But that doesn't explain what actually went wrong. The only thing that changed with the deployment was the disks, so why didn't the disks work well with our systems?

The first thing we did was to run the load test again. The results were not good. It would appear either the initial load test went wrong or the behaviour of disk B had changed since our initial test. Further investigation showed that after writing 4 tebibytes worth of data, performance hit the floor (a phenomenon we still don't fully understand).

It was at this point we decided to focus on testing just the disks, so the original set of tests that selected disk A were dug up and run against disk B. These tests were used at a time we had a very different system, one that has since changed somewhat, but this shouldn't matter for tests that operate outside our software stack. IOzone was the test tool used - an excellent filesystem benchmarking tool which runs a comprehensive set of read and write tests. Running this and graphing the results showed that disk B was roughly on par with disk A in terms of reading, but around 5 times slower at writing things. Perhaps unsurprisingly, data obtained when running IOzone against disk A more or less matched what was written on the data sheet.

Eventually however, the 'okay, but what is it doing?' questions started cropping up, and trying to reason with a file containing 25,000 lines of unfamiliar C, laden with different IO methods, threads and support for many different Unixlikes (and all the #ifdef's that come with that) can get quite difficult.

So I wrote a deliberately dumb tool called "wsc" - it stands for write, sync & close, and that's more or less exactly what it did; that and timing these operations.

This new testing tool was configured to time how long it took to write, sync and close one gibibyte using various record sizes (amount of data sent to the write syscall), and was run against both the old and new disks. These tests showed that in the best case, one gibibyte could be written to disk A in about 2.1 seconds (487MiB/s), whereas disk B took 11.2 seconds (91MiB/s). Testing with and without the O_DIRECT flag revealed that the problem really was the action of writing to the disk, and short of delving into the Linux kernel code (which was briefly considered) we couldn't be any more sure.

This is where we started considering all the knobs and levers we could twist and pull, bearing in mind disk-related properties, such as flash cell geometry, the behaviour of our RAID controller, and any on-controller or on-disk cache. We also started looking at disk C, the "enterprise version" of disk B.

Before we go on, it's worth having a quick overview on how an operating system interacts with a disk, and how SSDs in particular work. This largely reflects the knowledge gained/reaffirmed from the research we did while testing. If you're familiar with the ins and outs of SSDs and flash memory, and just want the story, feel free to skip to the section below.

So, how does a computer talk to a disk anyway?

The OS will talk to a device called a host bus adapter (HBA), and ask it to send certain commands to a disk somehow (there are standards for this part such as AHCI, but this is not important). This adapter will be connected to a disk controller, the host-facing part of the disk, and talk a certain command set - which is essentially no different from a language - over some kind of physical interface. The command set is usually either ATA (AT Attachment) or SCSI (Small Computer Systems Interface), and the physical interface these days tends to be either SATA (Serial ATA) or SAS (Serially Attached SCSI). Our disks are SATA, connected to a SAS HBA that supports SATA devices.

With these command sets, sections of storage are addressed by a logical block. Logical Block Addressing (LBA) is a linear addressing scheme, much like regular computer memory. In both command sets, the size of a logical block has typically been 512 bytes, however, modern versions of both command set specify ways for the disk to communicate a different value - 4096 byte logical blocks are becoming common on "Advanced Format" disks.

There may be an additional component between the HBA and disk. Some systems may have a piece of hardware called a RAID controller (sometimes disk array controller) with RAID standing for 'Redundant Array of Inexpensive Disks' (or, if you're a disk manufacturer, independent). Hardware RAID controllers essentially emulate larger and/or fault tolerant disks, using actual disks as a backing store. Often RAID controllers are integrated with HBAs.

What's an SSD, anyway?

'SSD' is an initialism that expands to Solid State Drive.

An SSD contains no moving parts, and despite being referred to as disks in this blog post, aren't actually disk shaped at all. SSDs are typically composed of a bunch of NAND flash memory cells connected to a flash memory controller (or SSD controller). The flash memory controller fills the role of a disk controller and is what translates command set instructions into actual operations on the flash memory.

What's a NAND?

Flash memory comes in many flavours. When we talk about flash memory, we deal with units known as cells which can be NAND or NOR (whether the floating-gate transistors that make up memory cells are wired up in a way that resembles a NAND gate or a NOR gate), single level or multi-level (SLC or MLC, essentially a measure of bits crammed into each floating-gate transistor in the cell). Most SSDs these days are NAND MLCs, but, while interesting, this is all information we don't need to know.

What we do need to know is that in NAND flash memory, data is addressed and read in pages (a smallish quantity of data, often kibibytes). However, writing is asymmetric - the exact process depends on the value being set. Writing zeros has a finer granularity than writing ones*. In flash memory parlance, writing zeros is called programming and can be done on a bit-by-bit basis (by selecting specific bits in a page). Writing ones is erasing, which is an operation applied to the whole of an erase block** (a larger quantity of data, often mebibytes).

The consequence of this is that changing a one to a zero is easy, but changing a zero to a one requires erasing a few mebibytes and rewriting all the old zeroes bar the ones you don't want back. This is called write amplification.

Ok, so you need a little memory somewhere to do this, why is this important?

Because erase blocks stop working after a number of program-erase cycles. It stands to reason that a simple 1:1 mapping of logical block ranges to erase blocks would cause some erase blocks to fail sooner, as not all data on a disk is rewritten constantly (think of your 'Downloads' folder verses a collection of pictures from 2012). The erase process is also comparatively slow.

Because of this, most SSDs do some kind of wear levelling - distributing utilisation over all the available erase blocks in order to extend the lifetime of the disk. SSDs tend to be black boxes in this respect, and there are more than a few kinds, e.g. static and dynamic wear levelling. There are also different approaches to this problem, such as compression and block deduplication, which can be used alongside wear levelling to reduce amount of data written. The precise details of these techniques are seldom revealed by disk manufacturers and as a result are still largely magic to us.

What's a TRIM?

ATA TRIM (or, for SCSI speakers, UNMAP) is a command you can send to a disk to mark a particular logical block as unused.

But disks don't think in terms of used and unused space!

This is correct. Disks have a fixed size and every bit on them is either 1 or 0. It's up to the filesystem to decide whether some part of the disk holds data - when deleting a file, most filesystems simply mark the space occupied by that file as unused in their own data structures, overwriting it when the space is allocated for something else.

However, with flash memory, the program/erase disparity (in both write granularity and time taken) means there are potential optimisations if programmed pages are known to be discardable. For instance, if the disk knew certain blocks or pages were unused it might not bother programming those zeros back when the erase block they are in gets erased, or a disk might automatically erase blocks consisting of nothing but unused logical blocks when the disk is idle (garbage collection). This is where information from TRIM commands comes in useful.

TRIM is simply a source of additional information that might allow the disk to manage these better. In short, TRIM might speed up block erasure, future block allocation and increase longevity of the disk.

Is there anything else SSD related?

Yes. Over-provisioning and drive write-back cache.

What is over-provisioning?

Over-provisioning is where the disk is technically slightly larger than advertised, and that additional portion of cells are used as 'spares', to be mapped in and out to spread out writes further. This should improve disk lifetime and possibly latency to a point, as it allows the unmapped cells to be erased if needed.

What is a drive write-back cache?

Most disks (SSDs or the spinny kind) have their own volatile memory for use as a buffer or cache.

On spinning disks, this memory size is often somewhere in the range of 8MiB to 64MiB, and is often as a read cache. On SSDs, this memory is seldom mentioned - I couldn't find many numbers on spec sheets, but some of the fancier SSDs seem to come with 512MiB or even 4GiB for the really large ones. On both kinds of disk, you can often use this memory as a write-back cache.

A write-back cache aggregates write operations, flushing data to its backing store either periodically or when running out of space. If this sounds like the page cache mentioned earlier, then that's because the page cache is a kind of write-back cache.

When enabled on a disk, write operations return as soon as data is written to cache, rather than the backing store. This is great for latency. There are other benefits, on spinning disks, a write-back cache (along with command queue reordering) can reduce the total number of seeks needed (saving time and reducing mechanical wear), and on an SSD, this can reduce the total number of program/erase cycles needed (saving time and reducing flash cell wear).

The downside of using a write-back cache on a disk is that if there's is a power failure, disks may lose data the OS thought was written. In our use case, data loss isn't the end of the world - we can just pull the data from our upstream again.

What about filesystems?

Filesystems are a very important part of the whole stack. Many filesystems have tuneable parameters to optimise for specific disk layouts and RAID configurations, and EXT4 (the filesystem we use) is no exception.

With spinning disks, it makes sense to contain contiguous data on the same track - this minimises seek time and wear. With RAID arrays, it makes sense to stripe based on the size of a track for similar reasons. SSDs don't have tracks, but the potential optimisations for RAID arrays map quite nicely to some SSD concepts, namely pages and erase blocks.

EXT4 happens to support two tuneables for this, stride and stripe width, which affect filesystem block allocation.

Stride controls how data is read from the disk. It's intended to match the amount of data a RAID array will read or write to a single disk before moving to the next.

Stripe width controls how writes are dispatched. It's intended to match the amount of data across one stripe of a RAID array. The idea being that when writing this amount, you can minimise parity recalculations and rewrites.

The stride concept maps well to both pages and erase blocks when you consider it as being "the smallest unit of data to write", and stripe width maps very well to erase block size, when you consider its purpose. By batching up writes for certain files to fit neatly in an erase block, you make it less likely that the erase block used will be erased and reprogrammed for a subsequent write.

These attributes are set upon filesystem creation, and the values used if left unspecified depend on the reported geometry of the disk (assuming you are using mke2fs from e2fsprogs).

How did you find all this out?

Mostly searching the web and discussion with colleagues. There was some source code diving of e2fsprogs to verify certain behaviour, and the odd manual filesystem inspection (I have a newfound appreciation for the od program).

The venerable Wikipedia has an excellent set of articles on the operations of flash memory (somewhat technical), the subject of write amplification, TRIM) and disk buffers - also worth mentioning is the article on caching in general).

For a more explained but no less excellent article on the operation of flash memory, see the splendid Ars Technica article on SSDs, along with the Ars Technica article on TRIM for a full view of the picture.

For some detailed information regarding EXT4, the EXT4 wiki page on disk layout is an excellent resource, and for a more gentle, if opinionated, introduction to the stride/stripe options and how to set them (we used their equations), see this wonderful blog post by 'thepadawan42'. A more authoritative source can be found in the blog of Theodore Ts'o (he's one of the developers and maintainers of EXT4).

Caveat lector, do bear in mind that some of the tweaks and features (such as stride/stripe and write-back cache) can reduce performance as well as increase it. As mentioned earlier, SSDs are black boxes and what might theoretically be good for them can turn out to work badly in practice due to some hidden implementation detail - this is behaviour we have observed. There is no substitute for structured testing!

For legacy reasons, our disks are connected to a HBA in RAID controller mode, with a 2-disk RAID 1 array for the system disks, and many single disk RAID 0 arrays.

Each of the 'RAID 0 arrays' has an MBR partition table with a single partition spanning most of the disk, formatted as EXT4 with a different inode ratio (64KiB per inode to be exact).

By this point, we had ruled out buying more of disk B (although we still had a lot of them), and were looking at disks C and D.

We looked at many things:

Whether the amount of over-provisioned space could have an impact.
Whether enabling drive write-back cache could improve things.
Whether the filesystem partition alignment could have an effect.***
Whether setting the stride and stripe width properties of EXT4 made a difference.***
Whether setting the stripe size property of the single disk arrays made a difference.***
Whether our application load tests were even fair on all candidates.

We also looked up whether our HBA used TRIM in its current configuration. It turns out, in RAID mode, the HBA did not support TRIM. We did do some trim-enabled testing with a different machine, but these results are hard to compare fairly. In any case, we can't currently enable TRIM on our production systems.

Disk C supported altering the amount of over-provisioning it provides, however changing this didn't seem to do much.

"wsc" based tests showed considerable improvements adjusting stride and stripe related settings, with disk A showing very little difference between configurations, but with other disks having different sweet-spots and bad configurations. Application load tests didn't show a great deal of difference, and in disk D made things worse.

But crucially, enabling drive write-back cache seemingly had next to no impact on disks A and D, but improved disks B and C substantially. To the point where we are using our current stock of disk B in production machines now.

And perhaps one of the biggest realisations for us: our application load test wasn't fair.

...?

To elaborate, our application load test typically consists of three or four machines. One will be the system under test (or "victim"), one or two of them will be load generators, and the third will be a mock origin. The mock origin we had been testing with simply returns files consisting entirely of dots.

Now, we don't do HTTP compression (not during load tests, anyway) and our cache doesn't compress either, so the fact that a file consisting of N dots is highly compressible doesn't matter for most of the chain.

Now recall that some approaches to write amplification include compression and block deduplication...

It just so happens that disk A does these things, probably turning several gibibytes of writes into a few kibibytes of flash usage. The other disks tested do not appear to do these tricks.

After replacing the mock origin with something that produced data that looked truly random (this was measured with the nifty ENT tool), the incredible load test results from disk A started looking a lot like the other disks. The difference is remarkable - for 1.7MiB media segment, we had one test report a mean response time of 918ms, which increased to 3183ms when random data was used. Additionally, write-back cache also seemed to have an impact on disk A with random data, with mean response times dropping to 2035ms.

Investigative outcomes

We learned a few things:

We should run canaries for longer and cover all different traffic profiles.

How long is longer? Who knows! We've settled on a couple of weeks. It's important to acknowledge that we can't know exactly how long.

We should test with and without disk caches.

As a team developing and maintaining caching infrastructure, we should know that caches are important. It seems that the impact of write-back cache can vary and any new disk should be tested with it both enabled and disabled.

There is no silver bullet to testing.

Hardware can be unpredictable so keep an eye on things.

Oh, and somewhat representative load tests are important.

Even for testing disks.

Disk A, the unassailable incumbent, was found wanting on a more level playing field.

* Whether it's a 1 or a 0 that's easier to write depends on the logic level used, logical high (where a high voltage equals 1 and a low voltage equals 0) is probably the most common, and the convention used here.

** Note that the term block in the context of flash memory should not be confused with the logical block in logical block addressing, which is how data is addressed by the host.

*** Frustratingly, there were some things we couldn't verify, at least not with the way we work, namely the actual filesystem partition alignment of our disks. The RAID controllers we use reserve some disk space for metadata. We don't know where on the disk that is and whether it pushes all of our partition/stride/stripe settings out of alignment.

Balancing, spikes and speed: Architecting media distribution cloud services

2017-02-14T11:04:00+00:00

Senior Software Engineer Henry Rodrick is part of the �� Design & Engineering - Media Distribution team. He describes how the team tackled the architectural challenges faced when building a microservice with strict non-functional requirements.

Cloud technologies have radically changed how most organizations develop and deploy software, and the �� is no exception. Within the ��, we’re using the AWS (Amazon Web Services) platform a great deal, for instance the CloudFormation infrastructure management tool and our own AWS based deployment system Cosmos.

While CloudFormation enables developers to programmatically define their cloud infrastructure, it doesn’t really provide a great way to deal with actual software deployments. Cosmos solves this problem by automating the process of building release images (AMIs) and providing an easy-to-use interface where releases can be advanced through the different stages of deployment (integration, test and live).

This is a great improvement over the ��’s traditional deployment cycle, which generally requires an operations engineer to carry out configuration and installation on in-house servers. Because of this, many teams have now moved to Cosmos for the majority of their development.

In the Media Distribution team, our main systems are content caching servers, known as caching cells, that make up the BIDI Content Distribution Network (CDN). For these systems, we still rely heavily on physical machine deployments to meet our performance requirements. The caching cells also need to be deployed in certain geographical locations, meaning that a cloud based architecture restricted to Amazon’s data centres wouldn’t work, even if the performance were good enough.

Not all of our services in BIDI have to be deployed like that, however. The “brain” of the platform, which deals with traffic management and configuration, is a suite of micro services which aren’t tied to any particular data centre. For these services, AWS/Cosmos promises faster deployments and less operational overhead than our internal deployment platforms. It’s often cheaper in the end as well, even though AWS’ fine-grained pricing model makes it hard to quickly determine whether that’s going to be the case.

One of the BIDI related services that we decided to deploy in the cloud is called Croupier and is used when making decisions about which CDN to redirect �� iPlayer clients to. Croupier is a lookup service which provides an HTTP GET endpoint where the iPlayer client’s IP is a parameter. The service is internal: it is only used by another (AWS based) �� service called Media Selector, which is called by iPlayer before the user actually gets to see any video.

The performance of Croupier directly affects Media Selector response times and thus iPlayer user experience as a whole. In addition, iPlayer traffic varies a lot over the day and we don’t want user experience to degrade in the evening when massive peak hour traffic kicks in, for instance.

Because of this, some strict performance and security requirements had to be considered when architecting this service:

Croupier should have a mean response time < 50ms. Requests are considered failures if they take more than 100ms. Only a small number of such failures is tolerated since 99.5% availability is required.
The service needs to be able to cope with an average of 3,000 requests per second, with point-in-time spikes up to 14,000 requests per second.
Deployments of new Croupier versions should not cause any downtime.
Clients must use a specific �� issued SSL/TLS certificate to connect to the service.

Architecting for high availability and zero-downtime deployments

There’s a large number of architectural options available through AWS. And even though “serverless” architectures such as AWS Lambda now exist, it is a bit of a paradigm shift from what most developers are used to.

For example, the type of IP lookups performed by Croupier can be implemented efficiently using a read-only, in-memory data structure, which can be generated once and reused by subsequent lookups (and even from multiple threads). It’s not directly clear if it’s possible to translate this to efficient Lambda code, since no such state sharing is possible.

Another problem with Lambda is that, as with much of AWS, you get a great set of building blocks but not necessarily the tooling. Normally Cosmos would provide this, but its Lambda support was still in development when work on Croupier started in March 2016.

Because of these concerns, we settled for a more traditional architecture; basically a cluster of near-identical servers sitting behind some kind of load balancer. In AWS speak this means using Elastic Compute Cloud (EC2) instances, managed by an Auto-Scaling Group (ASG) and load balanced using the Elastic Load Balancer (ELB, also known as the Classic Load Balancer).

So what about the availability and deployment requirements in this setup?

The Auto-Scaling Group came in handy in multiple ways, but perhaps not in the way suggested by its name.
First of all, the ASG used together with the ELB makes up a pretty nice automatic instance manager that terminates and re-spawns instances based on custom health checks.

For Croupier, we simply defined the ASG MinSize to make sure that we always have enough server capacity to deal with our expected load (based on load test results). An important step here was to implement the health check in a way that actually tests whether real Croupier lookups can be performed – a simple ping-like dummy endpoint would potentially report the instance as healthy even when it isn’t.

To make sure that the capacity is maintained without any downtime when deploying new versions of Croupier, we used the ASG’s AutoScalingRollingUpdate update policy. This policy provides a few useful parameters such as MinInstancesInService and UpdateMaxBatchSize, which allows us to replace instances one by one while maintaining our required number of healthy instances (A caveat is that the ASG’s MaxSize parameter must be set to at least MinInstancesInService + UpdateMaxBatchSize, otherwise the rolling update will fail because of conflicting policies).

Croupier at scale

In theory, the “auto-scaling” aspect of ASG sounds great: Instances are automatically provisioned (and removed) based on some performance metric, such as CPU or memory usage, to make sure that there’s enough computational power without wasting money on idle instances. This might seem straightforward in theory, but as explained in many articles online (for instance the Netflix tech blog), it’s a bit of a double-edged sword. First of all it’s difficult to deal with sudden traffic spikes, but even with predictable “smooth” traffic patterns a lot of analysis and care is needed. It’s likely that you will end up with a policy that is really restrictive about scaling down (to avoid a disaster), which means that your ASG is over-provisioned most of the time.

For Croupier, we decided to avoid automatic scaling altogether. In addition to the general difficulties of getting the scaling policy right, the Croupier traffic patterns are not actually that smooth. Request rates are expected to sometimes spike from almost zero to thousands per second!

The compromise solution we ended up with is to use the so-called t2 class of EC2 instances. These instances have “burstable” performance, meaning that they can deal with shorter bursts of CPU usage, as long as the average CPU usage over time doesn’t go above a certain baseline.

By monitoring the performance of these instances using CloudWatch, we can tell if the traffic baseline has increased over time and manually scale up the size of the ASG.

ELBs and TLS termination

The Elastic Load Balancer has two possible modes of operation: HTTP and TCP. Initially, HTTP mode looked like the better option for Croupier since the ELB would have the obvious advantage of knowing what’s logically happening over the wire, as opposed to just distributing TCP connections across a number of EC2 instances. In HTTP there would also be the advantage of built-in HTTPS termination.

Unfortunately, the TLS termination requirements for Croupier made it impossible to use HTTP mode. HTTPS configuration for ELB is fairly basic and doesn’t provide a way to perform the issuer verification we’re required to do (allowing certain �� certificates only). Because of this, we had to configure the ELB to operate in TCP mode and add custom TLS termination on the EC2 instances.

Our options here were either to implement HTTPS directly in the Croupier server software (a Java application) or put it behind a HTTPS capable web server such as Apache or Nginx using reverse proxying. In the end, we decided to terminate HTTPS traffic through Apache because a lot of �� specific configuration to deal with certificate verification and revocation was readily available.

Latency turned out to be the biggest issue with this solution at first. Apache out of the box is not optimised for long HTTP keep-alives (it actually seems to be optimised to mitigate memory leaks from things like bad old PHP scripts by closing connections often). Opening new connections results in extra TLS handshakes, which in turn require extra round trips causing latency spikes. Fortunately, most web servers (including Apache) allow you to monitor and configure keep-alive handling and other relevant things, but be prepared to spend some time load testing and tuning!

Latency spikes caused by load test clients simultaneously reconnecting, before tuning Apache.

The not-so-elastic load balancer

Similar to ASG, the Elastic Load Balancer attempts to make full use of the dynamic nature of the cloud and provision computational resources as needed. In this case, it means the size of the load balancer itself (how ELBs work under the hood is not publicly documented, but “size” in this context basically means how much traffic they can handle).

One important thing we learned about ELBs is that they start out in a default state, but are actually allowed to scale down to some minimum size if idle. This state turned out to have a huge impact on request latency as seen in the following load test graph (the majority of requests failing by definition, by taking longer than 100ms).

Load test performed using a “cold” ELB instance.

Amazon’s guide to load testing with ELBs suggests ramping up traffic slowly. However in our case, we really had to warm up first before testing and going live since the cold ELB wasn’t good enough (in latency terms) for any amount of traffic at all.

It’s possible to ask AWS support to “pre-warm” your ELBs to avoid scaling down between tests. The way we did this was by running load tests (simulating peak traffic) and then tell support at what time the ELB reached the warmed-up state.

Unfortunately, there isn’t much monitoring available to tell you what ELBs are actually up to. However, ELB load balancing is essentially DNS based, so regular name server lookups (using a tool such as dig) will give you an idea of when the ELB’s underlying instances are being replaced.

17:44:46 GMT
croup-123.eu-west-1.elb.amazonaws.com.        60 IN A        52.18.101.104
croup-123.eu-west-1.elb.amazonaws.com.        60 IN A        52.48.98.248
17:45:46 GMT
croup-123.eu-west-1.elb.amazonaws.com.        60 IN A        52.211.160.183
croup-123.eu-west-1.elb.amazonaws.com.        60 IN A        54.171.121.7

The ELB IPs will change roughly every 15 minutes or so when scaling up, so if the DNS entry has stayed the same for 30 minutes it’s likely that the scaling has settled down (as suggested by our testing, at least – Amazon doesn’t provide a lot of information about this).

Wrapping up

AWS is a great toolbox for cloud development. That being said, the “elastic” features are not a magic solution to scaling and still requires a lot of careful tuning and testing, except perhaps for services with a very trivial workload.

The ELB is not a silver bullet for load balancing, and many teams at the �� rely on custom solutions instead (plain DNS load balancing through AWS elastic IPs and Route53 for example). The platform is constantly evolving however and a new Application Load Balancer was recently introduced by AWS, but we’re still waiting for a truly general solution to the problem.

BIDI: The �� Internet Distribution Infrastructure explained

2016-09-22T14:28:00+00:00

In his previous blog post, Product Manager Federico Benedetto described how �� on demand content reaches our audiences. In particular through Radix, the �� system that delivers media to the Content Distribution Networks (CDNs), which are in turn responsible for distributing it to the final users across the UK and internationally. Federico now turns to another piece of the puzzle in the ��'s media distribution stack: the “�� Internet Distribution Infrastructure”, or BIDI for short.

BIDI is the ��’s own content distribution platform and is responsible for delivering media directly to the �� end users in the UK. As depicted in the diagram below, the clients are always presented with a ‘CDN mix’ from which they can fetch the media they are interested in. BIDI is part of this mix, alongside the commercial CDNs.

This set-up allows us to provide a better overall quality of service, by enabling clients to move between different media distribution platforms if they encounter problems when streaming from any particular one.

BIDI consists of several distributed HTTP caches, referred to as Caching Cells and grouped in ‘Batteries’. The content is served directly from these to the end users.

In addition, there is a Control Plane that manages the fleet of physical caches, and a Matchmaker service, which steers the incoming traffic to the best available point of presence. Both these applications are hosted in the cloud.

The BIDI caches are based on the same software stack as Radix. In particular they are based on a distribution of the Nginx web server called OpenResty. We also use ExaBGP for network related tasks. For more information on why we are using Nginx, I suggest reading Alistair Wooldrige's blog post. If you are interested in knowing more about how BIDI works, there will be more specific posts from the OTG Media Distribution team in the coming months.

BIDI is currently in use on selected iPlayer platforms only, but chances are that if you are using iPlayer on iOS devices or in your desktop’s browser, you may have already been using it without realising.

Why BIDI?

The �� owns and operates a complex network and server infrastructure, which is crucial to the functioning of all the �� online services.

We have also been an active member of the London Internet Exchange (“LINX”) for many years, and we have a long history of working with Internet technologies.

For an overview of the Internet history at the ��, I highly recommend reading this blog post from our Chief Scientist, and explore this iWonder guide on the 20 years of the �� on the web.

BIDI builds upon this rich heritage, and marks an important milestone for the development of an Internet first ��. By building and owning the infrastructure that allows us to directly reach our audiences, we are making sure that we are preparing for a future when distribution over the Internet will become an even more essential part of media delivery than it is today.

BIDI also allows us to work in partnership with the UK Internet Service Providers community and interconnect with them in locations other than London. By doing so we hope to contribute to the development of a regional and resilient architecture of the British Internet. BIDI is in fact delivering traffic via London based peering, but we also have a presence at IX Manchester. Moreover, BIDI servers can also be deployed within 3rd party networks, to reduce potential traffic congestion over specific network paths.

Quality of service is another important driver for our efforts. By managing our Internet streaming technologies and controlling the distribution costs, we can easily monitor the performance of our streams, take immediate action when we notice a problem, and enable �� products to access higher bitrates and better quality streams. Deploying our servers near the edges of networks (i.e. closer to the end users) also improves quality by reducing the network hops. Therefore packet round-trip time (RTT) will also be reduced.

By relying on a mix of commercial CDNs and our decentralised infrastructure we can provide useful mitigations when dealing with problems over specific network paths or suppliers, and we can offer a better streaming performance to our audiences and highly available media endpoints.

From R&D to production

BIDI is a great example of a successful transfer of technology from research to production.

Initially conceived jointly by �� Research and Development (R&D) and the �� Online Technology Group (OTG), the first functioning prototype was developed by R&D. The development of the BIDI prototype was a fundamental step which demonstrated that the concept of building our own Internet distribution platform was a feasible and valuable goal.

The prototype was further developed by R&D to a point that it was ready to be used for production traffic. A live trial was therefore launched, with the goal of learning as much as possible from serving traffic to a very small set of real users. By doing so, the model was finally proved outside of a lab, on real networks, with real users and content.

Following the success of the BIDI trial, the OTG Media Distribution team, initially supported by the R&D team, took over the development of BIDI. Over the last year, we have managed to increase the BIDI traffic share and capacity, while transforming BIDI from a prototype to a fully supported, production quality platform.

Today BIDI runs on a very different software stack from the original R&D prototype, but the key concepts and ideas haven't changed.

The fruitful collaboration between production and research is still ongoing. Lessons learnt from operating BIDI at scale are now feeding into the R&D roadmap, which itself is experimenting with new concepts and technologies that will pave the way for future BIDI developments.

What's next for BIDI?

Despite being a fully supported platform, BIDI is still at the early stages of its development. We are constantly building new features, improving the algorithms behind the Control Plane decisions and tuning the performance and throughput that we can achieve with our servers.

We are not planning on distributing all the �� traffic via BIDI, as using a mix of CDNs helps us achieve our quality of service goals. Despite this, there is a vast growth potential for BIDI which we will soon tap into, by increasing our network capacity, our partnerships with the ISP community and the range of supported clients and protocols.

In particular, we are currently working on supporting HTTPS distribution, as mentioned by Paul Tweedy in his blog post.

In the near future we will also add support for MPEG DASH streams, so that we will be able to deliver media to the new HTML5 player and we will soon be supporting iPlayer on TV devices and set top boxes.

Want to find out more?

If you like the sound of what I have just described and you would like to play a part in it you can apply here. If you would like to know more about BIDI, feel free to write to us at bidi@bbc.co.uk or get in touch with the OTG Media Distribution group engineering manager Umesh Telang.

Developing our Standard Media Player for Android

2016-01-20T14:15:36+00:00

Software Engineer Matthew Mould explains the decision making process and strategy behind the build of the ��'s new Android Media Player.

Users of the ��’s Android mobile applications are familiar with the need to install the separate �� Media Player application to play multimedia content. This application uses Adobe's proprietary technology to play back media encoded in RTMP and HDS media formats. The �� has been moving towards an open-source streaming format called DASH (Dynamic Adaptive Streaming over HTTP), and I have been working in a team to develop a native Android component called Standard Media Player for Android (SMP-AN), which uses Google's open-source ExoPlayer to play back DASH content. This component will be integrated into the wide range of �� applications available via the Google Play store. It’s a more efficient technology for the �� to use, thanks to its use of open standards, and removes the need for users to install a separate application - something which has long been a source of complaints.

We built the player as an application component, with a codebase to which integrating clients (�� iPlayer, �� News, etc.) could either contribute, or use out of the box. In building this component, we emphasised comprehensible, robust, flexible and well-tested code. The following offers an insight into some of the architectural decisions that we made while building it.

Stability of API

A key factor in producing an easily usable software library is to maintain a stable API. We therefore began our design by creating an interface called SMP (Standard Media Player), which exposed all of the methods that we wanted integrating clients to access such as play, pause and seek. At a top level this interface is implemented by a facade class called SMPFacade, a thin class that delegates calls down to the appropriate classes.

Clients therefore require no specific knowledge of the underlying architecture of SMP and we retain the ability to separate our code into an appropriate class structure.

Model View Presenter

In broad terms, we wrote SMP following a Model View Presenter (MVP) pattern, with a passive view. Our model exists with no knowledge of the view and the view layer is, in effect, a client of SMP. An integrating application can therefore flexibly use a bespoke view layer to control content, and use the player to play either audio-video or audio-only content with no changes required at all. Our tests reflect this division, with detailed unit tests covering the core player functionality and a small number of very simple tests covering our thin UI layer.

The fact that we had such thin UI tests circumvented the pitfall of ‘flaky’ UI tests that developers usually end up ignoring, and in keeping the UI tests small in number, we avoided the ‘ice-cream cone of tests’ anti-pattern in favour of a ‘testing pyramid’.

Test Driven Development

We were strict in our policy of Test Driven Development (see Growing Object-Oriented Software Guided By Tests by Freeman and Pryce). At the beginning of the project, because domain concepts and appropriate levels of abstraction weren’t immediately obvious, we wrote tests over slightly inconsistent test boundaries and reshaped tests as the domain became more clear. This approach is similar to Ian’s Cooper’s discussion of using small tests to explore an approach before consolidating them into more logical units, or deleting those that become obsolete.

When developing a new feature later in the project, such as subtitle capability, we wrote tests from the outside in - writing a test that interacted with SMP as if it were a client. We then wrote the simplest code to get the test to pass, immediately committed (and pushed) the code, and then refactored safe in the knowledge that our tests would go red if we broke any behaviour. This wasn’t an infrequent occurrence, but because our commits were very small it was easy to revert and try again.

Often in the first stage of getting a test to go pass, we would write code in the aforementioned SMPFacade class, but during refactoring we would force a new domain concept to emerge to which calls would be delegated. This latter step maintained SMPFacade’s status as a facade class. For more details on the logic behind what we chose to change when refactoring, take a look at Kevin Rutherford’s blog posts on connascence.

Test objects and mocks

We mocked only the boundaries of our system, which was represented by an ever-shifting diagram on a white board next to our desks (see this article on ports and adapters), so that to the greatest possible extent our tests were independent of our production code, rather than being clamps around every class which had to be removed and reattached for each refactor. We avoided the use of mocking frameworks and instead created fake objects for our tests.

As part of this strategy, we were able to avoid setting up mock objects in each of the tests in which they were used and consolidated mock behaviour into single objects, making the tests more comprehensible. This also aided us in the avoidance of DRY violations, as such violations would present a nuisance early in the development process and demand our attention. For example, if the same three steps need to be in the setup of every test, this suggests that there should be a class for that. Occasionally we went as far as writing contract tests that asserted that both our mock objects and the real domain objects behaved in the same way, for example with our adapter for SharedUserPreferences.

public abstract class PersistenceContractTest extends AndroidTestCase {

   @Test
   public void testThatReadingAnUnsetBooleanReturnsFalse() {
       Persistence persister = getPersister();
       assertFalse(persister.readBoolean("unset_value"));
   }

   @Test
   public void testThatWeCanReadWhatWeCanWrite() {
       Persistence persister = getPersister();
       String key = "valid_key";
       boolean value = true;
       persister.writeBoolean(key, value);
       assertThat(persister.readBoolean(key), is(value));
   }
   abstract Persistence getPersister();
}

public class SharedPreferencesContractTest extends PersistenceContractTest {

   @Override
   Persistence getPersister() {

       SharedUserPreferences sharedUserPreferences = 
new SharedUserPreferences(getContext());
       sharedUserPreferences.clear();
       return sharedUserPreferences;
   }
}

public class FakePersistenceTest extends PersistenceContractTest {
     @Override
     Persistence getPersister() {
         return new FakePersister();
     }
}

There were some lessons that we learned as our tests grew, typically around aspects such as a large amount of setup being required for each test. It is entirely possible that our test boundaries were too large. For developers who have been on the project from the beginning this could be great: total freedom to refactor safely! However, when welcoming new developers to the project some have found the amount of setup required for our tests to be a hindrance.

In an attempt to mitigate this, we created helper classes such as test object builders that, for example, recorded the objects they were creating to allow the tests to interact with mocks to simulate real world behaviour. We also created a bank of test data builders that provided arbitrary test objects and have had interesting discussions on whether test data should actually have a random element in it.

The arguments in favour of this are that it would ensure that we do not rely upon ‘magic’ values in our code, so would be more likely to find edge cases early. The argument against it is that introducing a stochastic element to tests makes reproducibility more difficult. We haven’t introduced random test data, but also haven’t drawn a firm conclusion on this matter. Answers on a postcard.

Test naming convention

To make tests easier to consume for developers unfamiliar with the codebase, we attempted a naming convention that describes tests in terms of their context, with individual tests describing the action and the assert. This follows a similar concept presented in Kevlin Henney’s ‘Programming with GUTs’ talk. For example:

public class WhenVideoIsPaused () {
	public void setup() {
		…	
	}
	@Test public void
	andStopIsInvoked_PlaybackStops() {
		…
	}
}

State machine

A media player is, by its nature, a stateful system. At any point the player can be playing, paused, loading or buffering, and only certain transitions are possible. Our reaction to this domain concept was to model our system as a state machine. The player transitions through each stage in its lifecycle, and is always able to react appropriately to messages from the outside world, be it with the appropriate transition or with no operation.

For example, when in a paused state, calling pause does nothing but play prompts a change to a playing state. Initially, we used if/else clauses to model the state machine, as part of our ‘quickest route to green’ strategy. Later, we introduced state classes, in which individual states inherited from a parent State superclass with default no-op implementations of all methods on the interface, so individual pieces of behaviour could be implemented in a bespoke fashion for each state.

Event Bus

At the beginning of the project, we discussed the use of an event bus, concluding that we would use one to announce events such as seekCompleted which would need to be written to developer logs, to our statistics systems or to client features external to SMP such as My Programmes. All of these concepts are independent of the inner workings of SMP, so it made sense for them to subscribe to output events without the core player itself needing to be aware of them.

We avoided leaking our event bus into the UI code or into tests so we would be free to either change its implementation or abandon it altogether. The event bus was unit-tested in isolation to ensure that it was behaving as expected. Initially our event bus was a singleton, as this was the simplest solution and there was no need to support multiple media players in the same application in our first release. We gradually refactored away from this, and now have one event bus per instance of SMP.

Primitive Obsession

An aspect of our codebase that has made it considerably easier for other developers to use was our avoidance of primitive obsession. Primitive obsession is a situation that emerges when primitives are used to pass around domain concepts, such as using an integer to represent time. The first issue that this can cause is a lack of clarity over what variables represent as they move through the system. Does int duration represent time in milliseconds, seconds or years? The second issue is that methods with multiple arguments will be more likely to suffer from connascence of position, which makes it tremendously easy to start mixing up variables. For example, you could easily pass arguments in the wrong order to a method with the following signature with no compilation error.

void showProgress(int mediaPosition, int mediaDuration)

It’s clearer to pass MediaProgress objects that hold information on both the unit and meaning of domain concepts. The same applies when passing around String programmeTitle and String episodeName. You will never want a programmeTitle to be used as an episodeName, so why not make use of type safety and push these concepts into classes ProgrammeTitle and EpisodeName?

An exception to this was with UI code in Android, where we avoided passing domain objects into the UI entirely due to a limitation in the refactoring tools in Android Studio. If the Test Artefact is set to ‘Unit Tests’ rather than ‘Android Connected Tests’, and one uses the refactoring tools to change a domain object, the code in the connected tests will not be changed, which makes it easy to inadvertently break the build! However, Android Studio is frequently updated, so that problem may have been solved by the time you are reading this.

Inheritance vs. composition

There is a lot of literature on the concept of ‘Composition over Inheritance’. In SMP we do use inheritance in some places, to a positive end. For example, with the states in our state machine and also in the classes that wrap primitives.

The arguments in favour of composition are that it increases flexibility and provides more informative domain objects. We fully accept both points, but rather than couching our standpoint as composition over inheritance, we take the slightly less dictatorial stance of ‘try composition first’. Composition will very often provide a better solution but inheritance does, in my opinion, have its place. As it was, we were careful to avoid inheritance until it was screamingly obvious that it was appropriate, such that we didn’t inhibit the flexibility of our code by introducing it at too early a stage.

Open/closed

Good code should be open for extension and closed for modification. An example of code that is not open for extension is something like the following:

if (mediaIdentifier.getClass() == A.class) {
	findMediaUsingStrategyA();
}
else if (mediaIdentifier.getClass() == B.class) {
	findMediaUsingStrategyB();
}

If I add a mediaIdentifier of type C, I then need to add another strategy and another clause to my if statement. Alternatively, using a registry we can have:

mediaStrategies.getStrategyFor(mediaIdentifier).invoke()

In this case, there is a piece of code for finding media that will not need to change when you add another media type. A further extension of this, not yet implemented, would be to make the media identifier able to contain the information on how to resolve its own media. That way, adding a new media identifier could be done without making any code changes to the media resolution pipeline itself.

Dependency inversion

We’ve had a very good run at inverting our dependencies in SMP, and to a degree we’ve managed it. Our media selection, our statistics providers and our monitoring rely on SMP, not the other way round. However, at the core of SMP lies ExoPlayer, and SMP really does depend very heavily on ExoPlayer. The lifecycle of SMP was informed by the requirements of ExoPlayer, and in its current state we would be unable to arbitrarily substitute ExoPlayer for another media player. SMP knows all about its preparation lifecycle and implementation details such as ExoPlayer’s track renderers. We’ve no immediate plans to address this but if we ever do need to change the underlying solution of SMP we’d have to turn our architecture into something more resembling a hexagon. It’s possible that attempting to invert that dependency too early would have been detrimental to our solution.

Final thoughts

Throughout development SMP has remained deployable and developers have found SMP to be an easy codebase in which to work. I think that both of these factors are indicative that we got something right with our approach. We’ve been able to add new features without needing to perform shotgun surgery on the codebase, and potential bugs that were introduced have usually been caught by our tests before they had the chance to scuttle beyond a developer’s laptop.

That’s not to say that we’ve been completely free of bugs. Generally when we’ve missed something, it’s been on the edge of our system. Perhaps it’s important to be mindful that even when the core of the system is well supported by a thorough rack of tests, there’s always some things that you’ll miss, and that’s why we’re eternally grateful for our attentive software testers. We’ve also had issues highlighted by our beta users, who were warned by the beta label that things may not work perfectly, and were able to provide feedback to the �� before we launch SMP to a larger audience.

In terms of things I’d have done differently, I think smaller tests might have made it easier for new developers to get started, but based on previous experience I’d rather have tests that err on the side of too large than those that get in the way of development by pinning down a codebase. I’ve also spoken mainly about our architectural approach. There’s a lot more discussion to be made on other aspects of software development: on the relationship between product, test and dev; on knowing when to push back on acceptance criteria; on interacting cohesively with other teams; and on early delivery of a minimally viable product. All of these have been important factors in the success of the SMP project.

Thanks to Ross Beazley for technical leadership on this project and significant contribution to this article.

Technology + Creativity at the �������� Feed