Technology + Creativity at the �� Feed

Optimising serverless for �� Online

2021-01-26T13:34:29+00:00

The �� page, powered by serverless.

Previously I talked about why Web Core, our new web technology platform is built using serverless cloud functions. In this post we’ll discuss our experience of integrating them with our architecture and how we’ve optimised them for our use case.

The �� makes use of many different technology providers to deliver its online services. We make use of AWS’s Lambda service to deliver the serverless functionality for Web Core. There are other platforms which provide similar functionality, such as GCP’s Cloud Functions, Azure’s Functions, and Apache’s OpenWhisk. The majority of points made can be applied to any of these platforms.

Serverless in Web Core

We make use of serverless in two of the high CPU intensive areas of Web Core. The first is the React app that renders the HTML. This layer understands which visual components make up a page, and fetches the data for them. The second is our business logic layer, which transforms data from many different �� systems into a common data model.

Unlike other typical architectures we don’t make use of an API gateway to launch our serverless functions. Instead we have our own custom APIs running on virtual machine instances. These sanitise requests, forward traffic to the relevant serverless functions as well as handling caching and fall backs. These APIs are high network, high memory, low CPU services. This best fits a virtual machine instance over a serverless approach due to the availability of different instance types. In addition the scaling pattern for the APIs is slower and more predictable as traffic grows, compared to the high CPU workload carried out by our serverless functions.

A simplified version of the Web Core stack, requests enter at the traffic management layer.

So why the intermediate caching layers, why not just call two serverless functions one after each other?

Firstly to reduce cost. If we had serverless functions calling serverless functions directly without intermediate caching, we would be chaining them. This is a problem as we would be paying for the first to sit idle, waiting for the second to return. This can get expensive quite quickly, especially when service calls are involved. If this were a virtual machine instance, we would be able to run multiple requests in parallel, using the CPU’s idle time waiting for a request to return. But most serverless solutions have a one-to-one mapping between request and function invocation. Every request results in a new invocation, meaning the costs can quickly add up. Performance is therefore critical, but additional caching layers ensure most functions can complete in under 50ms.

The second benefit of intermediary cache layers is that we can perform content revalidation (e.g. stale while revalidate). This allows the presentation function to return a response to the audience quickly (and terminate the first serverless function) while the second serverless function runs and populates the cache for the next request.

Optimising our performance

Out of the box, serverless performs well and can easily handle large work loads that you throw at it. However, as it is the workhorse of our stack, we wanted to make sure we can get the most value out of it. The longer the serverless function runs the more it costs us, but also the longer the user has to wait for a response.

Most things you can do to optimise a normal application also apply to a serverless function, such as intelligent caching of data, reducing time waiting for other systems to respond, and reducing synchronous operations. For the purposes of this post we’ll touch on Lambda specific tuning that we have implemented.

Selecting the correct memory profile

Unlike virtual machine instances, Lambdas only have a single configurable “capacity” dimension — memory. Our React app is fairly low in terms of memory usage (~200Mb), so ideally we would pick the lowest value (and so lowest cost). However changing the memory setting also impacts the available vCPUs. This in turn impacts our response times and cold start times.

We performed load tests against each memory setting to determine the average response time. There is a critical point at which further increases in memory have no improvement in performance, but do increase in cost. In our case we found 1024MB as the optimal price/performance point.

Cold start times

When you make your first request to a serverless function, it must copy your code bundle to the physical machine and launch your ‘container’ for the first time. If you make a second request, it’s likely to hit the same ‘container’ that you previously launched. If two concurrent requests come in at the same time, the first hits the existing ‘container’ you have in place, and the second will need a new one starting up. It’s this start up time which is referred to as our cold start time. This happens every time we increase the number of concurrents from a previous high, or we deploy a new version of the code.

Some of the factors that can impact cold start time include memory allocation, size of the code bundle, time taken to invoke the run time associated with your code, and any initialisation you may choose to do before the serverless function executes.

As our website is mostly request driven, cold starts result in our audience members waiting longer. It also causes an increase in connections held open and so a significant slow down in serverless function execution times which can cause issues for any service waiting for that serverless functions to return.

One advantage of the cold start process is that you can take care of any initialisation that might be required for when the function executes. This allows the actual invocation to be as efficient as possible. You also don’t pay for the cold start period, so essentially a few free cycles you can use to optimise the performance of your function invocation later. We used this time to establish network connections to all the APIs which are needed (removing the need for SSL negotiation later) and we also loaded any JavaScript requirements into memory for use later. This means that we are not having to perform this logic during every execution.

As mentioned earlier, we went through a process of identifying the correct memory profile for our serverless function. During that process we discovered that the memory profile also impacted cold start times, with a 512 MB memory profile increasing cold start times by a factor of 3 over a 1024 MB cold start time.

So in tuning our cold start times, we struck a balanced approach between per invocation overhead and per cold start overhead.

A graph showing minimum, average and maximum cold start times over a week for our react rendering function. The X axis is time, Y axis is cold start duration in milliseconds.

In terms of overall cold start performance, it’s very predictable, with our average cold start time sitting around 250ms, and our maximum peaking around 1 to 2 seconds depending upon time of day.

What type of performance do we see?

We are performing over 100 million serverless function invocations per day. The performance of each invocation is predictable and performs as we would get from a virtual machine or container based service. Our HTML render times are quick, with the p90 being around 220ms. This time includes the time taken to fetch the data from our data services.

A graph showing the average and p90 response times of our presentation React app serverless function over a 4 week period. X axis is time, Y axis is duration in milliseconds.

In terms of scalability, during benchmark tests we were able to go from a cold system at 0 rps with everything uncached to 5,000 rps within a few minutes. We would have tried a faster scale up, but we didn’t want to break our internal content production infrastructure. During our production use we have not seen a single function invocation fail. At 100 million requests a day, that’s some achievement.

Diagnosing and debugging

One of the disadvantages of serverless is the lack of a platform to remotely diagnose during an operational incident. If something went wrong whilst using an EC2 instance, once you’d exhausted logs and monitoring, you would typically SSH into the box and diagnose the health of the running device.

Moving to serverless for our compute has meant us having to adopt other platforms to support monitoring and debugging. To do this we make use of a combination of X-Ray and CloudWatch, brought together via Service Lens. Other platforms from other providers are available to do this including DataDog and GitLab.

A screenshot of an X-ray trace showing the service calls and timings of rendering our homepage.

On every request that enters our stack, we annotate the request with a trace identifier. As the request navigates our stack the request maintains the id. This allows us to build a graph of each request through the system, along with a timing waterfall.

If an individual page is erroring, we can look at the traces generated by that URL and understand where in the stack that issue lies. Through the Service Lens interface, we’re able to provide all the log files for that trace in a single place, which enables us to view any stack traces or errors generated by the code.

We have found that this form of debugging has enabled us to go from being notified of an operational incident to a log stack trace in a very short period of time, which is great for ensuring our website stays available and up to date.

Deploying to Live

One of the big surprises with serverless was the way in which it changes your approach to environments and deployments. Typical virtual machines require you to build an image and deploy rolling changes out to your infrastructure. If you want to run two builds concurrently (e.g. staging and testing) you would need a separate infrastructures running that software.

With serverless, it’s slightly different. We package and upload our code and it sits there ready to be invoked, at any scale, but critically in isolation away from the other versions you have already uploaded. Containers do operate in a similar fashion, but could still impact the underlying hardware on which your critical service might be running.

Within the Web Core project, we have the ability to build and deploy our serverless function on each commit to GitHub (although typically we wait until the pull request stage). This can then be invoked by a developer, tester or product owner and get the full functionality as it if were audience facing, even when it’s in development. In essence this gives us an environment per pull request!

When a feature or change is ready for deployment, we alias the function to deploy the change to live. Making a particular commit hash the ‘production’ alias, will expose that serverless function to the audience. A rollback is just a case of updating the alias again.

Thanks for reading!

The platforms that run �� Online are always evolving and changing as our technology providers improve their services. We’re always on the lookout for how we can deliver the best audience experience. The �� website can reach 60 million browsers a day, with our users requesting up to 20k pages per second.

Compute comes in many flavours, and after careful consideration we chose to use serverless for the task. We are now running over 100 million serverless function invocations per day, with the technology being cost effective, able to respond quickly to changes in demand and performant to audience requests.

The technology has worked well from the outset, however we have learned a number of tricks to improve its performance (including optimising cold start times and behaviour). We have also overcome some of its limitations by adding non-serverless layers to our stack.

We have been able to take advantage of the monitoring services integrated with serverless, improving our ability to diagnose and debug issues quickly. Thanks for reading.

Delivering �� Online using serverless

2021-01-20T11:31:49+00:00

A �� News article powered by serverless.

Previously we wrote about how we’ve moved �� Online to the cloud. This post looks more closely at why Web Core, our new web technology platform is built using serverless cloud functions.

The �� has some big traffic days

Before we talk about serverless, here’s some context to the workloads we need our technology platform to handle.

During the 2020 US Election, the �� News website received visits from over 165 million unique browsers from around the world. During the election week the Web Core stack averaged 80,000 requests per minute, peaking at 120,000 requests per minute to our platform. Those requests are the ones which made it past the traffic management layers (caches) in front of us. At its peak our edge traffic managers and CDNs saw 2.5 million page views per minute (around 41,000 requests per second).

At first glance, our traffic profile is typically cyclical — busy during the day, and quiet overnight. However, it isn’t completely smooth and predictable as the graph below shows. Instead, it fluctuates based on the current news, social media traffic, as well as calls to actions on TV, radio and when we send push notifications to our apps.

A view of how a request travels through the Web Core Stack

In addition, different parts of �� Online will peak at different times, and in unpredictable ways. If a major news story breaks, �� News can peak in an instant. And �� Bitesize, during the first UK national lockdown, became consistently busy during the school day, as school children moved to learning online.

As an example, let’s take the breaking news story around the London Bridge attack in November 2019. The traffic profile at our edge cache for this page resulted in a 3x increase in traffic in a single minute (4k req/s to 12k req/s) followed by a near-doubling of traffic a few minutes later (12k req/s to 20k req/s).

These key moments are critical for the ��, and they are the moments in which the audience turn to us. We must not fail. Any technology we choose must be able to respond to these traffic patterns.

A chart showing requests per second over the 7 day period 3/11/2020 to 10/11/202 for bbc.co.uk/news. The traffic is mostly cyclical, with a few exceptions as news stories break.

Choosing the right platform for the job

Choosing the type of compute depends upon factors associated with your workload. These can include the number of requests, duration of work, and resources required (CPU, Memory, Network). Being a website, requests are short lived — up to a second or two — and don’t require persistence between requests. The requests are typically CPU bound, rather than restricted by network or memory.

Virtual machines (e.g. EC2 or Compute Engine) are useful for workloads that change no faster than you’re able to add capacity, or for work loads that can tolerate delay in scaling (e.g. queue based event systems). They are also useful for short lived sessions that can tolerate scale down events. For our use case — rendering web pages — virtual machines don’t tolerate large bursts of traffic well enough, as they can take a few minutes to add capacity. As mentioned previously, our traffic load can double in less than a minute, and so autoscaling might not react quickly enough. Our experience has taught us that we need to over provision the number of instances you have available (e.g. 50% more than what you need) in order to deliver enough capacity to manage scale events. Even trickier is understanding when you have over provisioned and when you should scale down. Virtual machines comes in many different flavours, tailoring for high memory, high CPU or a combination of both. So it’s possible to pick a compute platform that matches your workload, response times and cost profile.

Containers (e.g. a Kubernetes cluster or Fargate) which run on top of fixed compute. Like virtual machines they are useful for traffic volumes which slowly change over time. While you’re able to start up a new container quickly to handle a new session, you’re still limited by the underlying compute instance on which the containers are running. The underlying compute hardware has to scale to meet the demands of the running containers. Containers are great for long lived, stateful sessions, as they can be ported between physical hardware instances while still running. This allows for them to be redistributed across physical hardware as the workload grows and shrinks. A great use case for containers would be an online game, where you need sessions to persist as you reorganise your workloads.

Serverless functions (e.g. Lambda or Cloud Run) are, in essence, containers running on top of physical hardware. They are ideally suited for handling unpredictable traffic volumes and for non-persistent and stateless connections. As a customer of the service we don’t need to worry about how the underlying hardware is provisioned to deal with load. Instead, we simply initiate requests to the service. The burden of provisioning the containers, and scaling of the underlying infrastructure is handled for us by the service provider. So why can they handle the scaling better than we can using containers? — The simple answer is economy of scale, in that the provider manages the work loads of many customers shared across the same physical hardware platform, what is a large increase in workload for us, is small in comparison to the service providers overall workload.

Reducing operational complexity

One of the biggest advantages of serverless over fixed compute is that of reducing the operational complexity of running the service. Reducing the engineering effort in running and maintaining our systems allow us to focus more on the audience product and experiences. The further away you move from the bare bones hardware, the less the engineering cost for the developing team. What used to be engineering effort for DevOps teams, is now commoditised.

With Virtual Machines we need to consider general maintenance and security, everything from patching the underlying operating system through to ensuring we have configured the machine for running at scale (such as available file handlers). We need to ensure the machine is secure from an external attack, whilst being remotely administrable. We also have to manage things such as log rotation and temporary file storage. All of these things go away when you move towards serverless.

Managing the scaling rules of a fixed compute server is complex and requires extensive testing. You have to consider aspects such as measuring your workload, capturing metrics, understanding the correct properties to scale on, and by how much. Understanding and implementing these things come with some considerable effort.

Next time… using serverless in Web Core

This blog post has outlined some of the considerations we took when choosing a technology platform for Web Core.

In my next blog post I’ll talk about how we make use of serverless, including how it fits into our architecture, its performance and some benefits we’ve gained from its use.

The complexities of creating a new 'follow topic' capability

2021-01-13T13:49:58+00:00

Exploring how the visual emphasis of our Follow UI works alongside other calls to action.

The �� is committed to creating a truly personalised experience for everyone. We believe that a more personal �� helps us build a deeper connection with our audiences, and helps us to extract more value from online. We have so much to offer across our products and services and one of our challenges is to help our audience discover the content we have.

We have a number of approaches to achieve this but alongside automated personalisation like smarter recommendations, we want to enable our audience to tell us what they want to see more of. So, we’re exploring the idea of enabling audiences to follow topics, a new capability that could work as a service for all of our teams and products.

This consistent and collaborative approach is not only more efficient, but would allow users’ preferences to follow them wherever on the �� they are. This blog posting will try to convey the complexities around creating this new follow capability.

Context

The �� creates thousands of different content items on a daily basis, ready to be consumed by people in the UK and all around the world. But how might we improve the discoverability of that content? How might we offer new content to those who are interested in it?

The �� team started working on the idea of ‘topics’ in summer of 2019. Working with colleagues from around the ��, we have developed the technology to present a Pan-�� topic to the audience.

Very early on in the work on topics it was recognised that there was an opportunity to harness the power of topics by allowing the audience to choose which topics they personally find interesting so we can personalise their experience of the �� and notify them when relevant content is available. Examples of this already exist with the �� such as the My Sport functionality in the �� Sport app; subscribing to podcasts in �� Sounds and adding a programme to your favourites in �� iPlayer.

The prior implementations have been undertaken in a product-specific fashion, but we are building ’follow’ in a unified way, aiming for a seamless audience and technical approach across all products. Behind the scenes, the mechanism for all these types of ‘following’ is common. How might we create an experience which harnesses all these ‘follow’ events to aid in content discovery by putting new and relevant content to our audiences quickly and efficiently?

Principles

For further background, read Matthew Clark’s blog post.

Platform Consistency

All new developments will occur within the ��’s existing web technology platform unless it is not technically possible. WebCore utilises a mono-repository and because all the teams can access everyone’s code, it becomes much easier to be consistent across all our teams’ work. The graphical components are all stored in a central repository and are available to all teams to utilise and integrate into their own developments.

Reuse

All existing solution building blocks need to be taken into account and reused where applicable. If anything new is developed, it must be straightforward for other teams to utilise or develop further to meet their own needs.

Collaborate

Collaboration with other teams is key to this project’s success. This work is being carried out within a complex organisation and as such, numerous teams and colleagues will be stakeholders in it. Openness and a readiness to collaborate will pay dividends later.

The Key Considerations and Complexities:

What concepts should the audience be able to follow?

We have chosen to do this, initially, at the topic level. For example, audience members will be able to follow their favourite musical artists, news correspondents or even locations. As topics are representations of the concepts that the �� is creating content on, this feels like a more natural fit given that new content will keep being added to inform audiences on. While standalone content items can be revised (often with minor or major edits) after publication or when the story develops, this feels like too granular a level for a user to specifically follow, making a save or read later function more appropriate. To ‘Follow’ an article, for example, is more of a ‘read later’ function.

We need to understand how a ‘follow’ button should operate and how it changes state based on the current context it’s displayed within.

We need to consider if the audience member is already following a topic or not, if we need to offer the ability to unfollow that topic or not.
We also need to consider preferences we’ve already been given for other follow-like implementations and if it’s appropriate or not to use that data.
It might also be applicable to present a follow count to indicate how popular a particular topic might be.
We will also make the button mechanism available to our app ecosystem to keep the mechanism consistent across all our platforms.
We need to build the logical components in such a way that they will operate in the same way regardless of where the follow button is presented.
For example, if the follow button is offered on a Pan-�� topics page like Manchester or Mark D’Arcy we must ensure the behaviour is consistent and intuitive.
How will it work within accessibility guidelines?
How will it work for our apps?

Legal constraints and how this will operate within the law

Considering GDPR and previous preferences provided to us by our audience members, can this new capability function within the law and if not, what must we do to ensure we are legally compliant and transparent to our audience?

How will this operate within the overall ecosystem of the �� Online estate?

The �� Online estate is complex and ever shifting. We must ensure we abide by open standards and reuse existing services to ensure a good fit today and for the future. Also, we must collaborate with teams internally to ensure our internal stakeholders are fully aware of our progress.

How will we ensure we don’t create echo chambers for audiences?

We need to carefully consider how we present followed content, so that it is presented coherently alongside both recommendations and related content.
Followed content will be shown next to editorially curated selection, therefore, we will also need to consider how we deduplicate and prioritise content.

How do we maintain adequate levels of data to be functional and keep that minimal dataset secure?

We have a very robust data security process to satisfy. At every design step, security of the data is a key requirement.

What data do we need to record to adequately record a Follow action?

This data must then be easily parsed and follow our own internal standards and schemas.
This data format must be compatible with the User Activity Service — our centralised location for storing user activity data — and be registered with it.

Where is the result of following one or more topics displayed? i.e. how are followed topics surfaced?

This is a key question for any product team. The ��page team will have different needs than for example, the Sport team. But, the underlying system must be able to cope with this in as an efficient manner as possible.
A key consideration is how much should we develop for each delivery? We could work for a long period and deliver in a more big-bang approach, or does it make more sense to develop in small iterations accepting a changing interface and functionality to our audience?

How will this level of personalisation work within the WebCore stack’s caching strategies?

This is less of a concern for our team as we’ll be using the outcome rather than heavily contributing to it. However, it is of great importance and so we will keep in close touch with the team working on it to ensure no assumptions are made about what is needed.

Do we have the technical capability to process a potentially complex set of follow activities?

Our �� Data Capabilities Team is developing specific technology based on AWS ElasticSearch for this very purpose.
This system will capture metadata about our content from various internal sources to allow us to query that metadata in novel ways to find the content we need on a per user basis.

How will the resulting follow actions be surfaced?

The Follow Product and User Experience team have broken down the approach into four separate phases:

Audience members will be able to see what topics they’re currently following, follow new ones and choose to unfollow others from a topic index page or the �� homepage.
Our audience members will be able to see the topics they have followed across the News and Sport websites and apps. This will come with the new ability to manage what they have followed across the ��.
Also, our audience will get the ability to follow a topic from the content level i.e. from within an article.
Follow will become a unified approach across the �� online estate helping breakdown any barriers between content and aiding discovery for all audience members.

One outlet for follow is in alerts and notifications. There is a team working hard on understanding how best to achieve this whilst offering the most value to our audience. It’s a complex area so we’re keeping close on that work to ensure we’re ready for it.

Conclusion

It is early days for ‘follow’, but the ground work is already in place for it to be a great success. The �� topics product is maturing and is already powering the majority of the Sport website and app. It also powers the �� homepages and we’re currently working closely with the News team to help them bring the �� News topics up to date and into the WebCore world.

We’ve detailed our key principles and considerations around the follow action and how we plan to implement it. It’s a challenging area to work within but all the building blocks are sliding into place, and we’re starting to look at how and when we can roll out the first phase. Following a topic is something people understand, as they’re used to doing it on social media.

We’re working hard on bringing this functionality to our audiences as soon as we can make sure it meets the level of quality our audience expects from the ��. We aim to bring together all our content to all our audiences in such a way that the traditional silos of content are blurred to the point of invisibility.

�� Food at 20: Its biggest year

2021-01-08T08:46:27+00:00

Banana Bread

20 years ago when �� Food was first created we would never have imagined how we’d be celebrating its 20th anniversary. The average weekly traffic across all weeks in 2020 was 3.4m unique UK browsers, which was up +74% on the average for the previous calendar year. The peak weekly traffic came at Easter with 6.4m UK unique browsers.. That’s quite a birthday party!

2020 wasn’t an ordinary year

Back in March 2020, as stocks in shops diminished, more people found themselves at home and people juggled a working life with childcare, there were overnight changes to everyday behaviours up and down the country.

On 16th March the country was asked to start restricting movements and interestingly, within a couple of days of the press conference, we saw that Google search trends heavily featured bread searches. In fact, 7 of the top 10 Food and Drink related searches included bread. It comes as no surprise then that 10 of �� Food’s most popular 40 pages that week were bread related, with a huge nod towards banana bread. People weren’t only turning to bread though, as another 10 of �� Food’s most popular 40 pages were related to baking in general. There was a particular period in lockdown where the top 20 recipes contained no family cooking. It was 19 baking recipes — plus Dalgona coffee (a viral trend that had a moment and performed fantastically for us at the time).

The below outlines what went on to be the most popular recipes of 2020:

Fluffy American pancakes

The �� Food search became popular as we approached summer, and even became the most popular page on �� Food. This is because people started searching for what to do with specific ingredients that were in their store cupboard, either because they were shielding and couldn’t go out or because of limited supplies in shops.

We also began to see article content drive traffic, particularly articles related to coping through lockdown — health, budgeting, limited ingredients. Below are the top 20 articles of 2020:

Carrot cake

As the year went on the trends observed above continued, sometimes to a lesser extent dependent on the layers of lockdown applied at any given time and location.

January: no real trends (Fluffy American pancakes, 171k unique UK visitors)
February: pancakes! Fluffy American pancakes, Basic pancakes with sugar, and How to make pancakes rounded out our top 3 in Feb, totalling 1.16m.
March: Fluffy American pancakes, banana bread starts its rise to stardom, and easy white bread, totalling 900k.
April: Banana bread, 657k.
May: Mary Berry’s scones for VE Day — 570k.
June: Fluffy American pancakes — 321k.
July: Fluffy American pancakes — 259k.
August: Fluffy American pancakes — 254k.
September: Fluffy American pancakes — 205k.
October: Fluffy American pancakes — 256k.
November: Fluffy American pancakes — 254k. But, Christmas Cake close behind with 238k.
December: Gingerbread men with 398k — the honey-glazed ham a little behind with 326k.

Mary Berry's scones were popular to celebrate VE Day

Popular peaks

These trends, alongside regular annual events, helped to determine the most popular peak days on �� Food:

24 December — Christmas Eve (Honey-glazed gammon; Article — The Christmas treats we need to make a comeback)
12 April — Easter Sunday (Roast leg of lamb with garlic & rosemary; Yorkshire puddings; fluffy American pancakes; roast calculator)
18 April (Search; ��page; fluffy American pancakes)
13 April — Easter Monday (Article — how to use your freezer better; banana bread)
3 May (Fluffy American pancakes; Search)
29 March — first Sunday of lockdown (Banana bread; fluffy American pancakes; Search)
4 April (Budget homepage; banana bread; Search; fluffy American pancakes)
2 April (Article — how to make bread with plain flour; Article — Chefs’ top tips for when we’re short on supplies)
11 April — Easter Saturday (Search; fluffy American pancakes)
19 April (Search; fluffy American pancakes)

Following the most successful year for �� Food we then completed this with the biggest Christmas we’ve seen. During Christmas week we achieved 5.1m UK unique browsers on �� Food, this beats our previous highest Christmas peak (set in 2019) by 33%. Our peak hours came between 1pm and 6pm on Christmas Eve, followed by 11am to 1pm on Christmas day. The most popular recipe (by quite some way) during Christmas week was the Honey-glazed gammon which received 226,523 visitors.

Honey-glazed gammon

Budget cooking and Daily Kitchen Live

With changes happening very quickly, we had to look at how �� Food could be there to ensure the nation made the most of the food that was in their store cupboard. A new topical budget index was put together incredibly quickly so we could respond to an audience need. It combines articles and recipes; the editorial team commission recipes and feature articles that educate audiences in healthy eating and eating better on a budget. They also produce educational video content to teach basic cooking skills and work with qualified dietitians and experienced home economists to make sure the health and budget advice is evidence-based.

In April, ‘Saturday Kitchen’ turned up the dial and turned into a daily show, ‘Daily Kitchen Live’.

As all the recipes came into the �� Food website, we fed in our search trend findings so we could use it to fill any commissioning blanks and in turn attempt to increase reach further.

Changes to trends: Reactive SEO

When conducting any keyword research for SEO purposes, we would typically take into consideration the average monthly search volume of terms. The average would obviously be influenced by seasonality, but the terms with higher search volumes are typically much more competitive as a lot of sites compete to rank and gain traffic.

Lockdown meant that there was nothing “average” about search behaviour and traditional search volumes went out of the window.

'How to make' Google search volumes

Using Google Trends we were able to analyse the changing behaviour, as in the above screenshot, which shows the rise in queries related to ‘how to make’ in the Food & Drink category during the first week that the UK went into lockdown at the end of March.

By conducting trends analysis you can then see top and rising queries related to a phrase or topic in order to determine whether your site already contains related content or if new content needs to be produced to fulfil this new user intent. It can take some time to commission, test, photograph, and upload new recipes, so adapting or promoting existing content was key to maximising visibility during this period.

‘How to make …’ became an extremely popular range of queries during the initial lockdown period, as the UK began to adapt to the new challenges of 2020.

If we look at Google’s Search Console data, beginning on the date of Boris Johnson’s initial press conference advising British people to stay at home, we can see that there were approx 2.5m visits to bbc.co.uk for queries containing ‘how to make’.

The query driving the largest amount of traffic during this time was ‘how to make a face mask’, which the News team had covered. The other four are food related, which was reflecting the growing trends in home cooking and baking during lockdown.

In order to quickly analyse what was driving organic traffic, a Google Data Studio report was spun up, allowing easy access for all stakeholders to specify a date range in order to view the most popular recipes which were being accessed, along with the search queries which drove them.
It was at this point that the banana bread baking craze was in full swing and began to register record-level organic traffic to the Food site.

Changing Spikes

Typically, there are two organic traffic peaks for the �� Food site during the year. The biggest peak is during the week building up to Christmas when the public are planning their shopping lists (or leaving searching for recipes right until the last minute on Christmas Eve or Day as millions seem to do!).

The second peak is at pancake day, when the nation is split between those who prefer Delia Smith’s pancakes with sugar and lemon juice, and those who fancy American-style fluffy pancakes. Perhaps unexpectedly, the American-style have consistently been one of the most popular recipes on the entire Food site.

In the below graph we can see the spike for Pancake Day traffic at the end of February, before lockdown was announced. With restaurants and takeaways closed across all nations of the UK during a similar period, demand for recipes skyrocketed.

Overnight, this particular banana bread recipe became one of the most popular pages across the entire �� online estate, with over 11 million organic visits between the UK’s lockdown beginning on 23rd March and ending on 4th July (for England — end dates vary per nation).

Top Search Terms Driving Traffic During the UK’s Lockdown (23/3/20–4/7/20):

Keyword: banana bread
Visits: 487172
Top Landing Page: /food/recipes/bananabread_85720

Keyword: shortbread recipe
Visits: 423045
Top Landing Page: /food/recipes/shortbread_1290

Keyword: pizza dough recipe
Visits: 412820
Top Landing Page: /food/recipes/pizzadoughbase_70980

Keyword: banana bread recipe
Visits: 348937
Top Landing Page: /food/recipes/bananabread_85720

Keyword: pancake recipe
Visits: 340006
Top Landing Page: /food/recipes/fluffyamericanpancak_74828

Keyword: scone recipe
Visits: 311669
Top Landing Page: /food/recipes/tea_time_scones_77839

Keyword: banana cake
Visits: 277678
Top Landing Page: /food/recipes/easiest_ever_banana_cake_42108

Keyword: carrot cake recipe
Visits: 273081
Top Landing Page: /food/recipes/classic_carrot_cake_08513

Keyword: american pancakes
Visits: 271192
Top Landing Page: /food/recipes/fluffyamericanpancak_74828

Keyword: bread and butter pudding
Visits: 242924
Top Landing Page: /food/recipes/breadandbutterpuddin_85936

Christmas Trends

Using the experience of 2019 alongside our historic traffic data, we had been preparing for �� Food to receive its greatest amount of seasonal traffic at Christmas 2020.

This period can be split into three distinct sections:

Pre-Christmas (1st-23rd December)
Christmas (24th & 25th December)
Post-Christmas(26th-31st December)

Pre-Christmas

Although it can be tempting to start this period even earlier as some people make Christmas cakes as early as September, the pre-Christmas period of 1st-23rd December is when we start to really see the traffic ramping up to an annual high. Comfort food and mulled wine takes centre stage in December as the temperature drops further.

Top 5 Search Terms Pre-Christmas 2019:

The top 5 terms saw only slight changes in 2020, with more Brits enjoying eggnog than ever before.

Top 5 Search Terms Pre-Christmas 2020:

Christmas

Whilst the majority of research for shopping purposes has been done in advance, once Christmas Eve hits, the public begin to search for how to actually cook the turkey they have purchased.

Rather than searching for the vague term “christmas dinner”, searchers are typically looking for how to cook the specific elements of their meals.
Perhaps surprisingly, Google Trends indicated that red cabbage was the most searched recipe during this period in 2019 and our own data also puts it on top spot.

Top 5 Search Terms Christmas 2019:

Red cabbage lost its top spot last year.

Top 5 Search Terms Christmas 2020:

Leftover Turkey and Ham pie

Post Christmas

Once Christmas dinner is out of the way, it is time for the leftovers to shine, with turkey pies and curries moving to top of the trends.

What’s Next?

Although reactive SEO arguably played a bigger part last year than any before, it is still necessary to plan ahead for events we know will always be big in the food calendar — with Veganuary now in full flow and then Pancake Day just around the corner, there’s plenty to sink our teeth into.

�� Online: 2020 in review

2020-12-07T13:44:53+00:00

2020 has clearly been a very unusual and oftentimes painful year in many ways for just about everyone on the planet. Like many organisations, we’ve had to shift the majority of our staff to working from home in a very short timeframe — this was easier for some teams and people than others.

It’s well documented that in troubling times, the world comes to the ��. Nowadays, that means we see big increases in demand for our online services. That extra demand and adaptation to new working patterns has been coupled with enormous changes in our output — far fewer TV programmes being made, much more frequent breaking news events, a significant bump in the volume of online education we provide and regular live press briefings, to name but a few.

With all this going on, it’s easy to forget about the fantastic work which has been going on, so we wanted to follow on from last year’s “�� Online — 2019 in review” and document 2020. Lots of our teams have contributed so you’ll be able to see some of the highlights of their achievements.

�� Children’s and Education

The Children’s and Education team are responsible for building and maintaining 4 flagship CBeebies app propositions, 1 C�� app proposition, 350+ interactive games and content items, the C��, CBeebies and Own It websites along with the Bitesize website and app.

Jacob Clark has written a blog post “Shipping Progressive Web Apps everywhere”, which outlines in detail the journey we’ve been on these past 12 months transforming how we build our 5 flagship Children’s apps. Shifting away from proprietary native app frameworks to building on modern, open, web standards, developing universally distributable web apps on a new platform.
We’ve been working hard over the course of the past year ensuring children and parents have access to resources to Daily Lessons to continue their education as well as developing new, personalised and adaptive learning experiences.
Across our games estate, we’ve been working on ways to provide users with onward journeys into more games they might be interested in through our cross platform “Exit Tile” whilst developing features such as Shops within our core game engine technology Genie.

�� Account

The �� Account team are responsible for enabling sign in or registration for a �� Account. This helps to enable a personalised service for our audience members across TVs, the �� website and mobile applications.

In 2020 we’ve brought lots of great improvements to �� account:

Children’s profiles: We’ve worked closely with the iPlayer team to enable our younger viewers to see the content that’s more relevant to them using our new children’s profiles on televisions.
Password breach checker: Building on an idea from our engineering 10% innovation time, we aim to help our audience to create safer passwords by checking if they have been used in a previous data breach. Read more here.
Showcasing Account Memories: To remind users of the amazing content they have enjoyed and to associate happy memories with the ��, we’ve started to play back some of the user activities within the Account space.
Performance improvements: We’ve iterated and optimised our applications to enable better resource utilisation and autoscaling. This has allowed us to cope with the increased demand for our accounts during lockdown.
Rolling refresh tokens: To improve the experience for our TV users, we’re keeping regular users signed in for longer.
iOS and Android: We’re making better use of the capabilities of these platforms to make it easier for our users to sign-on to multiple �� apps and websites.

�� World Service News

The World Service News teams are responsible for building and maintaining an open source platform called Simorgh which is used for the ��’s 41 language World Service News websites. We’re made up of 13 Engineers, 2 Business Analysts, 2 Product Managers, 2 Delivery Managers, 3 Testers and 7 User Experience Designers.

This year we’ve been working on:

Simorgh: Over the past 12 months we’ve migrated our pages which are spread across 41 discrete sites from a legacy PHP monolith to a new React based application. This application is called Simorgh, an open source, isomorphic single page application developed by the World Service languages team.
Performance improvements: A large proportion of the �� World Service audience are on slower 2G and 3G networks, they use lower end budget-friendly android handsets or feature phones. Blocking JavaScript requests dropped by 100% from 9 to 0, JavaScript requests dropped by 79% and total page weight is now 60% smaller than before. We have made other efforts to improve our web page performance, meaning our worldwide audiences can access stories quicker than before.
New reading experiences: We have started work on a project to deliver a new article experience to our users, which will allow our editorial teams to more easily create content for our audiences.
Improved media experiences: We have launched new ways for our worldwide audiences to access audio and video content on the World Service News websites.

�� Website (including News, Sport and ��)

This year has been a huge year for the �� website, which has completed its rebuild and move to the cloud. Tens of millions of people now consume articles, video clips, and other content on a rebuilt site that’s faster and more accessible. Multiple pages have been rebuilt, including the �� Page, topic pages, articles, clips, and search. And the technology used has changed, from PHP to Node.JS/React. It’s also partly hosted on serverless technology, and we believe we’re one of the largest sites in the world to do so.

This recreation of our website, ready for the next decade, has been a huge programme of work. It’s tackled duplication and cross-team challenges, ensuring new site is as efficient and high quality as possible. Matthew Clark has written a separate post on the approach taken to make such a large change happen.

Radio & Music Services (�� Sounds Back End)

Radio & Music Services are a team made up of around 12 software engineers and 5 software engineers in test, along with product and business analysts. We’re responsible for building the API that encapsulate the product functionality of �� Sounds. This API is used across web, apps and other clients to provide the Sounds Experience, drawing on content from around the ��.

This year we’ve been working on:

Audience: Rolling out Sounds to International users.
Tooling: Migrating our build system to Code Pipeline.

�� Voice+AI

The Voice+AI team are responsible for building & maintaining several Voice-based and AI-supported services in the ��. We’re made up of 45 engineers, 9 Product Managers, 5 Delivery Managers and a team of UX designers and editorial.

This year we’ve been working on:

Beeb, the voice assistant from the ��: We’ve built, from the ground up, a fully-functioning voice assistant and released a Beta demonstrator on Windows 10. Built in partnership with Microsoft, we’re using several new technologies, including DirectLine Speech and BotFramework, a custom Natural Language Understanding model, and a stellar bespoke neural voice that we developed for the ��. Beeb for Windows 10 is available today on the Microsoft Store.
�� Corona Bot: Launched to the �� News Facebook pages’ 50 million followers. Built using Azure’s QnA Maker Cognitive service, the model for the chatbot was trained using editorially curated question and answer pairs on the latest information from lockdown rules, to relevant medical information with support from the NHS.
Songbird: In partnership with �� World News we launched a ‘listen to this article’ feature. An entirely new content format for the �� the feature provides text-to-speech rendered articles using our bespoke neural voice. Try it out on The Life Project.
Local news bulletins: At the beginning of lockdown in March, we launched an ambitious project in partnership with Nations and Regions to make available news updates from all 40 local stations in England, Scotland, NI and Wales. In the midst of a pandemic, local information is more important than ever.

�� Online Technology Group (OTG)

The Online Technology Group is responsible for the provision and management of the ��’s core online services such as networking, DNS, CDNs, audience-facing TLS, load balancing, caching, routing and high-availability facilities to content origins, our on-premise hosting platforms, and media analytics.

Here’s what some of our teams have been working on in 2020:

Website traffic management

DNS: Migrating our core DNS platform to a new supplier and building a robust, fast, low-cost, automated deployment and testing CI/CD pipeline which is driven from source control.
Statistics: Adapted our Reporting API collector endpoint to also receive Web Vitals data from our client-side code.
Website traffic management: Removed our old traffic managers and caching layer from the majority of traffic on www.bbc.co.uk and www.bbc.com.
Quality & efficiency: Made our www edge services more consistent & efficient (cache offload) across CDN & in-house traffic managers (GTM).
Traffic peaks: Like many organisations, the events of 2020 have resulted in regularly/constantly elevated traffic to our websites. We’ve seen around 40% more than we did in 2019 — a big bump on the usual year-on-year increases.

Media Distribution

�� Internet Distribution Infrastructure (BIDI) is an in-house Content Delivery Network (CDN) designed for high volume distribution of iPlayer video and audio streams.

Traffic peaks: BIDI’s traffic peak reached 1.2 Terabits per second in 2020, serving nearly 250,000 concurrent iPlayer clients.
SmartShard: A new playback routing algorithm called SmartShard has been developed, greatly improving cache efficiency and automating capacity management around the network.
Capacity: BIDI estate size has doubled in 2020. BIDI now consists of 102 caching servers, across 24 datacentres, using over 1,200 Solid State Drives. Deployments are run in both �� sites and within 3 UK Internet Service Provider networks.
Resilience: Automatic failure detection is now active, reducing service disruption when either the hardware or underlying network fails.

For a look at BIDI growth so far and plans for the future, see: virtualUKNOF November 2020 — Building the ��’s media CDN.

iBL (iPlayer API)

We are the iBL team — the iPlayer Business Layer, and we are responsible for the data that powers all �� iPlayer apps — mobile, web, and smart TVs. From promotions on the homepage, categories, channels, to personalisation, user data, schedules and recommendations. We are a team of 5 engineers, a product manager, and a delivery manager.

In the past twelve months we:

Improved promotions and relevancy: Creating “hero” imagery to showcase high profile content, and using audiences’ watching history to highlight content they are interested in rather than treating everyone the same. This includes messaging why we recommend certain programmes, and tailoring homepage promotions for younger audiences.
Safeguarding younger users: There is a breadth of content on iPlayer, and not all of it is suitable for all ages, so we are taking steps to allow users make better informed decisions.
Technical Reliability: We’ve worked on our resiliency a lot — we have improved capacity and redundancy across about a dozen of our database-backed services this year. We’ve also moved some of our error handling to a CDN so we can more reliably serve peaks of demand.
Developer workflows: We’ve already had our “infrastructure in code”, using CloudFormation. This year we have also moved our CI pipelines into code, and include infrastructure changes in our CI/CD workflows to make sure the code is always up to date with what is actually deployed. We are also iterating on our internal documentation, using OpenAPI schemas to ensure the data we exchange internally is what we expect it to be.

�� iPlayer — Off-Product Discovery

We are the people that share the iPlayer catalogue and availability with partners and link up voice interactions from non-�� platforms. It’s what allows you to choose to watch a programme on iPlayer from somewhere which isn’t inside iPlayer! Next time you say ‘watch his dark materials on �� iPlayer’ to your fireTV stick or Chromecast with GoogleTV think of us!

Big deliverables in 2020 have been:

Amazon catalogue and Voice integration: Following on from many months work, we are now delivering our standard catalogue and integrating with the Amazon video platform. We completed the work to allow you to find content and then control the playback of this content in iPlayer using voice interactions as well as from the remote. You can now see that Dr Who and Killing Eve have the newest episodes on iPlayer, available for no additional charge.
GoogleTV catalogue and Voice integration: A very different technical solution completed quickly by following on from the lessons learned working with other partners and having a solid understanding of the user journeys that voice control opens up. Again, we have developed and delivered the code to link up google’s infrastructure with the ��s and allow you to watch great content in iPlayer via voice control alone.
LG Catalogue and Search: Having a strong, well-defined catalogue format and a means of integrating this with partners allowed the team to complete a VoD catalogue and promotions integration with the team from LG in a very compressed time period. This is what powers the row of links to content above the iPlayer logo on LG TVs and allows the LG voice search to find content in the iPlayer catalogue.

Digital Publishing

The Digital Publishing team work for the creators: the journalists, editors, curators, programme-makers, podcasters and more who make �� online. This year we’ve been working on how they do what they do, moving away from online’s late-1990s roots to ways that get the content they make to the right audiences, at the right time (and more cost-effectively, too).

We’ve built:

Passports, which rethinks how we describe the content we make. That means better recommendations, better curator tools, and a focus on how the audience discovers content. Looking for relaxing music in Sounds or good news stories? Passports gets it to you.
Curation tools to allow editorial to showcase content. Alongside pan-�� Topics, this lets us builds collections that span the ��’s services, and will soon be used in ��page.
“Optimo”, a tool for writers that moves beyond the classic 500-word story and supports the articles audiences are asking for. It’s starting in �� World Service, and will be a big focus in 2021.
Signal Box, which brings together production data (things like ‘how much content did we make?’) with content data (‘what was it about?’) and audience data (‘how many people consumed it?’) in one place. That answers the questions our content teams need to ask.

Technology Strategy & Architecture (TS&A)

Technology Strategy & Architecture is a community of technologists and architects who help ensure the �� makes technology decisions that provide the best value and most effective technology systems to deliver world class services to its audiences. It’s 8 different teams cover areas such as security, connectivity and digital products and represent disciplines from data science to broadcast engineering to architects who oversee the infrastructure of the systems we all use. Below is a snapshot of what some of the TS&A teams have been working on in 2020:

�� Blue Room

The Blue Room is the ��’s consumer technology research lab. The team offers access, insight and guidance on consumer technologies, new experiences and changing behaviours, through its Blue Room demo spaces across the UK and ‘on the road’ at public and industry events. This is what the Blue Room team has been up to in 2020:

Launched ‘Blue Room in Zoom’ and delivered 400+ Blue Room tech insight sessions remotely to over 3,500 �� colleagues across the world.
Demonstrated at on-line external events including �� Digital Cities and Sounds Amazing 2020.
Presented ‘Synthetic Celebrities’ at the ��’s first ever virtual Creative Diversity Festival (CDX), giving audiences a glimpse of the future, with demos of the latest in synthetic media.
Opened the ��’s Innovation Showcase to bring together some of the best of the ��’s emerging tech, including exciting innovations in augmented reality, voice and immersive listening.

�� Datalab

�� Datalab is responsible for using machine learning to provide great recommendations across the ��’s product portfolio.

This is what �� Datalab have been up to in 2020:

Significantly progressed development of the ��’s in-house recommender systems and successfully launched a personalised recommender for �� Sounds, as well as recommenders for several World Service language sites and the new �� News app.
Facilitated another pan-�� effort to build on the ��'s Machine Learning Engine Principles, a framework for responsible development of technology.

�� Digital Ecosystem

�� Digital Ecosystem is a small team of architects working mainly with the �� Platform team to create foundational software capabilities for digital transformation. In 2020 the team have been focussing on:

Building a suite of advanced software services to make �� content media and metadata simple to access across the �� and for our Partners, allowing us to get the best value from our content for our audience.
Defining the shape and technology for the ��’s next generation data platform to help us understand how to serve our audience better.
Project Origin - a collaboration between the ��, Canadian Broadcasting Corporation/Radio Canada, Microsoft and The New York Times to focus on one aspect of the disinformation challenge - content source verification. The goal of Project Origin is to apply clear signals to videos and pictures from publishers so you know the content is coming from where it says it is and has not been manipulated.

�� Product Architecture

�� Product Architecture helps �� teams to develop and deliver audience experiences across all surfaces and modalities. In 2020 the team has:

Provided architecture support and guidance to the �� Voice and AI team across client facing products, such as Beeb and the ��’s presence on Alexa, as well as efforts such as Songbird, which brings Text to Speech capabilities to bbc.com’s The Life Project.
Successfully introduced the C4 Architecture Model to D+E teams for architecture visualisation and started to roll out Structurizr to evaluate the suitability of a tool in the production environment.

�� News and Weather Apps Team

We are responsible for building and maintaining the News and Weather apps, across Android and iOS, for UK and international audiences.

�� News app beta: We are reinventing the News app from the inside out to give it a fresh feel. This includes recommendations, new ways to showcase the top stories, short video clips and an article overview to present articles in a short, digestible way (pictured). The first version is being rolled out to both Android and iOS before the end of the year.
ABL: The Apps Business Layer is a backend for frontend technology that will produce app-specific data at its endpoint. This will make a faster, cleaner and more flexible app experience on the device with more work being done on the backend.
Weather app: We started work on an updated weather app with split screen, animated icons and a weather widget for iOS 14

�� Research & Development

This year �� Research & Development is 90 years old. We have played a huge part in broadcast technology innovations from Digital Television to Object Based Media. Whilst the technology evolves, our purpose hasn’t. We look forward 3, 5, 7 years. All the time asking how we use technology to keep public services relevant.

Here are some things we’ve been working on in 2020:

AI: Autumnwatch used R&D’s AI system - YOLO (You Only Look Once) to identify different animals captured by multiple cameras saving hours of production time.
5G: �� R&D, including IBC’s Accelerator programme, were integral to an early stage proof of concept experiment exploring cutting edge 5G remote production; and 5G Records a major EU project looking at professional production workflows across audio, video and immersive.
Sustainability: Our Sustainability team released research exploring the energy used to deliver and consume television and radio.
Streaming: We published the technical specification for Dynamic Adaptive Streaming over IP Multicast, which would enable the distribution of live television at scale over the Internet.
Sounds Amazing: Something that was better in Lockdown! We brought together producers, presenters, reporters and engineers to talk about what will be next and later for audio. Over 3000 experts attended this virtual conference from around the world.

And lots more - /rd

The lessons learnt creating a design system for �� Online

2020-11-23T10:33:48+00:00

Dave Morris and Josh Tumath are part of something called the Presentation Layer team; a team working on the design system and React front-end for the new �� website.

A short history lesson

�� GEL is the design framework for the �� and was one of the very first design frameworks to be published online back in 2010. GEL sets out guidelines to allow UX designers working on the different sections of the �� website to create a consistent user experience and visual identity. This was fantastic as a set of guidelines for designers and developers to follow when creating websites for �� Sport, �� News and �� Bitesize (to name just a few). Plus, all the other experiences for mobile apps and TV.

However, as Matt talked about in his recent blog post about moving to the cloud, we’ve been going through a cultural change in the ��. We’re not building separate websites under one domain name any more. Through the WebCore project, we’re now building one website. And one key part of that is the design system.

Design systems are the single source of truth for both the design and implementation of a digital product, whether that’s a website or app. GEL can now be injected into the design system instead of being left to interpretation by teams reading and digesting the guidelines.

While there are plenty of design systems which are only built for one brand, ours has to serve a wide variety of destinations at the ��. Destinations are the places which people come to the �� to visit and they have to feel different to each other. There are informative destinations like News and Sport but also more playful ones like C�� and CBeebies. This is tricky to do in a structured design system. And it’s something we’re learning to do best as we go.

Our design system is split into five areas:

Foundations. Basic tokens like font sizes and spacing units.
Components. These range from simple UI snippets like a button to complex ones like a navigation bar. Each component has accessibility and reusability considered from the first iteration.
Layout components. Special components with a specific job to arrange other components in a layout, like Grids, Sidebars and the humble Stack.
Levers. The theming system. There are three levers (so far) in the design system: the font palette, the core colour palette and the brand colour palette. We want more levers. More on this later.
Containers. The top-level components that assemble other components together and fill them with data to form a stand-alone experience that can be used on a page. For example, the Article container fetches the article content from the business layer, and assembles the Heading, Tag List, Rich Text and Image components to make an article.

Our community in �� Design + Engineering have been building up the design system for over a year now, and we want to share with you the lessons we’ve learnt in that year.

1. Make it as easy as possible to share

We’ve taken a very different approach to the GEL design system compared to others out there. Our team, the Presentation Layer team, are not the sole maintainers of the design system. We haven’t pre-anticipated requirements and made in-advance a library of components like buttons, input boxes, navigation bars and dialog boxes. We haven’t created the design system as a versioned library that developers need to pull into their project and keep up-to-date.

So what have we done instead? We’ve done three things:

The Presentation Layer, our React-based rendering layer for the new �� website, is a monorepo that includes the design system. Only web pages created in the monorepo can use the design system. And, most importantly, any changes to components in the design system are rolled out instantly.
Layouts, Components and Tokens are only created when they are needed. We typically wait until there are at least two use-cases for a component before creating it. We didn’t even have a Button component until a few months into the project!
Our team does not solely own the design system. Every team building with the design system jointly owns it. Working together as a community, we all build and make changes to components.

Each team contributes and reuses from the design system. It’s a give and take relationship

By doing these three things, every page is always using the up-to-date version of the design system and using it consistently. The design system does not stand alone. It’s built into the wider platform. So it’s not possible to build something bespoke to a particular team’s needs; everything can be shared.

2. Do things on a need to make basis

We intend to build the one �� website together because previous approaches have led to fragmented results. Therefore, a top down approach of ‘we build these things and you use them’ doesn’t play nicely with this way of doing things. It shouldn’t be down to one central team to create the components all teams will eventually use. Our product teams do their own user research, so they know much more about how the audience are using �� products and services than our team does. They’re the ones best placed to create new components and improve what we already have available.

We knew this approach was going to be difficult. Teams across the �� are used to working in silos with autonomy to create components and features to meet their own needs whilst using the GEL guidelines as a steer. We’re essentially pressuring a move from vertical silos to horizontal shared ownership through the design system. Any new additions to the design system should be considered by how anyone might end up using them.

Because of this culture change, we’re striving to keep the components simple to start with. Remember that this design system has to serve multiple destinations at the ��. If we start with something complicated then it’s only going to get more complicated and unmanageable by the time we’ve figured out how it can be used by everyone.

Keeping things simple means to avoid making components until they’re needed. In some cases it also means to not make them with all the bells and whistles until it becomes a requirement to do so. This allows us to avoid complicated discussions and move forwards in the short term.

We’re there to join the dots between all those other teams creating stuff in the design system and offer guidance and support in building and improving components. We create the standards and principles for all teams so that we can make sure accessibility, reuse opportunities and consistency are at the heart of what everyone makes.

3. The reinvention of the wheel

Prompting teams to find and use solutions already in the design system can be challenging. It’s something we’ve had to tackle whilst teams are getting acquainted with the design system.

There are some products at the �� with well-established online experiences which are now rebuilding in the cloud using the design system. This can bring along with it a like-for-like mentality. Some teams (but not all) expect to create a carbon-copy of their existing experience using the design system.

There’s a whole array of articles out there which cover a topic of “what happens when a design system gets you 90% there” — these are our experiences on the subject. We know design systems are fundamentally driven by re-use. Creating something new which is similar to another pattern or component inside the library can add unnecessary weight, take up lots of designers and developers time in creating those new components and cause confusion in the future with those looking to find the right component for their needs.

We’ve been building things in isolation for years at the ��, it’s sometimes hard for teams to think outside of their silo and horizontally about what the business needs in a component library. The design system, and the Presentation Layer Team are now influencing teams to think more horizontally.

A simple tactic we employ in these circumstances is to ask the teams “what problem are they trying to solve?”. This flips the priority of “we need this” to “a user needs to be able to do this”. We try to approach each new request for a component with a re-use first way of thinking. By doing this, we’re setting an example for the rest of the teams building using the Design System.

We can then have a conversation around the existing components available and potential iterations which all teams could benefit from. If we do find that there’s a requirement for a new component — we work with that team to build it with reuse considered. It’s easy to spot the same problems in different areas on our radar. We then join the dots, and get the relevant teams involved to help everyone work more collaboratively.

4. Be strict to allow for flexibility

To drive re-use, we need to be able to create components once; not multiple times for each of the different brands. So, to support the variety of destinations on the �� website, we came up with a theming system we call ‘the levers’. Anyone creating a page can choose from a set of font combinations and colour swatches to theme it. And each component will work with all of them out-of-the-box.

A theming system might not seem like a ground-breaking idea, but it’s pretty powerful when �� News leadership ask, ‘When can you get this video player page on the �� Sport website ready to use on News as well?’ and we call pull some levers and say, ‘Hey presto – it’s already done!’

Just pull some levers and hey presto — you’ve got a website! (Provided you plug in the right data sources for the content too)

To make this work, however, we needed to be strict about what sets of colours and fonts are allowed to be used in Components. We had to resist the temptation to use our own colours in specific places.

Colours and fonts are given ‘semantic’ names based on where they’re intended to be used, rather than what they look like. For example, the text for heading text is called ‘primary’ and the colour for borders is called… you guessed it… ‘border’!

That doesn’t mean they’re set in stone, though. We’ve already had to add new colours and change them as we get newer use-cases, and we hope to add more levers in the future too. How great would it be to have different levers to change the type of motion applied to an experience? Or to help us create fantastic children’s experiences where there may be crazily different requirements for a website to cater for those types of audiences.

5. Documentation is hard, but important

One thing we always worry about is our documentation. We have a lot of it: accessibility guides, design guides, development guides, tutorials, component docs, principles, values…. And to make matters worse, they’re all over the place: on our internal wiki, in our cloud storage, on GitHub and in Storybook.

Storybook catalogues our reusable components and contains other documentation

Building a successful Design System and platform hinges on the quality of the documentation. It needs to be kept up-to-date. It needs to be discoverable. It needs to be easy to navigate.

Just like with components, documentation can start off simple but grow unwieldy as people add to it bit-by-bit. Taking the time to audit it, get feedback and clean it up is really important. That’s something we’re doing at the moment.

The Presentation Layer team duties are part design system curation and part support team. Having a complete and well organised set of documentation means we can run a tight ship. It helps us to automate a lot of support requests for both designers and developers.

6. Design for longevity through obsolescence

Finally, the Design System has got to be fit for the future. By designing our content and layouts as components, it’s easy to rip them out and replace them when the time comes. And we shouldn’t be afraid to do that!

Intrinsic Web Design is becoming a hot topic in the world of web design. As we start to embrace newer features of CSS like Grid and intrinsic sizing units, it will be easy for us to swap out our existing layout components with something more intrinsic.

We hope to talk more about Intrinsic Web Design in a future blog post.

============================================

It’s been over a year of designing, building and learning as we go. Above all, we’ve learnt that we need to be flexible as we work together with everyone from different teams. The progress that’s been made so far has been brilliant, and we’re excited about building on the foundations we’ve made to make experiences that our audiences will love.

This article is also featured on �� GEL and the �� Design + Engineering Medium blog.

Moving �� Online to the cloud

2020-10-29T10:56:20+00:00

This is the first in a series of posts in the coming weeks about how �� Online is changing, making use of the cloud and more.

Over the past few years, we in the ��’s Design+Engineering team have completely rebuilt the �� website. We’ve replaced a site hosted on our datacentres with a new one, designed and built for the cloud. Most of the tools and systems that power the site have moved too. We’ve used modern approaches and technologies, like serverless. And we’ve refreshed the design, approach, and editorial workflow, ready for the future. Hundreds of people have been involved, over several years. And it’s just about complete.

This post is the first of several looking at the what, the why, and most importantly the how: the approach we took to creating a new site that’s ready for the future. Delivering quality technology change, quickly and effectively.

Examples of pages that have been recreated on the cloud in the past year

Context

The �� website is huge. Over half the UK population use it every week. Tens of millions more use it around the world. It has content in 44 different languages. And it offers over 200 different types of page — from programmes and articles, to games and food recipes.

As is the case with tech, if you stand still, you go backwards. Until recently, much of the �� website was written in PHP and hosted on two datacentres near London. That was a sensible tech choice when it was made in 2010; but not now.

The ��’s site is made up of several services (such as iPlayer, Sounds, News and Sport). For them all, we need to ensure they use the latest and best technology. That’s the only way to ensure they’re the best at what they do. They all need to be hugely reliable at scale, as well as fast, well-designed, and accessible to all.

And so, over the past few years, it’s been our strategy to recreate �� Online. Almost every part has been rebuilt on the cloud. We’ve taken advantage of the many benefits that the cloud brings — such as the flexibility to provision new services almost instantly. And we’ve used best-practice tools and techniques — such as the React framework, and the DevOps model. We’ll look more at the approach in a moment, but first, let’s discuss the underlying principles.

Principles

Rebuilding a massive website could easily suffer from the Second System Effect. It’s all too easy for new projects to be over-ambitious with the requirements and the approach. A push for perfection makes it tempting to pick the most sophisticated solutions, rather than the simplest. We needed to prevent this, to ensure good value and delivery at pace. Here are some principles that helped us do this.

1) Don’t solve what’s been solved elsewhere

When building something as large as �� Online, it might be tempting to consider everything from scratch. Doing so provides the most control, and leaves no stone unturned. But the cost in doing so can be huge. An off-the-shelf solution might only give you 90% of what you want, but if can be delivered in 10% of the time, it’s probably a worthy trade-off. This applies to tech, UX, business analysis, and pretty-much everything else. Most problems have already been solved somewhere else; so don’t solve them again.

A great tech example of this is the use of serverless. Around half of the ��’s website is rendered serverlessly with AWS Lambda. Managing virtual machines (or containers) is expensive — keeping them secure, reliable and scalable takes time. Serverless mostly solves that problem for us. And problems solved elsewhere mean we shouldn’t do them ourselves.

2) Remove duplication (but don’t over-simplify)

When there are many teams, duplication is inevitable. Two teams will each come across the same problem, and create their own solution. In some ways this is good — teams should be empowered to own and solve their challenges. But left unchecked, it can create multiple solutions that are incompatible and expensive to maintain.

As we’ve rebuilt �� Online, we’ve removed a lot of duplication and difference that has built up over the years. Multiple bespoke systems have been replaced with one generic system. It’s a double win, because as well as being more efficient (cheaper), we can focus on making the new single approach better than the multiple old approaches. It’s because of this that the �� website now has better performance and accessibility than ever before.

However, we must be wary of over-simplification. Replacing multiple systems with one looks great from a business point-of-view. But software complexity grows exponentially: each new feature costs more than the previous one. There reaches a point where two simple systems are better than one sophisticated system.

As an example, we chose to keep the ��’s World Service sites separate from the main English-language �� site. The needs of the World Service (such as working well in poor network conditions) were specialist enough to warrant a separate solution. Two simpler websites, in this case, are better than one complex site.

The rendering of the English-language site (left) and World Service site (right) is kept separate, to not over-complicate one solution

3) Break the tech silos through culture & communication

Creating a new �� website has involved many teams. To be successful, we needed these teams to align and collaborate more than we’ve ever done before. Otherwise, we could easily create something that’s less than the sum of its parts.

It’s hard to overstate the value of communication. Without it, teams cannot understand how their work fits alongside that of other teams. And without that understanding, they cannot see the opportunities to share and align. Teams may even start to distrust each other.

Communication brings understanding, and that allows the culture to change. Instead of teams doing their own thing in isolation, they naturally share, collaborate, and flex to each other’s needs. They go beyond what is strictly their team’s remit, knowing that others will too. Which ultimately makes a better solution for everyone.

Over recent months I’ve heard teams say things like ‘that other team is busy so we’re helping them out’, or ‘we’re aligning our tech choice with the other teams’. It’s a level of collaboration I’ve not seen before. By understanding the bigger picture, and how everyone plays their part, we’ve created a level of trust and alignment that’s a joy to see.

4) Organise around the problems

This ‘onion diagram’ helped explain how common concerns & capabilities (in the centre) should be solved once. This frees up teams to work on the more specialist capabilities in the outer rings.

Even with great culture and communication, multiple teams won’t necessarily collectively come together to build the right thing. This is, of course, the much-quoted Conway’s Law.

“Organizations which design systems […] are constrained to produce designs which are copies of the communication structures of these organizations.”
Melvin Conway

The �� has historically had separate websites — for News, for Sport, and so on. Each one had a separate team. To change that, and build one website, we needed to reorganise. But how? One gigantic team wouldn’t work, so instead we split teams into the most efficient approach. We created teams for each page ‘type’ — a home page, an article page, a video page, and so on. We also created teams to handle common concerns — such as how the site is developed and hosted. Altogether, it minimised overlap and duplication, and allowed each team to own and become expert in their area.

5) Plan for the future, but build for today

When designing large software systems, we’ve a tricky balance to find. We must plan for the future, so that we meet tomorrow’s needs as well as today’s. But we also don’t want to over-engineer. We cannot be sure what the future will bring. Requirements will change. Cloud providers will offer new technologies. The rate of change in the world — particularly the technology world — is higher than ever before.

There is no substitution for good analysis, planning, and software design. Researching the opportunities and choices available is key in ensuring a project sets off in the right direction. But we must resist the danger in over-thinking a solution, because it may be for a future that never comes.
The beauty of agile software development is that we can discover and adapt to challenges as we go along. Business plans and architectures need to evolve too, based on what we learn as the project evolves. Don’t solve problems until you’re sure they’re problems you actually have.

6) Build first, optimise later

Expanding on the above, we must be careful not to optimise too soon. Large software projects will inevitably run into performance issues at some point. It’s super-hard to predict when and how these issues will appear. So, don’t. Use the benefit of agile development to respond to performance issues only when they become real.

As mentioned above, much of the �� website now renders serverlessly using AWS Lambda. At the start of the project, we were suspicious of how quickly Lambda could render web pages at scale. We had an alternative approach planned. But in the end, we didn’t need it. Performance using serverless has been excellent. By not optimising too early, we saved a huge amount of effort.

7) If the problem is complexity, start over

This principle is Gall’s Law:

“A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
John Gall

Removing complexity from an existing system is hard. And in our case, we had multiple complex websites that we wanted to combine. The collective requirements of these sites would overload any one system. So, we had to start again, going back to the basics of what common abilities were needed.

8) Move fast, release early and often, stay reliable

Part of the the monthly status report we make for the WebCore project.

Finally, a very practical principle: ensure you are able to move quickly, so that you can learn and adapt. Release early, and often — even if it’s only to a small audience. As discussed earlier, predicting the future is notoriously hard. The best way to understand the future is to get there quicker.

The counterargument to this is that change brings risk. And with a popular service like �� Online, reliability is critical. The �� has always had a strong operational process (including 24/7 teams managing services, and a DevOps approach to ensure those who develop systems are also responsible for maintaining them). We’ve continued to invest in this area, with new teams focussing on infrastructure, and on the developer experience (DevX). We’ll go into more details in a future blog post.

Smaller releases, done more often, are also an excellent way to minimise risk.

High level technology overview

Putting the above principles into practice, here’s a super-high overview of how the �� website works.

Let’s look at each layer.

Traffic management layer

First up, all traffic to www.bbc.co.uk or www.bbc.com reaches the Global Traffic Manager (GTM). This is an in-house traffic management solution based on Nginx. It handles tens of thousands of requests a second. Because of its scale, and the need to offer extremely low latency, it partly resides in our own datacentres, and partly on AWS.

For parts of our site, a second traffic management layer is sometimes used. (Internally, these are called Mozart and Belfrage). These services, hosted on AWS EC2s, handle around 10,000 requests per second. They provide caching, routing, and load balancing. They also play a key part in keeping the site resilient, by spotting errors, ‘serving stale’, and backing off to allow underlying systems to recover from failure.

Website rendering layer

The vast majority of the ��’s webpages are rendered on AWS, using React. React’s isomorphic nature allows us to render the pages server-side (for best performance) and then do some further updates client-side.

Increasingly, the rendering happens on AWS Lambda. About 2,000 lambdas run every second to create the �� website; a number that we expect to grow. As discussed earlier, serverless removes the cost of operating and maintenance. And it’s far quicker at scaling. When there’s a breaking news event, our traffic levels can rocket in an instant; Lambda can handle this in a way that EC2 auto-scaling cannot.

A new internal project, called WebCore, has provided a new standard way for creating the �� website. It’s built around common capabilities (such as an article, a home page, and a video page). It’s created as a monorepo, to maximise the opportunity for sharing, and to make upgrades (e.g. to the React version) easier. By focussing on creating one site, rather than several, we’re seeing significant improvements in performance, reliability, and SEO.

Spot the difference: comparing the older page (left) with the new cloud-based version (right). The numbers are Lighthouse performance scoring.

As discussed earlier, we’ve kept our World Service site as a separate implementation, so that it can focus on the challenges of meeting a diverse worldwide audience. (This project, called Simorgh, is open source and available on GitHub.) Our iPlayer and Sounds sites are kept separate too, though there is still a considerable amount of sharing (e.g. in areas such as networking, search, and the underlying data stores).

Business layer

The rendering layer focuses just on presentation. Logic on fetching and understanding the content is better placed in a ‘business layer’. Its job is to provide a (RESTful) API to the website rendering layer, with precisely the right content necessary to create the page. The ��’s apps also use this API, to get the same benefits.

The �� has a wide variety of content types (such as programmes, Bitesize revision guides, weather forecasts, and dozens more). Each one has different data and requires its own business logic. Creating dozens of separate systems, for each content type, is expensive. Not only do they need the right logic, but they also need to run reliably, at scale, and securely.

And so, a key part of our strategy has been to simplify the process of creating business layers. An internal system called the Fast Agnostic Business Layer (FABL) allows different teams to create their own business logic without worrying about the challenges of operating it at scale. Issues such as access control, monitoring, scaling, and caching are handled in a single, standard way. As per our principles, we’re making sure we don’t solve the same problem twice.

Platform and workflow layers

The final two layers provide a wide range of services and tools that allow content to be created, controlled, stored and processed. It’s beyond the scope of this post to talk about this area. But the principles are the same: these services are moving from datacentres to the cloud, tackling difference and duplication as they do, and ensuring they are best-placed to evolve as �� Online grows.

On to the next chapter

So, �� Online is now (almost) entirely on the cloud. And it’s faster, better and more reliable as a result. We’ve discussed the key principles on how we made that change happen. And we’ve seen a summary of the technology used. More on that in later posts.

Most excitingly, this isn’t the end; just the start of something new. The tech and culture ensure we’re in a brilliant place to make �� Online the best it can be. And that’s what we’ll do.

I’m massively proud of all we’ve achieved — to everyone involved, thank you.

Around the world with TLS 1.0 (Part 2)

2020-05-18T08:27:23+00:00

Following a twitter conversation which was initiated by Hanno Böck, asking about experiences with disabling TLS1.0 and 1.1, I committed to writing an update on my late 2018 blog post, “Around the world with TLS 1.0”. This is that update.

I’ll keep this post brief and aim to keep the comparisons pretty direct. If you haven’t already, I’d recommend reading Around the world with TLS 1.0 for context and methodology. Let’s dive in…

Global view

First of all, I looked at our “global view” of TLS usage. This covers TLS usage on www.bbc.co.uk and www.bbc.com from every country we served:

November 2018 (original) data

TLS Version	Number of requests	Percentage
TLSv1.2	2,002,516,373	97.96%
TLSv1.1	4,529,764	0.22%
TLSv1.0	37,160,210	1.82%

February 2020 data

Context: We have two traffic edges currently (one of which replaced the traffic edge in the 2018 data), one for UK and mainland Europe (which supports TLS1.3), another for “rest of world” (which does not yet support TLS1.3)

UK, mainland Europe & “rest of world”:

TLS Version	Number of requests	Percentage
TLSv1.3	1,163,496,361	48.45%
TLSv1.2	1,218,683,970	50.75%
TLSv1.1	901,942	0.04%
TLSv1.0	18,164,567	0.76%

This shows a ~68% reduction in TLS1.0 usage globally over the 15 months or so between the two datasets. That’s pretty significant and is more than I had expected.

Incidentally, if we look exclusively at our UK/mainland Europe traffic edge (where TLS1.3 is enabled) we see ~69% TLS1.3 — so the adoption rate is strong:

TLS Version	Number of requests	Percentage
TLSv1.3	1,163,496,361	69.07%
TLSv1.2	506,655,701	30.08%
TLSv1.1	500,971	0.03%
TLSv1.0	13,879,940	0.82%

Per-Country view

Let’s examine how TLS1.0 usage has changed on a country-by-country basis. Again, we’ll find the percentage of HTTPS requests which used TLS1.0 for countries which made ≥ 10,000 HTTPS requests over 3 days. I’ll represent this as a comparison view for simplicity:

Country	Num requests (Nov. 2018)	% TLS 1.0 (Nov. 2018)	Num requests (Feb 2020)	% TLS 1.0 (Feb 2020)	% reduction
Bosnia and Herzegovina	35,031	100.00%	418,582	0.90%	99.10%
China	2,261,506	86.93%	2,549,943	19.79%	77.24%
Montenegro	28,712	48.74%	193,059	0.61%	98.76%
Croatia	113,948	43.75%	1,210,262	7.79%	82.19%
Uganda	150,225	34.48%	1,619,262	6.22%	81.95%
Honduras	97,644	29.55%	916,586	6.77%	77.10%
Ethiopia	180,473	26.04%	2,186,672	6.67%	74.38%
Democratic Republic of the Congo	12,775	25.67%	138,347	3.80%	85.20%
Nigeria	1,224,923	25.13%	9,621,049	8.08%	67.84%
Cote d'Ivoire	14,717	23.68%	170,716	8.11%	65.74%
Myanmar	164,751	21.25%	2,333,043	1.53%	92.80%
Hungary	175,327	20.20%	4,042,959	0.15%	99.24%
Cameroon	11,618	15.02%	217,951	6.87%	54.29%
Tanzania	76,469	14.93%	4,874,370	7.17%	51.95%
Somalia	189,509	12.98%	1,236,812	2.58%	80.12%
Sudan	16,273	12.93%	517,011	6.73%	47.92%
Mozambique	10,348	12.39%	228,480	3.31%	73.28%
Taiwan	195,132	11.01%	5,991,350	3.68%	66.55%
Zambia	29,070	10.41%	902,829	2.36%	77.31%
Morocco	32,932	10.04%	1,998,655	2.81%	72.03%
Uzbekistan	17,135	9.38%	1,270,560	2.46%	73.74%
Japan	489,215	9.15%	14,841,878	1.33%	85.44%
Hong Kong	426,542	8.97%	368,286	2.43%	72.91%
Algeria	24,760	8.97%	78,643	5.59%	37.65%
Romania	62,019	8.79%	52,821	1.78%	79.75%
Zimbabwe	19,253	8.15%	12,272	1.90%	76.69%
Egypt	52,061	7.60%	189,551	2.72%	64.21%
Turkey	234,372	7.32%	185,453	1.56%	78.69%
Philippines	94,536	6.95%	81,734	2.09%	69.93%
Ghana	44,913	6.71%	24,535	1.09%	83.76%
Belarus	28,211	6.68%	9,250	0.73%	89.07%
Kenya	73,939	6.39%	48,674	1.31%	79.50%
Nepal	38,569	6.00%	9,477	0.36%	94.00%
Bulgaria	27,659	5.96%	5,952	0.36%	93.96%
Malawi	15,501	5.85%	8,170	2.03%	65.30%
Jordan	13,419	5.73%	9,279	0.74%	87.09%
Indonesia	119,720	5.40%	63,831	0.98%	81.85%
Ukraine	86,505	5.35%	66,016	0.62%	88.41%
Republic of Korea	83,370	5.33%	42,123	0.98%	81.61%
Saudi Arabia	79,834	5.21%	108,438	1.54%	70.44%
				Mean reduction	76.97%

This shows some even more significant reductions in TLS1.0 usage for some countries, the mean reduction being ~77%.

Some interesting observations from these data:

Hungary has both the largest reduction (99.24%) and the lowest percentage (0.15%) usage of TLS1.0
Algeria saw the smallest reduction in TLS1.0 usage, at 37.65%
China has the highest percentage usage of TLS1.0 at 19.79%

Let’s update our view for the UK and USA against the 2018 data:

Country	Num requests (Nov. 2018)	% TLS 1.0 (Nov. 2018)	Num requests (Feb. 2020)	% TLS 1.0 (Feb. 2020)	% reduction
Great Britain	23,778,043	1.43%	9,288,530	0.71%	51%
USA	2,373,620	1.47%	1,557,219	0.40%	72%

This is interesting in its own right, both the UK and USA have smaller (albeit it only a little smaller for the USA) reductions than the mean from the “2018 worst offenders” list, above. This is perhaps because the UK and USA have a smaller base of real users on TLS1.0, with more usage being “is the internet working” checks running on old platforms, corporate proxies etc. (we seem to be used for lots of these sorts of tests, which is hopefully a compliment!).

It’s worth updating the countries which have the largest percentage usage of TLS1.0 — the list above was the “worst of” 2018. Here’s the top 10 countries with the highest percentage of TLS1.0 usage in Feb. 2020:

Country	Number of requests	Percentage of TLS 1.0 usage
United States Minor Outlying Islands	389,725,509.	100.00%
Antarctica	4,979,351	100.00%
Kosovo	276,524	100.00%
Niue	12,758,637	100.00%
American Samoa	5,063,507	100.00%
Christmas Island	1,633,591	100.00%
Mayotte	8,590,803	100.00%
Svalbard and Jan Mayen	998,549	99.99%
Pitcairn Islands	425,550	99.98%
Tuvalu	5,770,681	99.98%

Yikes, lots of countries with 100% (rounded to 2 DP) TLS1.0 usage. It seems that most of these countries are relatively small (in comparison to the “worst offenders” in 2018) so maybe the above is the result of one or a few legacy systems in each country/territory.

Clients

As in 2018, it’s useful to know what is making all these TLS1.0 requests. The table below is slightly improved over the 2018 data (please see the original post for info). These data are global and show the top 10 by “Operating system” and “User Agent” fields which are parsed from the User Agent request header as a normalisation stage:

Operating system	User Agent	Percentage of TLS 1.0 usage
Unknown	Unknown	38.02%
Android 4.2.2	Android Browser 4	2.54%
Windows 7	IE 7	2.30%
Android 4.4.4	Unknown	2.03%
Windows 7	IE 9	2.02%
Android 4.4.2	Android Browser 4	1.97%
Android 2.3.6	Android Browser 4	1.93%
Mac OS 10.11.6	Chrome 53	1.85%
Windows 8	Firefox 16	1.80%
Unknown	WebKit 533	1.77%

“Unknown” means that the parser library doesn’t know what the Operating System / User Agent is — either because it’s uncommon or ancient! What we see here are very outdated Operating Systems and User Agents — essentially these seem to be combinations of:

Old Operating Systems with old TLS stacks and User Agents which use the Operating System TLS stack
Old User Agents with old TLS stacks which don’t use the (sometimes more modern) Operating System TLS stack

The top 10 User Agents whose Operating system and User Agent are both unknown are:

Nokia6280/2.0 (03.60) Profile/MIDP-2.0 Configuration/CLDC-1.1
CITRIXRECEIVER
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
SGOS/6.7.3.9 (S400–30; Proxy Edition)
Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)
Dorado WAP-Browser/1.0.0
Mozilla/4.0 (ISA Server Connectivity Check)
ProxySG Appliance
WinampMPEG/2.00

So yep, as expected, generally ancient User Agents and the usual suspects. Most notably, it appears we have essentially fewer “real” (as in “used by people”) User Agents which negotiate TLS1.0, leaving a higher proportion of TLS1.0 usage from what appear to be automated systems. This makes sense if you consider the changes in Operating systems over the 15 month span between my two datasets — Windows 10, for instance, has gone from around 38% to 57% (desktop) market share (largely replacing Windows 7) and brings with it a much more modern TLS stack. Similarly, many users will have upgraded mobile phones, tablets and other devices.

Conclusion

TLS1.0 has seen a significant reduction in usage of around 77% for our audiences over the 15 months since I wrote the original blog post but usage of TLS1.0 in some geographies remains stubbornly high. The trend is clear though, TLS1.0 usage is absolutely on the wane and whilst the long tail of this usage will undoubtedly drag last for years, usage patterns are moving in the right direction (at least in our audience).

We operate with a single edge configuration (in terms of TLS) around the world so we need to take a decision on when the right time to remove TLS1.0 (and 1.1) support is — balancing the security risks against the hard cut-off for users. Something we have put some thought into is a mechanism for warning our audience of such breaking changes — we’re not there yet with it but it’s definitely something I’d like to have as a deprecation process which aims to inform the end user and ideally, show them a workable upgrade path so they can continue to use our services, if they so choose.

Let me know if you have questions or would like more detail on an element shown here and I’ll do my best to get you the information. Please either leave a comment below or send me a message on Twitter.

An accessible digital �� – 2019 in review

2020-01-15T10:19:46+00:00

�� for Everyone – take 1 clapperboard

The �� Accessibility Champion Network is probably one of the oldest in the industry. This year it surpassed 200 champions. There is also an alumnus of 75+ now encouraging inclusive design and accessibility elsewhere in the industry. Here are the top highlights of many accessibility improvements in �� digital products this year:

UX&D and Internal Tools:

The ‘�� for Everyone’ video was published by GEL, along with draft technical docs. Neurodiversity won awards. Internal S&SD did screen reader user research for Optimo, a new article editor for News journalists. They also reviewed and improved accessibility of Audiences, the new site for audience research, reports and data.

iPlayer and Sounds:

The web media player introduced subtitle size controls. iPlayer added an ‘Accessibility Help’ link for assistive technology users. iPlayer and Sounds both made efforts to respect an individual’s 'reduce motion’ choice, which can be set in their OS preferences. iPlayer Mobile adopted an ‘accessible first’ mantra to work on a new player interface, available to opt in and try via the app settings. Work has included research with screen reader users and raising issues with OS providers. Behind the scenes work was completed that enabled both iPlayer TV and the new mobile player to add in-player controls for audio description and sign language. iPlayer and Sounds both have active accessibility guilds, and have integrated accessibility into processes, engineering methods, and testing strategies.

iPlayer Accessibility Help link

Children’s Games & Apps:

Research relationships continued with Seashell Trust and Everyone Can. It enabled and supported user research for CBeebies, C�� and Bitesize games and interactive content, in turn guiding work on relevant content and the ‘Accessible Games Guidelines’. Bitesize Primary
improved keyboard accessibility across their existing practice games. They also launched 7 new games, with 8 more in development, that deliver on at least one accessibility ‘pillar’.

News:

News index pages became more consistent, easier to navigate, and introduced the �� Reith font, which is clearer and easier to read on small screens. World Service sites are being re-built with accessibility embedded at every stage. This has extended the champion network outside the UK. Accessibility swarms checked new components built for elections. And Visual Journalism considered accessibility from the start of each project. A scrolling story format that is keyboard accessible and has a toggle for animation is one example of the outcomes from that. The integrated approach shows how Catharine Green becoming Accessibility Lead was a decision that established accessibility as a clear priority for News.

Sport:

Following the example of News, Sophie Beaumont became Accessibility Lead for Sport, and an Accessibility Guild was set up. They embedded accessibility considerations into all team processes and built up a prioritised backlog of improvements. The teams have since created new pages and features that are some of the most accessible Sport experiences yet.

Weather:

A redesign of tide table pages for Coast and Sea section means they are now responsive, more legible and ‘glanceable’, and adapted for touch devices. The improved navigation for future dates also links to an accessible table version of the information with live updates provided to assistive technology. And animated indicators update in real time while adhering to an individual’s ‘reduce motion’ preferences.

Web Core:

This year a big change of direction toward a centralised approach took shape. The goal is a shared, non-versioned kit of parts, from which to create digital products and services. The team are building brand new components following the GEL technical docs. The docs provide pragmatic advice for actual implementation with accessibility built in. This approach should also mean that future accessibility improvements roll out fast.

Voice & AI:

New tech on the block means a lot of user research. This has ensured the ��’s collective Alexa skill is intuitive to navigate while reducing distraction and cognitive load. And it’s informing work begun on ‘Beeb’, a voice assistant from the ��. Accessibility is a key requirement of both the user interface and brand development.

�� Studios:

The commercial arm of the �� may need a slightly different approach, but accessibility is still a key requirement. A redesign of Top Gear has provided the perfect opportunity to educate the team and make accessibility part of the end-to-end process of delivery.

Elsewhere:

Small changes, like improving link text in the cookie banner or checking the accessibility of a new local search feature before implementation, can ensure a better experience for many people. Though our champions don’t keep to small. Many blogged and presented this year, including Peter Gould and Nicola Quinn. They presented “How to Bake Muffins” at a #BuildThe�� event open to the public. A heart-warming tale about why accessibility should be considered from the start of a project. Peter also built taba11y in his spare time, a useful Chrome plugin already being used in-house.

taba11y enumerating tab stops

Coming Up

Most �� teams now ‘bake in’ accessibility. Look out for improved universal sign-in and children’s sign-in from �� Account team. And also improved audience participation experiences. Research is being conducted to support insights into how children with varied and complex impairment use technology. Other research is looking at how to make interactive content more accessible for neurodiverse users. And the new accessibility guilds and groups in Bitesize, iPlayer& Sounds, and Voice & AI should make an impact throughout 2020.

Conversations have definitely moved on from explaining ‘why’ accessibility is important. We now talk collectively about ‘how’ we deliver accessibility. A champion network that keeps accessibility part of the daily conversations is pivotal to that shift. The �� sets an accessibility benchmark that many organisations strive to emulate. No easy feat in the constantly changing and evolving digital space. An active network of Accessibility advocates, our Champions, Leads, Groups and Guilds, is what makes that possible.

Understanding public service curation: What do ‘good’ recommendations look like?

2019-12-17T13:24:25+00:00

Unless you know exactly what you’re after, finding the �� content that is exactly right for you can be a little like looking for a needle in a haystack. Some challenges we face at the �� are:

1. We produce a vast quantity of content, at a rough estimate, around 2000 pieces of new content a day. In terms of our ‘shop window’, we have a good number of promotional areas and slots – on homepages, in our apps, on our schedules and on our social media feeds – but simply not enough to accommodate all our content.

2. When content makes it to a prominent promotional slot, it will, of course, get good traffic, but there is no guarantee that the person for whom that content is most relevant will see it at the time. And once that content loses its place it will be hard to find, indeed, no one will know it exists unless they are armed with determination and the greatest of digital search skills.

3. Colleagues working in curation have an incredible ability to seek out the best content and ensure that it is labelled in a way that maximises its impact, but they simply can’t read, watch or listen to all of our output. There is just too much of it.

So far, so familiar.

Personalised recommendations, fuelled by the power of machine learning, are what every forward-thinking media and tech organisation is doing to put the content that audiences would most enjoy right in front of them. The �� has automatic recommendations in Sounds , on iPlayer and on some news language services like �� Mundo, which use strategies like content similarity, popularity and collaborative filtering.

The curatorial challenge for the �� becomes more interesting and complex, because our guiding principles relate so strongly to delivering public service value to the audience. We cannot simply work out what would get the most clicks, and show that to our audiences (although ensuring content is attractive to our audiences is important). Instead, we are required:

To provide impartial news and information to help people understand and engage with the world around them
To support learning for people of all ages
To show the most creative, highest quality and distinctive output and services
To reflect, represent and serve the diverse communities of all of the United Kingdom’s nations and regions and, in doing so, support the creative economy across the United Kingdom
To reflect the United Kingdom, its culture and values to the world

These requirements, written into our Royal Charter are why editorial values are woven into our Machine Learning Engine Principles and why those working in machine learning work closely with editorial teams. I recently conducted a deep exploration of what public service curation means to an organisation like the ��, so that we can begin to identify some of the signals that can help us as we build next level public service recommendations services.

Around 80 editorial staff, involved in some way with digital curation or content creation, took part in the discussion. Around 40 Design & Engineering colleagues observed those discussions, mostly those working in areas related to data science and engineering, metadata and research. Five criteria for public service curation emerged:

1. We want our content to reach and engage as large an audience as possible. We have a role in the national conversation by bringing the most important and resonant stories of the day to the attention of our audiences, as well as prioritising content, that is popular and has universal appeal for the greatest number of people. Examples of this might include content that brings people together like Strictly Come Dancing or content of global importance like Seven Worlds, One Planet, both of which have an impact on the national and cultural conversation. Popular and impactful content also represents good value for money.

2. In a seemingly contradictory move, we don’t always optimise for peak popularity. We make and promote content that aims to appeal to different audiences, groups, communities, regions and perspectives. We want to showcase content that feels personal. Take the podcast Netballers, which is about a sport with less national impact than, say, Premier League football. In terms of feeling relevant, it’s a win for women, young and BAME audiences. And it reflects an aspect of British culture. By its nature, it doesn’t get as many downloads as the Peter Crouch podcast, but it serves to inform and entertain those with a passion for netball.

3. We strive always to bring audiences something new - newly released music, new writers, new presenters, untold stories, news about events and stories of national significance, information about emerging technologies, research, discoveries, perspectives, major drama series, ways of storytelling. This quest for the new is the kindling that starts the national and cultural conversation in the first place. The �� produces a lot of this type of content, but it is not easily identified by engines which are based on collaborative filtering alone*.

4. We provide useful, helpful, practical information, explainers and fact checks, which makes us a trusted source of information: that could be a news story, or revision notes for a Chemistry GCSE, or recipes for a quick mid-week meal, or sports results.

5. We have enormous breadth and depth which means that there is something for everyone: there is variety by topic, tone, format, duration, location, level of expertise and age suitability and relevance.

Our long term ambition is to use this thinking to build recommendation systems which can broaden our audience’s horizons, providing different perspectives and stories and experiences that they might not otherwise have come across. But this is complex - and a recommender that can effectively deal with as multi-faceted an editorial issue as impartiality is extremely challenging and will take significant time to develop.

For now, curation in the context of our recommenders involves upholding our editorial values and finding ways to surface the most relevant and compelling content for each user. All the content the �� makes for our UK audiences, one way or another, is public service, so our recommendations will of course always have a public service flavour.

I’m excited that we have already built models with business rules about increasing breadth (in the Sounds recommender) and depth (in the World Service recommenders), and as well as reflecting editorial values around sensitivity. For example, on the Sport recommender, due for release next year, we’ve taken the curatorial decision that content from rival teams will not be shown together in a set of recommendations.

We can learn so much about public service by sharing the editorial point of view as we iterate and refine our approach. I’ll be working in collaboration with editorial colleagues and data scientists to ensure these public service curation criteria inform the ��’s future recommendations engines.

Lastly, we also recognise that the power of machine learning - of which recommendations is a part - can only get us so far. Machines cannot understand all the subtleties, complexities and nuance of editorial decision making. An algorithm will have trouble identifying what is entertaining or fresh or authentic without significant human assistance. A machine can help only up to a point to accurately tag content in a metadata system before a human verifies the machine’s choices and hits ‘publish’.

So at the ��, we’ll be maintaining a human hand in content creation and discovery. We need both humans and machines to best serve our audiences; editorial colleagues are highly skilled at making and promoting our content and machines can help amplify those skills. More and more, what recommendations can do is help locate the needle in the haystack in the first place - and exactly the right needle for you.

*Which is why we have built a factorisation machine for our Sounds recommender which combines collaborative filtering with content matching. For more detail see Developing personalised recommendation systems at the ��.

�� Online: 2019 in review

2019-12-12T09:38:21+00:00

A large number of teams within the �� work on various aspects of of our online services. Responsibilities span the entire stack, quite literally everything from low-level networking all the way through to audience engagement. In this post, we’ll give you a brief overview of what some of our teams have been working on in 2019. Each contributing team has written an introduction to explain their role and responsibilities, along with some highlights of their work this year. If anything in particular catches your interest, please leave a comment and we’ll get some follow up deep-dive blog posts written to give more detail.

Programmes Team

The programmes team is based in Broadcast Centre, White City, London, and manages /programmes, /blogs and /schedules, as well as tooling for ourselves and the editorial staff who publish to these platforms.

We finished off our migration from the ��’s on-premises infrastructure to AWS, including a complete rewrite of the codebase. This has resulted in render times being slashed down to 115ms on average.
To increase the reliability of /programmes we've implemented a stale-while-revalidate cache using Redis that compresses and stores all of our queries for at least 6 hours. This means that our application will be able to survive extended downtime from our data providers without a major interruption in service.
We moved from a Jenkins CI/CD environment we shared with a couple of other teams to our own Codebuild environment fully under our control in AWS. Build times and our automation tests generally are faster without having to queue to be run.
We added structured data to enable search engines to understand the complicated nature of /programmes content better, thus helping our end users find the right content faster through their favourite search engine.

Online Technology Group (OTG)

Website traffic management

Launched our new, in-house website traffic management (TLS termination, layer 7 routing, caching, origin HA) service in two UK PoPs (our largest audience) as part of our worldwide rollout, alongside the existing mainland Europe and Ireland PoPs.
Gradually enabling TLS1.3 on our website traffic managers, as platform technology allows.
Built “bq-logs”, a large-scale, low cost, auto-scaling, reliable, extensible system to receive, enrich and import our traffic management access logs (or any structured or semi-structured data) into Google BigQuery. This system requires no day-to-day management, currently processes around 1TB of logs every day and enables access to all �� online teams to the data it contains, maintaining access control over personal data.
Built a very simple, fast, low cost, auto-scaling Report URI and Reporting API endpoint which feeds reports into bq-logs (above) for CSP, NEL and similar reports.
Began deploying a consistent baseline of HTTP security headers, such as CSP, NEL, X-Content-Type-Options, X-Frame-Options, Expect-CT.
Deployed CAA records on bbc.co.uk, bbci.co.uk and bbc.com.
Enabled HSTS on the bbc.co.uk and bbc.com apexes.

Media Distribution

Along with our colleagues in R&D, Media Services and iPlayer, successfully delivered the FA Cup Finals and all of Wimbledon 2019 in glorious live streaming UHD, to compatible smart TVs.
BIDI (our in-house media CDN) reached a milestone of serving 1 Terabit of iPlayer video on-demand traffic per second in 2019. BIDI is now running from 78 servers embedded within 15 data centres, across 4 ISPs.
New BIDI cache hardware model designed and validated, which provides 119% more throughput, 316% increased programme storage capacity and is 88% more power efficient at peak throughput. All whilst providing an enhanced quality of service to the audience and at a lower cost per unit to the ��.
Radix, our in-house media content origin platform has been refreshed. The new platform provides a better quality of service and increased capacity to meet the demands of iPlayer's on-demand catalogue for the next three years.
Software improvements helped reduce SSD drive write rate on cache servers by 75%, improving latency and increasing drive longevity.

Media Services

Media Services is the team responsible for audio and video workflows across the �� Online. We transcode and manage access to media, deliver video to linear playout systems, and develop the media players for Web and native iOS and Android. We've been making some subtle but important changes under the hood. Here are the highlights:

We improved our Audio Description workflow so that we can deliver audio described programmes to iPlayer before they are broadcast on linear TV. This meant that we could deliver the entire second series of Killing Eve with audio description, so visually impaired viewers could enjoy the whole series before broadcast like anybody else. One of our developers, Marcel, wrote a blog post with all the details here.
We integrated a new transcoding service from Bitmovin into our workflow that dramatically speeds up the delivery of content to iPlayer. Last year, for example, fans of Strictly Come Dancing had to wait hours after broadcast for the show to appear on some platforms: it took over 2 hours to transcode each episode. This year, we reduced that time to just 10 minutes by splitting the programme into small chunks and processing them in parallel. This is just one of several transcoding services we use to optimise our media transcode function for different scenarios.
"Content Enrichment" is a term that's being used a lot in planning meetings. If we can better understand our own content, we'll be able to provide better recommendations to the audience. One way we can do this is by creating transcripts of the audio and video we produce. Media Services has now launched a managed transcription service using �� R&D's tool “�� Kaldi” that will make it easier for other �� teams to transcribe their content at a fraction of the cost of other speech-to-text providers on the market. So far we’ve transcribed 75,000 hours of archive audio!
The Playback team have been busy. The web team rebuilt the ‘continuous play’ experience so that programmes and clips will automatically play in sequence. Alongside that, they commenced a major programme to rebuild the web media player from scratch in order to take advantage of new technologies available in browsers, such as Media Source Extensions, Progressive Web Apps and Service Workers. This also enable us to leave behind older or legacy technologies like Flash, iFrames and suchlike (noting that we will still support older browsers for as long as they are needed even as we move to the new player sometime next year). The new player will also sport the new design which we’re excited to roll out.
The Mobile playback team have been equally busy; we onboarded several new apps over the last year and have been working with them to enable full use of the players. Again, much work ‘under the hood’ is ongoing to maintain an uninterrupted delivery of �� content to those apps with a robust and reliable service.

iBL

iPlayer Business Layer is “The Beating Heart of iPlayer”. If content is the lifeblood of iPlayer, iBL is the team responsible for controlling its flow through the product. We achieve that by gathering the metadata, curation, business logic and algorithms in one place: we are iPlayer as an API.
Some of the highlights of what we worked on in 2019:

Curator DSL - In order to help our editorial teams with their work, we have developed a DSL for selecting a subset of episodes on iPlayer, which can be refreshed automatically rather than needing manual updating of groups. This is using existing commissioning information such as genre, but also analyses and labels episodes based on descriptions and other data - see also the content enrichment work Media Services is working on above.
Personalised Bundles - On our journey for iPlayer to be Total TV, we have introduced more personalised “Bundles” of content on the iPlayer homepage, such as the “If you Liked” bundle, which suggests shows based on your recent watching history and communicates that reason to you.
Content to Content Recommendations - We have taken some of the concepts �� R&D have developed for their Recommender System into production and running some of iPlayer’s recommendations using this system. Stay tuned for more in this space in the future!
Live Events - 2019 saw the return of Glastonbury after a fallow year, and the �� is committed to show much more live coverage across the board, including sports. This year, we have revamped our live event workflows after some user testing: From dynamically updating the titles and images on the playback screen, and on the homepage, based on what’s actually happening on stage or tennis court right now to grouping content into sensible slices, we’ve made many improvements to iPlayer, and won two UX UK Awards 🏆.
Full Series - Following recent Ofcom regulation changes we are expending the number of series available on iPlayer significantly, and are improving the ways in which iPlayer handles this. We allow browsing by series now, use better imagery and promotional materials for a whole programme rather than individual episodes, and we keep track of users’ watching progress and current episode.
Experiments - We have run dozens of API driven experiments in the live product this year, many of which resulted in changes to the audience experience. From re-ordering categories for an U13 (under 13) audience to a more useful way we generate recommendations for new users who don’t have a watching history, we hope that these under-the-hood improvements have made some people’s lives easier.
Scotland - Scotland got its own TV channel, and obviously this is featured on iPlayer.
Programme Publication Workflow - With the increasing size of the iPlayer catalogue, we’re undergoing a migration to an event-driven workflow to update available episodes to ensure everybody gets the latest Episode of Ru Paul’s Drag Race UK as quickly as possible.
Better client workflows - Our GraphQL APIs uses Persisted Queries, which means that if any iPlayer client would like to update what they query from the API, they need to register their new query first, we need to review it, and then add it to our API. This used to be a manual process, which we have now automated, allowing clients to update their queries in minutes rather than hours or days.
Operational Visibility - All the new services to deliver all the new features above have introduced some changes to our operational workflows. We have worked with the ��’s operations teams to allow them to handle most actionable events without needing to call out an engineer. Across the whole department and organisation we have also started to share a lot more background on incidents and how we can work together to learn from them.

iPlayer Web

The iPlayer Web team are responsible for building and maintaining the face of iPlayer on the web, across thousands of different types of device.

Analytics & Data - With audiences at the heart of everything we do, we need to ensure we can capture data around user behaviour, so that we can fully understand how users use the product. We have therefore increased our focus on understanding user data and insight, with more advanced implementations of analytics.
Childrens’ experience - iPlayer has lots of programmes but not all are suitable for our younger audiences! We have started creating a safeguarded and child-friendly experience in iPlayer Web, for those users who are signed in with an Under 13’s �� Account. We have run several experiments on this experience, for example, simplifying the user interface to see if young people find it easier to discover content. There are many more improvements to come, but to see our progress so far please check out how to register for a child account
End-of-playback experience - We have run over 10 experiments this year, around the end-of-playback experience. We’ve been trialling various changes to the user interface, to assist users in discovering new programmes that they’d like to watch.
Live Events - As mentioned by iBL, we designed and launched a brand new live events experience on iPlayer, making it easier to discover the various live streams available for users to watch at various live events such as Glastonbury and Wimbledon. We even won 2 UXUK awards for this new experience!
React Migration - We finished moving all of our custom client-side code to React components - the last things to move were the main iPlayer navigation and footer. This makes for a more consistent codebase, allowing quicker development and experimentation across the product.
End-to-end test infrastructure - We started making improvements to our end-to-end acceptance tests and infrastructure, with an aim of making tests quicker, more stable and run more frequently. By upgrading our testing framework and running tests in parallel, we saw time to run a full regression of the playback experience more than half, making it even quicker to ship new features to the audience!

Technology Strategy & Architecture

Technology Strategy & Architecture (TS&A) is a collaborative group which brings together expertise from across �� Design & Engineering. It’s the team that looks ahead and assesses what technology the �� will need in five years' time, and maps out a plan of how to get there; and ensures that we have the right priorities in place so we can digitally transform the ��.

The Blue Room ran a successful conference and fair in October with a focus on responsible use of Artificial Intelligence. 300+ internal and external guests took part and were offered a range of discussions on Machine Learning developments, creativity, regulation and ethics and invited to explore their hands-on technology fair. A summary of the conference, and an exhibition walk-through can be found here.
�� Datalab facilitated a pan-�� initiative to create the ��’s first Machine Learning Ethics Guidelines - specifically for the �� Recommender Engine project but with the expectation that they’ll be rolled out further to other machine learning initiatives.
Our substantial work on developing personalised recommender systems has started to take effect with the first in-house recommenders being deployed. You can read more here.
Following publication last year of a blogpost on the �� Capability Framework, developed by TS&A, the EBU asked to share the framework to support the creation of an EBU-owned model which could be adopted by its members. This July, the EBU technical committee formally approved the work to create an EBU media industry capability model. Read our follow-up blogpost here.
The �� Machine Learning Fireside Chats are now available as a podcast on Soundcloud. The ML Fireside Chats are friendly bi-monthly events to discuss and share knowledge in the ML space. To find out more, join our MeetUp group. To join the conversation follow us on Twitter: @��_MLFireside

RMS (Sounds)

RMS is the service that powers the �� Sounds experience across Mobile and Web. Our API provides a view into the content, and guides the experience that clients receive when using Sounds. Some of the work we’ve done (or are working on) this year includes:

Local Radio Stations in the Listen Live Rail - To help make Sounds more personalised and relevant to the audience and increase user engagement, we started displaying a Local Radio suggestion to a user. If a user has a postcode set and has enabled personalisation, we will find the nearest Local Radio Station to that user’s postcode and display it at the end of the Listen Live Rail, along with the National Radio Stations.
Shared, distributed cache to support Personalised Content - We’re investigating caching at different layers, and sharing the cache with other instances of the same service. In order to support more personalised content, we have started a long term investigation into new caching strategies. RMS serves hundreds of thousands of requests per day, and historically, most of those were for static or infrequently changing content that every user could consume. With the move towards more personalised content, we will have to change our caching strategy, as each user will receive a different experience. The variance of pages that could be cached grows massively with each piece of personalised content, and it is not practical to cache this in the way we’ve been doing it before.

Research & Development

R&D activity at the �� has been going on for almost 100 years. A specific department was established when the �� Research Department was formed in 1930. Doing research and development activity is Clause 65 of the �� Charter. In 2011 the �� Executive Committee’s paper interpreted Clause 65 to mean the following for the corporation’s research and development activity:

Drive Innovation in support of �� strategy and public purposes
Horizon scan, acting as early warning radar in the face of technical change
Provide an expert team to solve problems or maximise opportunities
Add tangible value to the wider media industry
Maintain �� partnerships, and with them, primary position in the wider ecosystem

Our 2019 highlights include:

�� Click used our Object-Based Media tool Storyformer to make their 1000th episode.
We introduced our personal data store prototype �� Box designed to give audiences control over their personal information.
Casualty audiences benefited from our new trial on Taster which made it easier for people with a hearing impairment to enjoy �� dramas the feature allowed audiences to set the audio mix to suit their needs.
R&D tested the use of 4G/5G broadcast technology to deliver live radio services to members of the public in Orkney. The island was chosen because it has limited fixed broadband, little or no mobile coverage and poor digital radio coverage.
Connected Studio, part of R&D launched Makerbox the destination for developers, producers and storytellers to use cutting edge �� R&D and third party originated tools to help them create new ways of telling stories to audiences.
We continued our High Dynamic Range trials and increased the jeopardy by tackling the 2019 FA Cup live.
The �� Proms was used to test Notes our new web app that sends you extra information about what you’re seeing and hearing to your device, as it happens.
We launched our Diversity & Inclusion policy on what speaker opportunities we will be involved with to help to improve diversity in the tech industry.

News and World Service (Web)

�� News and World Service online bring �� news content to the web both for our domestic UK audience and for our international audiences, around the world.

This year our main focus has been on moving large parts of our website to the cloud and to our new technology stack (Including the ~40 world service language sites). As well as delivering coverage of the December general election, and improving our existing web products.

Specifics:

Released a new open-source World Service platform that is “performance-first” to ensure that our users around the world are getting a first-class experience that caters for different needs
Continued to migrate from our legacy monolithic applications to a more microservices-oriented approach (mostly made up of Node.js/React and Ruby microservices on AWS).
Added support for personalised content on the News front page with top stories tailored to each user's preferred nation.
Developed a new Elixir-based “preflight” service that is responsible for the resiliency and availability of our web platforms.
Upgrading our existing Elections infrastructure and creating new components for general election coverage in the UK, India, Nigeria, Thailand, Indonesia, and Sri Lanka amongst others.
Using AWS Fargate ECS clusters to render images with R and publish directly to Twitter and other platforms.
Used natural language selection templates to create semi-automated editorial content in English and Welsh for the general election.
Moving our iframe services to Shadow Dom.
Working with our data science teams to deliver an API to provide content recommendations based on users recent article reading history.

Weather website

The �� Weather website serves an average of 8 million users per week, offering 14-days of hourly forecast information, warnings, interactive maps, and the latest editorial analysis and videos from the �� Weather Centre.

In 2018, we relaunched a significant portion of the �� Weather website on the cloud, expanding our use of AWS lambda to transform and render incoming weather data. Several features, however, remained on the ��’s outgoing on-premise platform. Our primary aim for 2019, therefore, was to complete our migration to the cloud, and in the process deliver a visually consistent and performant experience across the entire site.

Highlights include:

Automated UK Weather Warnings and Flood Warnings pages.
A refreshed Monthly Outlook.
Migration of our article, video and photo galleries to the cloud, here’s an example article.
Incorporating the ��’s new typeface, Reith, in line with our new TV graphics.
Showcasing Weather Watchers photos (user-generated content) on our UK forecast pages
New and improved Tide Tables information

In early 2020, we’ll release a new look for the remaining section of the site: our specialised forecasts covering Coast and Sea.

Search

The Search team runs the back-end services that power most of the search features on �� Online including the website, mobile, TV and voice applications. We’ve been busy completing our migration from the on-premise “Forge” platform to AWS, enabling improvements to our search features going forward.

Migrated search engines, services and data feeds to AWS
Created a content enrichment pipeline and associated editorial tooling on AWS
Started migration of search results pages to AWS
Invested in Relevancy Engineering skills

Navigation

The Navigation service provides the �� blocks, navigation links and footer on the website, as well as delivering the cookies/GDPR banners, analytics integration, the �� Reith font, etc.

Supported migration to the new �� website analytics system
Created a new EU cookie banner to allow our audience to opt in or out of the various cookies we use in �� Online
Supported insertion of GDPR banner in �� CoreServices/Webcore products
Improved click-per-view tracking support, enabling a feedback loop for improvements to Search etc
Created a replacement geolocation service

Application Security (AppSec)

Conducted a plethora of security architecture surgeries and threat modelling sessions in collaboration with �� Online teams as they develop or change their services
Re-launched the Security Champions forum in January 2019 for our online teams
Introduced an initial training pathway for �� Security Champions where new members have to take secure coding training, threat modelling training and data protection training in order to qualify as a formal �� Security Champion
Ran a pilot of a computer based training secure coding training tool and migrated to that from the previous classroom based course. This was successful so the platform was opened up to the entire �� Security Champion community as a part of their training pathway
Launched a newly developed threat modelling course and have trained 120 �� Security Champions across the year
Ran a red team testing event against �� iPlayer and �� Sounds in order to identify vulnerabilities in their codebase and to help productionise our red teaming capability
Hosted an OWASP Manchester vs Secure Code Warrior event at �� MediaCityUK
Launched a proof of concept deployment of OWASP DefectDojo, a vulnerability consolidation tool for �� Online teams and InfoSec
Launched a proof of concept deployment of OWASP Juice-Shop, fronted by Juicy-CTF, for the �� Security Champions to test their knowledge and skills in a safe environment.

So there you have it, a snapshot of what some of the teams in �� Online have been working on in 2019. To the best of my knowledge, this is the first time we’ve published a roundup and whilst not all our teams were able to contribute this year, I hope we’ll continue this each year and improve as we go.

As I mentioned earlier, if there’s something which particularly interests you or a question you’d like to ask, please leave a comment and we’ll do our best to cover them in follow-up blog posts.

Thank you to all the �� Online teams who contributed. All that’s left to say is that we hope you had a great 2019 and wish you all the best for 2020!

Mixital to close

2019-04-24T08:27:57+00:00

In 2015, we launched Mixital to help young people become digital makers. Brands from Doctor Who to EastEnders to Strictly got involved to inspire fans to make games, videos, animations, scripts and even dancing robots.

In all, more than one million new creations have been submitted to the �� using Mixital’s maker tools. And it has won or been nominated for six national awards – including a Children’s Bafta.

Its key success was in simplifying digital making to enable everyone – including those with only basic digital skills - to get creative with the ��. It meant audiences could engage in new ways with their favourite brands. And the brands themselves could showcase the creativity of their fans.

“Doctor Who is always looking for new ways to engage with its fans. We had nearly 100,000 original games submitted using the game maker kit that featured all the Doctor Who characters. And we’ve had thousands of scripts sent in too. It shows how creative our community is and how much they like to engage with us even when the show’s not on air,” said Matt Rogers, Digital Producer, Doctor Who.

The �� is now building on these learnings and exploring how we might best incorporate creative participation into what we do online. That means we’ll be closing down Mixital over the next year and helping our creative communities grow further on bbc.co.uk.

We’ll also be sharing the insights and learnings from Mixital with partners like Nesta and Freeformers who collaborated with us in setting up Mixital.

Mixital was launched as part of the ��’s Make It Digital campaign. It built on the insights from Nesta’s Next Gen. Report (2011) and Young Digital Makers Report (2015) that highlighted the growing skills gap in making with technology.

Digital technologies are rapidly changing how we learn, the skills we need and the future of work. It’s becoming increasingly important for young people to get creative with these technologies.

“Our research shows how vital digital skills and creativity are to the future of employment. And also the huge appetite young people have to make digitally. Mixital built on the work we did with the Digital Makers Fund and, in turn, we look forward to building further on its insights and learnings,” said Joysy John, Director of Education, Nesta.

The programme commissioned original research and worked directly with young people. The findings highlighted the need for easy-to-use maker kits and a safe, fun space to experiment, learn new skills, build confidence and showcase submissions - all essential ingredients of creativity.

�� Executive Product Manager Jon Howard said: “Children were taking part in creative activities as entertainment. This wasn’t just casual engagement; the quality of the creations demonstrated a high level of skill. Most surprising and exciting were the emergent behaviours, young people using the digital tools in ways that weren’t originally conceived – devising games that we hadn’t thought possible. We knew we’d uncovered a rich seam of creativity among young people.”

One of the innovations underpinning Mixital was the Youth Panel: a group of 18-21 year olds who met the �� production team regularly to help shape the product. They even built and launched their own maker kit with �� Radio 1Xtra.

“It’s been great working with Mixital. The Youth Panel – many of whom were Freeformers – really enjoyed working alongside �� engineers and designers to create and launch their own product. I know how much they learned from these experiences,” said Adam Freeman, CEO of Freeformers.

From April 2019, it will no longer be possible to register with Mixital or submit any new content using the tools. And from April 2020, Mixital itself will be removed from public view.

For more information please see Mixital FAQs.

Around the world with TLS 1.0 (Part 1)

2018-11-30T11:36:00+00:00

Recently, a draft IETF proposal to formally deprecate TLS 1.0 and TLS 1.1 was published and soon afterwards, Mozilla, Google, Apple and Microsoft coordinated their announcements that they intend to remove TLS 1.0 and TLS 1.1 from new versions of their primary web browsers.

Removing TLS 1.0 and TLS 1.1 in newer web browsers is a good step forward, which I hope will drive up the number of websites and services offering TLS 1.2 and TLS 1.3.

Some of the above announcements provided statistics on TLS 1.0 and TLS 1.1 usage in modern browsers, since it’ll be modern browsers from which TLS 1.0 and TLS 1.1 are removed. The numbers I saw stated in the announcements (TLS 1.0 at 1.1% and TLS 1.1 at 0.1% usage) looked much lower than some I had seen in our per-geography data — likely because their data is globally aggregated.

“Best check our data to see how it’s looking, eh?” I thought…so I did.

Methodology

I put in some work earlier this year to make it easier to use the HTTP access log data from �� traffic management services. We now have an automated ingestion pipeline which takes the access logs from their existing AWS S3 storage buckets, verifies, parses, enriches and transforms them before loading them into Google BigQuery (in a GDPR-compliant manner, of course). The net result is that we can now perform SQL queries across all of our traffic management layer’s access logs in a very short timeframe. This has been a game-changer in my opinion, we’re using the data to discover all sorts of things we never knew about usage of our services.

The data I used for this particular study show HTTPS (only, not HTTP) requests to www.bbc.co.uk and www.bbc.com from November 10th-13th 2018 — a total of just over 2 billion requests from 250 countries (including country: “unknown”).

Global view

First of all, I looked at our “global view” of TLS usage. This covers TLS usage on www.bbc.co.uk and www.bbc.com from every country we served:

TLS Version	Number of requests	Percentage
TLSv1.2	2,002,516,373	97.96%
TLSv1.1	4,529,764	0.22%
TLSv1.0	37,160,210	1.82%

view raw tls1-blog-global.md hosted with ❤by GitHub

So whilst our global aggregate view of TLS usage differs a little from e.g. the Firefox metrics, it’s not vastly different.

Per-Country view

As I mentioned earlier, the main purpose of this study was to look at how TLS usage varies by geography, as a contrast to the global view for our audience. My query counted the number of HTTPS requests and grouped them by the negotiated TLS version and also by the country (using the IANA name) from which the request originated. I then filtered out countries with less than 10,000 requests as they’re probably less reliable, statistically. Since the result set is pretty large, I then filtered to only include countries which have greater than 5% of TLS 1.0 usage. The results are as follows (ordered from highest to lowest percentage of TLS 1.0 usage):

Country	Number of requests	Percentage of TLS 1.0 usage
Bosnia and Herzegovina	35,031	100.00%
China	2,261,506	86.93%
Montenegro	28,712	48.74%
Croatia	113,948	43.75%
Uganda	150,225	34.48%
Honduras	97,644	29.55%
Ethiopia	180,473	26.04%
Democratic Republic of the Congo	12,775	25.67%
Nigeria	1,224,923	25.13%
Cote d'Ivoire	14,717	23.68%
Myanmar	164,751	21.25%
Hungary	175,327	20.20%
Cameroon	11,618	15.02%
Tanzania	76,469	14.93%
Somalia	189,509	12.98%
Sudan	16,273	12.93%
Mozambique	10,348	12.39%
Taiwan	195,132	11.01%
Zambia	29,070	10.41%
Morocco	32,932	10.04%
Uzbekistan	17,135	9.38%
Japan	489,215	9.15%
Hong Kong	426,542	8.97%
Algeria	24,760	8.97%
Romania	62,019	8.79%
Zimbabwe	19,253	8.15%
Egypt	52,061	7.60%
Turkey	234,372	7.32%
Philippines	94,536	6.95%
Ghana	44,913	6.71%
Belarus	28,211	6.68%
Kenya	73,939	6.39%
Nepal	38,569	6.00%
Bulgaria	27,659	5.96%
Malawi	15,501	5.85%
Jordan	13,419	5.73%
Indonesia	119,720	5.40%
Ukraine	86,505	5.35%
Republic of Korea	83,370	5.33%
Saudi Arabia	79,834	5.21%

view raw tls-1-blog-per-country.md hosted with ❤by GitHub

It’s pretty clear that there are very significant differences across the world in TLS 1.0 usage from country to country. We’ll dig into this in a little bit more detail in a moment but I should just mention for now that the data from China might be inaccurate as (to the best of my knowledge), www.bbc.co.uk and www.bbc.com are currently blocked in China (following our migration to HTTPS) so this could well be proxied/VPN’d traffic rather than traffic direct from users.

It’s interesting to make a comparison with the two countries which make up our largest user-base by request count:

Country	Number of requests	Percentage of TLS 1.0 usage
Great Britain	23,778,043	1.43%
USA	2,373,620	1.47%

view raw tls1-blog-gb-us.md hosted with ❤by GitHub

These data show what you’d probably guess, they’re similar and are just a little bit below the global values.

Clients

The next most obvious question is perhaps “what is making all these TLS 1.0 requests?”. The global most popular 20 (from over 90,000) user agents are:

User Agent	Browser/OS	Number of requests
Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0	Firefox 26 / Windows 7	2,243,786
CITRIXRECEIVER	Citrix receiver	1,762,744
Nokia6280/2.0 (03.60) Profile/MIDP-2.0 Configuration/CLDC-1.1	Nokia model 6280	1,054,119
HTTPClient/3.4.0 (Linux; Android 4.0.3; KFTT Build/IML74K)	HTTPClient / Android 4	1,045,245
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)	Firefox / Genio search addon	892,453
SGOS/6.7.3.9 (S400-30; Proxy Edition)	Symantec SGOS (proxy)	479,035
Dorado WAP-Browser/1.0.0	Dorado	468,775
Mozilla/4.0 (ISA Server Connectivity Check)	Microsoft ISA server (proxy)	453,251
Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1	Firefox 16 / Windows 8	439,674
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/6.1.6 Safari/537.78.2	Safari 6 / OSX 10.7	387,037
HTTPClient/4.0.0 (Linux; Android 4.4.4; SM-T560 Build/KTU84P.T560XXU0APL1)	HTTPClient / Android 4	369,724
Mozilla/5.0	Possibly Bluecoat (proxy)	342,443
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.59.10 (KHTML, like Gecko) Version/5.1.9 Safari/534.59.10	Safari 5 / OSX 10.6	302,765
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)	IE 9 / Windows 7	291046
Mozilla/4.0	Possibly Bluecoat (proxy)	247,455
ProxySG Appliance	Symantec SGOS (proxy)	246,598
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; GTB7.5; EasyBits GO v1.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; yie8)	Yahoo IE 7 / Windows XP	242,294
HTTPClient/4.0.0 (Linux; Android 4.4.2; SM-T310 Build/KOT49H.T310XXSBQB4)	HTTPClient / Android 4	218,775
MediaCAT/4.5.1	?	216,015
HTTPClient/3.4.0 (Linux; Android 4.3; GT-I9300 Build/JSS15J.I9300XXUGMK6)	HTTPClient / Android 4	213,459

view raw tls1-blog-uas-global.md hosted with ❤by GitHub

So we can see that there are some old desktop web browsers, some feature phones, some proxies and some HTTP libraries, mostly running on older Android versions (mainly Android 2 and 4). Further down the list there are lots more HTTP libraries and web browsers running on Android 2 and 4. We can compare this global view with the 20 most popular user agents from Bosnia and Herzegovina:

User Agent	Browser/OS	Number of requests
Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1	Firefox 16 / Windows 8	18,137
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)	Firefox / Genio search addon	1,344
Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)	IE 9 / Windows ?	1,251
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko) Chrome/24.0.1295.0 Safari/537.15	Chrome 24 / Windows 8	1,240
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)	IE 10 / Windows 7	1,229
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; Media Center PC 6.0; InfoPath.3; MS-RTC LM 8; Zune 4.7)	IE 9 / Windows 7	1,228
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17	Chrome 24 / OSX 10.8	1,213
Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1	Firefox 16 / Windows 8	1,213
Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2	Firefox 15 / Windows 7	1,194
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.14 (KHTML, like Gecko) Chrome/24.0.1292.0 Safari/537.14	Chrome 24 / Windows 8	1,174
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 7.1; Trident/5.0)	IE 9 / Windows ?	1,171
Mozilla/5.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1	Firefox 16 / Windows 8	1,168
Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US))	IE 10 / Windows ?	1,156
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)	IE 10 / Windows 7	1,143
HTTPClient/3.4.0 (Linux; Android 4.1.2; LG-E440 Build/JZO54K)	HTTPClient / Android 4	164
Mozilla/5.0 (Linux; U; Android 4.1.2; fr-fr; LG-E610 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30	Webkit ? / Android 4	132
Mozilla/5.0 (Linux; U; Android 4.2.2; hr-hr; TPC-71203G Build/JDQ39) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30	Webkit ? / Android 4	125
HTTPClient/3.4.0 (Linux; Android 4.4.4; E2105 Build/24.0.A.5.14)	HTTPClient / Android 4	98
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/534.50.2 (KHTML, like Gecko) Version/5.0.6 Safari/533.22.3	Safari 5 / OSX 10.5	54
Mozilla/5.0 (Linux; U; Android 4.3; en-; SGH-T999 Build/JSS15J) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30	Webkit ? / Android 4	46

view raw tls1-blog-ba-uas.md hosted with ❤by GitHub

Here we see fewer HTTP libraries, no feature phone or proxies but a greater proliferation of old desktop web browsers, notably lots of Chrome 24 (2013) and Firefox 15 (2012) & 16 (2012). There’s lots of old Android (especially v4, ~2013) in both result sets. Of course the user agent HTTP header is completely spoof-able so there may be some inaccuracies.

The concentration of year of client release is interesting though, I wonder why 2012 and 2013 are so common? It doesn’t seem to be tied directly to a TLS version change since TLS 1.0 was 1999, TLS 1.1 was 2006 and TLS 1.2 was 2008 (though it was revised in 2011). Answers on a postcard (or in a comment) please!

Is there anything we can do to reduce the TLS 1.0 usage?

The short answer, sadly, is “not really”. The longer answer involves waiting for the natural reduction in older Android versions as the devices running those OS’s fail and are replaced, hopefully with something which supports better crypto! The complication to this is in geographies which are not so wasteful as most “western” economies. In India, for example, older devices are much more frequently repaired than in the “west”, often by local repair agents whose skill and ingenuity can keep devices running for much longer than they do elsewhere.

What about TLS 1.1?

As it is for the rest of the industry, our TLS 1.1 usage is much, much lower than TLS 1.0 and TLS 1.2. This is typically because most user agents/Clients which support TLS 1.1 also support TLS 1.2, so TLS 1.1 doesn’t get a big slice of the action. Our data shows no countries with over 10,000 requests in the 3 days of data which also have TLS 1.1 usage above 1%.

Recommendations

Whichever metric(s) you’re looking, ensure that you don’t just look at the global/overall aggregated numbers, which often mask large regional/subset variations. The constituent communities of your audience often differ significantly, so it’s really important to understand how that affects your data and therefore your decision making process.

The same goes for percentages versus absolute numbers — for example, 0.2% of a large number of users is, in absolute numbers, still a lot of users. Don’t discount seemingly small fractions of a large user base without checking how many people that fraction represents!

P.S. Thanks to my children, Polly and Stanley, for the illustrations. I couldn’t find any suitable pictures so they drew some for me.

Round up: September 2018

2018-09-21T13:13:29+00:00

It's been a busy few weeks both within Design + Engineering and across the �� as a whole, so it's worth one of my occasional round-ups to bring everyone upto date....

The DG Tony Hall has announced both internally and externally about a major reshaping of our online services. There will be more on this on these pages over the coming weeks
News has launched the first ever bulletin for a mobile audience in India, as part of the expansion of the World Service
The �� VR documentary Damming the Nile has won a Rose D'Or award at a ceremony in Berlin. Meanwhile ‘1943 Berlin Blitz’ will be on show at the RAF’s Battle of Britain Air Show at Duxford this coming weekend
A new 3D audio drama Vostok-K Incident has been launched by R&D on Taster. Listeners can use mobile phones or tablets as extra speakers in this lastest experiment in Spatial sound
R&D has also resolved the "lag" issues that viewers experience during live internet sports streams
And we've had two schemes launch to encourage more women to get into technology roles at the ��

So a busy time and there's plenty more to come over the next few months as more services and launches are in the pipeline.

Privacy by design

2018-06-13T14:18:00+00:00

This little button means a lot to us.

When it’s switched on it allows you to view personalised content recommendations based on your historic �� activity data.

I work for the �� Analytics Services team in Cardiff and we’ve been working on what happens when you switch it off.

The �� decided that when you switch off personalisation, it shouldn’t link any activity data to your account. In fact, we think that it shouldn’t ever be possible to tie activity data back to you. This page about using the �� expresses it well:

“Data about how you use the �� will be anonymous. For instance, we’d be able to see that someone looked at a particular story on �� News, but we wouldn’t be able to tell if it was you.”

That’s actually quite a difficult software engineering problem to solve. The way most analytics tracking systems work is by using cookies or some other persistent identifier precisely to be able to tie together a user’s activity across multiple sessions. To get around that, we reset a user’s analytics identifier whenever they switch personalisation off.

How it's normally done

Normally, when you sign in or out of a service your analytics identifier will persist. This means that organisations can attribute data to you as an individual even when you’re signed out or have chosen not to receive a personalised experience.

How we've designed it

We’ve designed our analytics ID differently, with privacy in mind. We’ve designed it so that when you sign in to your �� account and disable personalisation, we can’t attribute activity back to you as an individual.

We’ve designed privacy into our analytics.

Putting users in control of their data

The General Data Protection Regulation, or GDPR for short, is one of the biggest changes to data privacy law in recent years. It is designed to put you in control of how your information is collected and used by organisations.

This change to our analytics services is a small example of how we are designing privacy as a feature to put users in control of their data. It is part of a wider opportunity to enable much greater control over how your data is collected, what you share and with whom. And in turn to drive a more relevant and nuanced personalisation of the ��’s services.

We see privacy not just as an exercise in legal compliance but as an opportunity to deliver greater value for users.

Technology + Creativity at the �������� Feed

Optimising serverless for �������� Online

Serverless in Web Core

Optimising our performance

Selecting the correct memory profile

Cold start times

What type of performance do we see?

Diagnosing and debugging

Deploying to Live

Thanks for reading!

Delivering �������� Online using serverless

The �������� has some big traffic days

Choosing the right platform for the job

Reducing operational complexity

Next time… using serverless in Web Core

The complexities of creating a new 'follow topic' capability

Context

Principles

The Key Considerations and Complexities:

How will the resulting follow actions be surfaced?

Conclusion

�������� Food at 20: Its biggest year

2020 wasn’t an ordinary year

Popular peaks

Budget cooking and Daily Kitchen Live

Changes to trends: Reactive SEO

Changing Spikes

Top Search Terms Driving Traffic During the UK’s Lockdown (23/3/20–4/7/20):

Christmas Trends

Pre-Christmas

Christmas

Post Christmas

What’s Next?

�������� Online: 2020 in review

�������� Children’s and Education

�������� Account

�������� World Service News

�������� Website (including News, Sport and ��������)

Radio & Music Services (�������� Sounds Back End)

�������� Voice+AI

�������� Online Technology Group (OTG)

iBL (iPlayer API)

�������� iPlayer — Off-Product Discovery

Digital Publishing

Technology Strategy & Architecture (TS&A)

�������� Datalab

�������� News and Weather Apps Team

�������� Research & Development

The lessons learnt creating a design system for �������� Online

A short history lesson

1. Make it as easy as possible to share

2. Do things on a need to make basis

3. The reinvention of the wheel

4. Be strict to allow for flexibility

5. Documentation is hard, but important

6. Design for longevity through obsolescence

Moving �������� Online to the cloud

Context

Principles

High level technology overview

On to the next chapter

Around the world with TLS 1.0 (Part 2)

Global view

November 2018 (original) data

February 2020 data

Per-Country view

Clients

Conclusion

An accessible digital �������� – 2019 in review

UX&D and Internal Tools:

iPlayer and Sounds:

Children’s Games & Apps:

News:

Sport:

Weather:

Web Core:

Voice & AI:

�������� Studios:

Elsewhere:

Coming Up

Technology + Creativity at the �� Feed

Optimising serverless for �� Online

Delivering �� Online using serverless

The �� has some big traffic days

�� Food at 20: Its biggest year

�� Online: 2020 in review

�� Children’s and Education

�� Account

�� World Service News

�� Website (including News, Sport and ��)

Radio & Music Services (�� Sounds Back End)

�� Voice+AI

�� Online Technology Group (OTG)

�� iPlayer — Off-Product Discovery

�� Datalab

�� News and Weather Apps Team

�� Research & Development

The lessons learnt creating a design system for �� Online

Moving �� Online to the cloud

An accessible digital �� – 2019 in review

�� Studios:

�� Online: 2019 in review