How can BBC Sport make the best use of new technologies, ever-increasing Internet connectivity, and social media growth, to offer more to audiences?

I'm Matthew Clark, Senior Technical Architect for Future Media Sport.

As I wrote in my last blog post, it's been a busy year, due to a plethora of new features made in time for this summer's Olympics. With the Games now over, it's time to consider what's next: what is the future of online Sport coverage? How can we make the best use of new technologies, ever-increasing Internet connectivity, and social media growth, to offer more to the BBC's audiences? That's a big question, so it's excellent timing that we have the Sport Connected Studio to help.

The BBC Connected Studio programme has its own website with all the details, but here's a summary:

Over the course of the year the programme works with ten BBC Online product teams of which BBC Sport is the 6th. You can attend as an individual or a business/organisation, and there's some financial support available for the later stages. (See the FAQ for more.) To give you an idea of Connected Studio's ambition and what has happened so far you can have a look at a post from Adrian Woolard, Programme lead on Connected Studio going through all the current pilots.

So, on to the brief for Sport. You can view it here, but here's my take.

We're focusing on live sporting events - moments where major sport events are happening and large numbers of people are following it online. What can be done to make this a more rewarding experience? In particular, we're looking at two scenarios:

2. The football season. Every Saturday, millions follow football games throughout the country (and the world). Most games aren't televised live, but there's plenty of coverage through radio, online, and studio TV coverage such as Final Score.

For both scenarios, the question is: what can be offered that makes following the sporting event better? Ideas can be pretty much anything, so long as they cover either or both of these scenarios. It has to be an online product, but that doesn't just mean web pages - interactive apps, perhaps for phones or internet-connected TVs, are encouraged too.

Not got an idea? Here are a few topics to whet the appetite:

• Choice: How does the audience want to choose and interact with multiple concurrent sporting events?

• Interaction: Can the audience be further engaged whilst watching live events?

• Second screen: How best should handheld devices work alongside TVs?

• Social engagement: How should use of social media drive a live sporting experience?

• Accessibility: How are more interactive experiences offered to those using screen readers or other accessibility tools?

• Companion data: What and how should information be added to video to make it more enjoyable and accessible?

• Varying audiences: Do people of certain age ranges, or sporting interests, prefer different experiences?

• Audience behaviour: How does real-time analysis of site interaction shape what should be on offer?

• UX: How can an interactive experience be built, for all size screens, without impacting the live video and detracting from the main event?

• Data mining: Can sport stats, user behaviour, and social data be analysed to provide new insights?

• Technical speed and scale: Can you deliver real-time updates to millions of users, whilst still staying reliable at key moments?

So, are you interested? If you've got innovative, creative, original ideas for online Sport coverage, make sure you apply. It doesn't matter whether your idea is big or small, or how thoroughly prepared it is. Get your skates on - the 'creative' day, where ideas are developed and pitched - is on the 21st November.

Matthew Clark is Senior Technical Architect, Sport, BBC Future Media.

BBC Olympics architecture overview. It shows how many components are involved!

Hi, I'm Matthew Clark, the Senior Technical Architect for BBC Online's Olympic website and apps.

Alongside colleagues Mike Brown and David Holroyd, it's been my responsibility to create the technical strategy that has allowed us to produce successful online Olympic products.

We've focused on the design and development to make sure the site and apps stay reliable and can handle high traffic loads, whilst offering more content than ever before. In this technical blog post I'll be looking at some of the challenges we faced and how we overcame them.

More traffic: Handling unprecedented audience levels

We expected the Olympics would drive far more traffic to our site than ever before, and it did. Planning for this load was not easy. There are over 60,000 dynamically generated pages, many with a significant amount of content on them, so efficient page generation is vital.

Content needs to be as 'live' as possible, so long-term caching is not an option. We use a range of caches (including Content Delivery Networks, Varnish, and mod_cache) to offload the bulk of the traffic from our Apache web servers. For content that's dynamic, cache lifespan (max-age) varies between a few seconds and a few minutes, depending on the context. This is particularly true for the new video player, which needs the latest data every few seconds to compliment the live video stream.

Page generation is done using PHP, which is stateless and receives all of its data through calls to a RESTful API. This API is the Java application layer that retrieves its data from a range of data stores (including MySQL databases, triple stores, and XML content stores). It's the most critical layer from a performance point-of-view, as load is high, calls can involve significant processing, potentially multiple data store calls, and limitations on what can be parallelised. Caching (mod_cache and Memcached) is again used to address the bulk of the traffic.

From data stores to screens, sites and apps

We spent considerable time modelling how traffic creates load on the whole stack. This was first done theoretically (through modelling of user behaviour, load balancing, and caching). We then did it for real - we used data centres around the world to place load equivalent to over a million concurrent users on the site, to confirm everything worked during busy Games load.

More video: handling 2,500 hours of coverage

A moment when all 24 streams were running at the same time. Can you identify what they are?

There's been plenty of discussion about the BBC's 24 video streams, and the challenges of creating them at the International Broadcast Centre by the Olympic Park. This is the equivalent to 24 new channels, offered over cable and satellite as well as via IP.

Once the channels are created, the challenge is to direct viewers to the right content at the right times.

Sport schedules have a habit of changing - extra time, delays due to rain, etc - and the sites and apps need to show this. When an event starts, the tools used by (human) controllers to control the video also log the start in our XML content store. This metadata is then picked up by pages and apps (via an API) so that, within a minute of the event starting, there are links to the content throughout the site, app, and red button. A similar process happens when coverage finishes. Olympic sessions can be as short as 45 minutes, so the faster a video stream can be made available, the better.

More screens: coverage on mobiles, tablets, computers and internet connected TV

The BBC has a four screen strategy where we develop for PCs, tablets, mobiles, and internet connected TV. For the Games we've offered an unprecedented amount of content to all four. In addition, there are Olympic apps for iOS and Android smartphones, a Facebook app, foreign language content for World Service sites, and a red button service for satellite and cable TVs.

Our architecture is the classic multi-tier approach - pushing as much logic as possible into shared components, so that the amount of development for each interface is as low as possible. This is DRY (Don't Repeat Yourself) at a multi-platform level. For web pages, a single PHP codebase creates both the desktop and mobile versions. This includes the iOS and Android apps, which use PhoneGap to 'wrap' the mobile website for most of its functionality. It has saved us having to rewrite functionality in native code. Certain other applications, such as the Olympic Facebook app, are different enough to warrant their own codebase, but still make the same API calls to the Java application layer, where most of the 'business' logic is held.

More content: Data is power

Video aside, there is a wealth of data required to make the Olympic site. The primary source is the Olympic Data Service, which blog posts from Oli Bartlett and Dave Rogers have already covered in depth. In brief, Olympic Broadcasting Services (OBS) provide a comprehensive data feed that covers all sports, and provides a wealth of data - from latest scores to medal tables. This, combined with stories from journalists, and other sources such as Twitter, creates the content for tens of thousands of results, athletes, country, and event pages.

The Dynamic Semantic Publishing (DSP) model, which understands relationships (triples) between all content and concepts, is the process that ensures everything automatically appears in the right place. All created content, including stories, medals, and world records, are tagged (normally automatically) with the appropriate athletes, sports and countries. This causes the content to appear on the appropriate page without human intervention.

In essence, it's this automatic curation of pages that has allowed us to offer such a broad range of product. The automation has kept maintenance to a minimum, freeing journalists to focus on writing content. It's allowed multiple products and thousands of pages to stay up-to-date without a large operational overhead.

More testing: Simulating an entire Olympics

With all this content, data, video, and technology, comes a huge engineering challenge: how do you test it? All development areas follow Test Driven Development (TDD) so there is no shortage of automated unit and component tests. But what happens when you plug everything together? How can you be sure that the right medals go to the right country, or that video works on all devices, or that results appear correctly for all 36 sports? Unlike, say, the football season, the Olympics only happen once every four years, and only last a couple of weeks.

There are no second chances.

We needed to be 100% sure that on day one of the Games, everything would work as expected.

To tackle this we set up an entire team, as big as any development team, with the job of proving everything would work when the Games started. We took a three-pronged approach:

1. We used as much of the Olympic technology as we could for other sporting events, such as F1 and Wimbledon. This was often offered as a beta service - and we're grateful to all who gave these a go.
2. We used the Olympic test events to get video and stats that would be similar to that of the Games. We ran this on our staging environment so that they could be made to look like real Olympic events without appearing on our normal site.
3. Most importantly, we created fake video and data that let us simulate the Olympics. We picked the most interesting moments of the Games (the opening ceremony, the first big day, etc) and created all the inputs as they would be at that moment. This was run in our staging environment and allowed us to see all sites and apps behave as they would when the Olympics were underway. (You can read more about how we made the simulated data in the post by my colleague Dave Rogers.)

This testing process lasted several months and caught a significant number of bugs and performance problems. Fortunately it paid off - I was a little nervous on the first day of the Games, but it passed without incident.

More for the future

With the end of the Games fast approaching, attention now turns to other areas of the Sport website.

Some features have already been applied throughout - for example, most live video coverage will be in HD from now on. Other services that we've offered for Olympics aren't yet ready for use elsewhere (mobile apps and video chapter points being two). Over the coming months we'll be working on bringing many of these features to the rest of Sport, and perhaps other parts of BBC Online too.

If you've any questions about the technology we've used during the Olympics, please get in touch using the comments below.

Matthew Clark is Senior Technical Architect, Knowledge & Learning, BBC Future Media