News feeds
Chopping Up Radio – collaboratively annotating radio programmes
- , ,
Abstract
With more and more TV and radio programmes being made and more and more ways of accessing them (e.g. digital broadcasts, PVRs or downloading) one of the major challenges is how to find things and how to describe things. At the moment the metadata describing these programmes is often very poor. To start to address this BBC Radio & Music Interactive have developed the Annotatable Audio prototype which is a wiki-like interface for listeners to collaboratively chop up radio programmes into segments and annotate and tag each segment.
We will be using our audience to generate additional metadata for our programmes. But why would our audience want to do this? Because they will get a much better described programme; increasing its findability and introducing new ways of exploring within programmes. There are potentially all sorts of searches and custom downloads of marked-up and chopped-up programmes that the BBC could provide with this metadata.
The interface for the application uses a combination of Flash for the audio playback and editing and HTML/Javascript for the descriptive elements to provide a familiar wiki-like environment. The SQLite and Python back-end provides an XML-based REST API to promote further use of the metadata.
Finding things
One of the major challenges for anybody on the internet is to make their content findable. And this is why companies in the search and aggregation arena like Google and Yahoo! are so powerful. The majority of the content of the web is written material and text is (relatively) easy to search and index.
But things are different when you are a media company and your core content is audio or video because now it gets harder to search for things. The interesting bits of radio programmes (i.e. the content) are encoded as audio - human speech, sounds and music – and not text. Conventional web search engines aren’t going to be able to find that content.
Metadata - describing programmes
To address this problem we have something called metadata, data about data. Metadata is used to describe TV and radio programmes to allow them to be presented in Electronic Programme Guides (EPGs), in listings magazines, on websites and in search engines.
And we've always had this in the broadcast world - information about what's on - but this becomes even more important in the world of on-demand, downloadable content because now there are thousands and thousands more rivals for the viewers’ and listeners’ attention and millions more pieces of content out there.
Here are two examples of metadata from the BBC website...
Figure 1. Book of the Week billing
Note how it gives you information about the repeat, some series information (“Book of the Week”), a genre (“Arts & Drama”), information on who made the programme, who the reader is, who wrote the book and what this episode is about.
Figure 2. Jo Whiley billing
Whereas in this case we just have a generic billing that doesn’t include any detail about that particular day’s programme.
Sometimes our current metadata is descriptive and rich and sometimes it’s not.
Approaches for getting metadata
So how can the BBC generate metadata for our radio programmes? I've divided the approaches into three possible sources:
From the production process
From machine processing
From the users
Now you might have thought that the BBC would already have detailed metadata about programmes because we commissioned, created and broadcast them but unfortunately that isn't necessarily the case. This is for a number of reasons - because we're a big unwieldy organisation, because production teams are there to make programmes, because we're got various unconnected IT systems and I could go on... One of the things we do at BBC Radio & Music Interactive is to try to solve this kind of problem; extracting data and joining things up to add value to what we’ve got. I'll look now in a bit more detail at the three ways of generating metadata.
From the production process: Fairly self-explanatory; either getting metadata from people in the radio production process or from machines used in the production process. For example, our hard-disk play out system, known as VCS, provides relatively detailed and comprehensive data about what songs are currently being played out on our radio stations.
From machine processing: We can process the audio to generate data; we could extract low-level features like frequency spectra or volume level and we could extract higher-level features like words using speech-to-text software.
From the users: We can ask our audience to help generate metadata for the programmes they have listened to. One advantage we have here is a dedicated community - people really, really love some of our radio networks and programmes. Anybody know any Archers fans?
The advantage of the first two is that we can do these processes before the programmes are broadcast - with the latter we need the content to be available to the public before they can do anything with it. But nevertheless it's the user-generated metadata that I'm going to concentrate on with our Annotatable Audio project because we think this is an interesting and unexplored area.
Annotating Radio
The aim of the Annotatable Audio project is to get our listeners to describe our programmes - specifically we want them to split our programmes up into sections and annotate and comment on them. This benefits both the audience and the BBC by adding the ability to easily move around within programmes, by describing the contents of programmes in detail and by making our programmes more findable in general.
Playback
Figure 3. The playback screen
The playback screen contains a typical audio playback interface; a timeline containing an audio waveform, a play head and play/pause/skip buttons. The area underneath the audio playback widget contains annotation with timings, titles, descriptions and tags. Marked on the audio waveform in green are the segments or chapters. Hovering over a segment shows a tooltip containing the title of that segment. The skip buttons can be used to move between segments with the annotation underneath scrolling to match and, conversely, clicking on an annotation's timing button will skip to that segment in the audio.
Editing
Figure 4. The edit screen
By clicking on the “edit/annotate” tab at the top of the screen the user can create and edit segments and annotations. As you can see in the image above there is now, in addition to the playback UI, a larger, zoomable representation of the audio underneath. Existing segments can have their start and end points edited by dragging their edges and new segments can be added by clicking the "Add Segment" button. We have put one restriction in - segments must not overlap. Although it is straightforward to support overlapping segments in the data model we decided it is overly complex to try to represent overlapping segments in the user interface - particularly when we then apply a wiki-like model of interaction.
When a segment is selected, by clicking on it, the annotation panel opens up below. From here a title, annotation and tags can be added or edited or the whole segment can be deleted. The editing process works just like a wiki - anybody can come and create, modify or delete segments and annotations. The most recent edit is always the currently displayed version of the programme and there is a versions page to track the changes and revert to a previous version.
Versions
In this prototype there is only a very simple versions page, under the “history tab”, that includes a time-stamped entry for every single change made to the programme and the ability to revert back to that version of the annotated programme.
Communities
As mentioned before, at BBC Radio & Music we have a loyal and fairly active online community who contribute to message boards based around programmes (e.g. the Archers message boards [http://www.bbc.co.uk/dna/mbarchers/]. This should provide a good starting point for building a community around the audio annotation. We are intending to launch it around a relatively low profile factual programme so we are starting off small. And we want to promote the annotation rather than the discussion angle, which at the moment is covered by our message boards.
Users who wish to contribute will be compelled to sign in using the BBC's Single Sign-On system following Clay Shirky's advice [http://www.shirky.com/writings/group_enemy.html] that social software should be designed with handles for users to invest in as well as having barriers to participation. Anybody will be able to view the annotations but if you wish to add a comment or annotation you must be logged in.
Obviously there is a community benefit to annotating and there is a benefit to the BBC but we may need a direct and obvious benefit to the user to encourage them to join in. One avenue we are exploring is to allow users to easily link to clips of programmes from their blogs or websites. In this way they can easily refer to something they have heard on the radio.
Development
Initial prototype
The initial development of the prototype for annotatable audio was led by the R&D team at BBC Radio & Music Interactive. This is a small team (currently two people) but for some projects we will put together a multi-disciplinary team from the wider department. In this case the core team consisted of one R&D producer, one software engineer, one Flash developer and one client-side developer. The work was done in one month - from putting the team together to deploying the finished prototype.
Technical details
The server-side of the prototype runs under a Linux and is written in Python, using the Quixote web application framework [http://quixote.ca/] and an SQLite database. We implemented a REST-like API, serving XML documents over HTTP to the client.
For example…
/audioannotation/api/programmes/3/
serves...
<programme id="1" version="78" audio_url="news.mp3" visualisation_data_url="news.txt" start_time="Tue Oct 18 15:08:00 2005" duration="300000" channel="BBC Radio 4">
<description purpose="factual">
<title>The News</title>
<text>News headlines at 12:00 on Radio 4</text>
</description>
<annotations>
<segments>
<segment id="5" version="8" start="40274" end="50272" timestamp="Tue Mar 07 16:34:19 2006">
<description purpose="factual">
<title>The headlines</title>
<text>News headlines at 12:00</text>
<tags>
<tag>news</tag>
<tag>headlines</tag>
</tags>
</description>
</segment>
</segments>
</annotations>
</programme>
Whereas /audioannotation/api/programmes/3/segments/5/ serves just the <segment id="5"...>...</segment> XML fragment.
HTTP GET is used to retrieve XML from a URL and POST updates an object with the given XML. Unfortunately Flash does not support use HTTP DELETE so we post defined XML messages to delete objects.
The client interface is a combination of Flash, HTML, CSS and Javascript. The audio playback and editing interface is built in Flash 7 and text annotation achieved through HTML and Javascript. The Flash component acts as the sole point of contact to the server-side interface and communicates the appropriate XML data to the Javascript in the page - we weren't completely happy with our approach here and we are now looking at replacing it with the Macromedia JavaScript and Flash Integration Kit [http://weblogs.macromedia.com/flashjavascript/].
The next version
We are currently at work on the next iteration of this project with the intention of producing something that can be piloted on the BBC’s website. The main work involved is to rewrite the server-side code so it is suitable for a public-facing application and meets the appropriate BBC guidelines. We are also redesigning the interface to make it fit more with the appearance of our websites, though we will make it skinnable to allow different radio networks to have different design aesthetics and different configurations of the interface. Here are some initial design mock-ups of how it could look on the Radio 4 site.
Figure 5. Mock-up of new playback screen
Figure 6. Mock-up of new editing screen
Initially the functionality of the application will be very similar to that of the prototype though you may notice in the screenshots that we have included a timed comment function. In addition to the annotated segments, users will be able to add short text-only comments at points along the timeline of the programme. These are intended to be short comments rather than the factual content of the annotated segments. We have also thought about allowing comments to be contributed by mobile phone text message - possibly during the live broadcast.
The most interesting applications will follow this – all the things that we can do with the chopped up and annotated programmes. Here are some of our ideas for what we might do:
Creating chapterised podcasts: Using the user-defined segments to automatically create chapter points within podcasts. iTunes and later iPods already support this feature in the AAC format and Apple provide a command-line tool for generating these files [http://homepage.mac.com/applepodcast/podcasts/Resources/static/podcast_chapter_tool_beta.dmg]. There is also a recent addition to the ID3 standard that allows chapter points in MP3 files [http://www.id3.org/id3v2-chapters-1.0.html].
Searching within programmes: Using the annotations and tags to allow users to search for content within our programmes. The BBC’s current metadata tends to be on the programme level and includes little detail on exactly what was talked about. Another approach to this problem is to use one of the many transcription services to generate transcriptions of our programmes.
Custom programmes: Use the tags to create customised programmes for download. Those segments of programmes that match particular tags and are from a certain date range are compiled into custom “programmes”, either built on the fly or pre-constructed. These new programmes could be based on a fixed set of defined keywords (e.g. “news” or “UK” or “jazz”) or they could be based on sets of personalised keywords selected by users.
Visualisations of radio programmes: We are interested in creating novel, visual representations of radio programmes using the data that comes out of the annotatable audio project. Shown below is an interface prototype built using Processing [http://www.processing.org] that represents a 5 minute radio news bulletin that has been segmented and tagged.
Figure 7. Visual interface to a news bulletin
And we can also use the annotatable audio application as an internal production tool for segmenting or adding metadata to programmes.
We are investigating the possibility of releasing our code as open source and possibly making some feeds of the data created available through backstage.bbc.co.uk [http://backstage.bbc.co.uk/].
Related work
Project Pad from the Spoken Word project at Northwestern University provides an open-source web-based audio annotation tool designed for use in education [http://www.at.northwestern.edu/spoken/p04annotation.html].
The DIVER project from Stanford University is a tool for capturing, annotating and sharing bits of video [http://diver.stanford.edu/].
And some of our initial inspiration came from the Flash audio interfaces used on Odeo [http://www.odeo.com/].
Acknowledgements
Thanks to Tom Coates for starting up this project, Chris Bowley for the Flash development, Helen Crowe for the HTML and Javascript, Paul Clifford for this Python knowledge and Bronwyn Van Der Merwe and Graham Beale for their design expertise.