FIcontent Consortium Grand Challenge

Abstract

We are looking for a new generation of content-accompanying applications for connected and hybrid TV using second-screen devices and our Social Connected TV Platform technology components.

Your competition entry should clearly demonstrate the added value of second-screen applications in the TV context. Your solution should focus on a tablet or smartphone (web) application synchronised with TV content. Your entry should show how more in-depth information can be provided or how added interactivity can help to involve users more directly.

Ask yourself: How does an additional connected device help create a richer experience rather than just more distraction? What should happen on which device? How can you make things really simple for the user? Where does the integration of social media components create added value?

Organisation Background

FIcontent is an EU-funded R&D project. Its consortium members have created and tested enabling technologies that support novel uses of audio-visual media content on connected devices. A major field of interest is in the area of connected TV services with focus on multi-screen interaction, personalisation and user tracking and privacy. The following companies are members of the FIcontent project: Technicolor, BBC R&D, German Research Centre for Artificial Intelligence (DFKI), Disney Research Zurich, ETH Zurich, Fraunhofer Institute for Open Communication Systems FOKUS, Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, Grassroots Arts, i2CAT Foundation Internet and Digital Innovation in Catalonia, ImaginLab, Institut fuer Rundfunktechnik (IRT), Lancaster University, Orange, Pixel Park, Rundfunk Berlin-Brandenburg (RBB), Sigma Orionis, Studio Gobo Ltd, Thales Communications & Security, The Walt Disney Company Ltd.

Challenge Background

The Social Connected TV Platform

Recently the FIcontent partners deployed the Social Connected TV Platform, a toolbox that offers the technological outcomes of our work to third parties. The Social Connected TV Platform currently provides five technology components, also referred to as enablers. Each of the enablers has been developed and is maintained by a FIcontent partner. Find a brief description of the enablers in the section below to get an impression of what the Social Connected TV Platform provides you. You will also find links to the enablers‘ wiki pages. At the wiki pages you are given more detailed information on the feature sets of the enablers, the API documentations, as well contacts to persons who can help you in case of any trouble with the usage of the enablers.

Content Enrichment SE

Provider: Fraunhofer Institute for Open Communication Systems FOKUS

Short description: create, share and distribute interactive video content across platforms and devices

Features:

• Creation of clickable objects within the video content
• Annotate and link video objects and scenes with any type of media
• Object based media discovery
• Cross-video navigation
• Provides multi-screen ready interactive video content

Technology:

• RESTful API (JSON)
• HTML5

API documentation: http://wiki.mediafi.org/doku.php/ficontent.common.enabler.contentenrichment

Audio Fingerprinting SE

Provider: Fraunhofer IAIS
Allows you to: Create mobile apps that recognise what users are watching on TV
Features:

• Computes fingerprints for an audio signal (language independent)
• Identifies content using a fingerprint database
• Allows you to show synchronised content

Technology:

• Android SDK
• RESTful API

API Documentation: http://wiki.mediafi.org/doku.php/ficontent.socialtv.enabler.audiofingerprinting

Audio Mining SE

Provider: Fraunhofer IAIS
Short description: Maintain rich indexes for your A/V content
Features:

• Analayses audio-visual content and turns speech into text
• Identifies segments and characteristic keywords
• Build indices for multimedia search

Technology:

• RESTful API

API Documentation: http://wiki.mediafi.org/doku.php/ficontent.socialtv.enabler.audiomining

Content Optimsation SE

Provider: Fraunhofer IAIS
Short description: Provide your users with content-based recommendations
Features:

• Enriches textual content by leveraging disperse sources
• Generates recommendations for audio-visual content

Technology:

• RESTful API

API documentation: http://wiki.mediafi.org/doku.php/ficontent.socialtv.enabler.contentoptimisation

Second Screen Framework SE

Provider: Intitut fuer Rundfunktechnik (IRT)
Short description: Build apps for connected TVs that communicate with second screens
Features:

• Full duplex communication between connected TVs and second screens
• Mechanism for device discovery based on QR-code
• Persistent device connection
• Automatic launch of applications on the second screen

Technology:

• For browser-based apps
• JavaScript libraries/API

API documentation: http://wiki.mediafi.org/doku.php/ficontent.socialtv.enabler.secondscreenframework

TV Application Layer SE

Provider: BBC R&D
Short description: Write an application once and be confident that it can be deployed to all HTML-based TV devices
Features:

• Abstracts the differences between various end-devices
• Gives TV application authors a consistent API to develop against

Technology:

• JavaScript

API documentation: http://wiki.mediafi.org/doku.php/ficontent.socialtv.enabler.tvapplicationlayer

Your solution should make use of them

Of course, you are not required to make use of all the tools provided by our platform. You can decide, which features you think are useful for implementing your idea and which can be integrated into your implementation with appropriate effort. You should use at least one component or show how a particular tool could help to foster your idea if you had enough time to integrate it.

Support

The documentation of our technologies is in progress and under continuous revision. Your feedback will help us to improve the documentation. In exchange we will help you getting started with our tools. Contacts are given on the respective Wiki pages for each of the components.

Maturity

The project FIcontent is still running and we are still working on the platforms components. This might lead to the assumption that the tools we are providing are not stable enough to start using without getting frustrated. Do not worry – the tools have reached a quite mature level. Most were developed in the first phase of FIcontent that lasted two years and have been fine-tuned and subjected to initial testing in the current phase of the project. The Second-Screen Framework, for example, is currently used in a large-scale trial of on-air HbbTV service offered by the ARD group.

The project’s wiki provides information on those functionalities that are already implemented and tested to our best knowledge. We will provide releases that implement these functionalities and no early prototypes. However errors may occur, e.g. we cannot guarantee 24/7-uptime for our servers.

Access

There are two ways to actually access our tools. Some of the tools are open source, like the TV Application Layer. URLs to code repositories are given in the project wiki. Other tools require the participants to ask the contact persons (as identified in project wiki) for access details. Links to repositories or login credentials will subsequently be provided on short notice.

TV-accompanying second-screen applications

As already mentioned in the “Abstract”, challenge entries shall demonstrate the added value of second-screen applications in the TV context. What is meant by the term second screen and what kind of “added value” are we expecting? The following breakdown on second-screen solutions and existing applications is setting the scene.

The listed applications include examples which have been developed in the FIcontent project. They give an impression on how our Social Connected TV Platform enablers can help you to create amazing interactive TV experiences. Further second-screen approaches which have been implemented by others are described below and give you an idea of the different aspects treated by prevalent second-screen applications. Surfing the web, you will find a lot more examples than we are able to provide here. But hopefully, this section can sketch the landscape of what has been achieved so far in order to inspire you during your creative development process.

The second screen and the added value for the TV viewer

The term second screen came up in the last 5 five years as a catch phrase that engages technologists, producers, advertisers, programme formats developers and many other players from the media and TV domain. In fact, there is uncertainty on what second screen actually means. Different articles provide different views on what the second screen is and what people do with it.

We focus on second screens as mobile connected devices, e.g. tablet computers or smartphones, or laptop computers which people use in an accompanying manner while watching TV. Accompanying here means that the content presented on the second screen has a contextual relation to the content being displayed on the TV screen. The content on the second screen might be additional information on a running TV programme, e.g. vita of the director of the current movie or background information on a report in a running newscast. Accompanying content can also be social media feeds, e.g. other viewers’ comments on a show currently being watched.

Dedicated second-screen applications can simplify the access to accompanying content for the end users. Check out the sections “Existing solutions based on FIcontent enablers” and “Other existing solutions” for more insights.

Find below some articles on the web that treat the topic second screen and its relevance. Feel free to have a look at:

• Wikipedia, “Second Screen”
• Red Bee Media Ltd., “Broadcast industry not capitalising on rise of the second screen”
• Streamingmedia, “The Rise of the Second Screen: Zeebox, GetGlue, Viggle and More”
• Google, “The New Multi-screen World: Understanding Cross-platform Consumer Behavior”

Existing solutions based on FIcontent enablers

To give you an impression of what is possible, this section gives an insight to second-screen applications that have been developed in the FIcontent project on the basis of technologies of the Social Connected TV Platform.

Second-Screen Framework

The following services include a main-screen HbbTV application and a counterpart Web application on the mobile device. The communication between them is realised through the Second-Screen Framework.

rbbtext

The rbbtext is an HbbTV application of the German public broadcaster rbb. It builds upon the classical teletext. It intends to use the facilities of the HbbTV standard for the navigation through and presentation of content and information in order to improve the user experience of the service. It provides an enriched version of the information in the classical teletext and is connected to the same content management system. In contrary to the teletext, users can navigate directly to a menu item by the help of the arrow keys on the remote control. Furthermore the improved graphical presentation of content with the help of HTML unleashes the potential of today’s high-resolution TV displays.

The second-screen enabled version of the rbbtext HbbTV application contains a new menu item that offers the user the option to connect a device. […] Once the companion application is launched, the user can make use of a synchronised navigation – what happens on the one screen happens on the other screen as well. Furthermore the user can hide the application on the TV screen from the second screen. This allows the user to continue watching the broadcast TV show while browsing through the range of information of rbbtext on the second screen. Another use-case is to open URLs on the second-screen in order to link to content which is beyond the scope of HbbTV. The editors of rbbtext can include links to additional information in the HbbTV application that cannot be displayed by the HbbTV device but by the second-screen device.¹

1^ From: “Second Screen for HbbTV – Automatic application launch and app-to-app communication enabling novel TV programme related second-screen scenarios”, Christoph Ziegler, Institut für Rundfunktechnik, in proceedings of IEEE 3^rd International Conference on Consumer Electronics, 9-11 Sept. 2013, Berlin

Screenshots of the rbbtext applications on the TV and on the second screen.

ARD EPG

The ARD EPG is an HbbTV application offered by the German public service broadcasting network ARD. The application provides information about current and upcoming TV programmes aired on channels of the ARD group. It has a full text search to find specific programmes. In addition, the ARD EPG lets users also go back in time to access content that already aired. For the shows that are available in the VoD catalogue of the ARD network, direct links to the catch-up content are available.

The related companion app enables users to navigate the EPG on the second screen. Users can hide the app on the TV screen from the second screen. They can go on browsing the EPG content on the second screen making use of the more convenient touch interface of the tablet computer. At the same time they can watch the current broadcast programme on the TV. Users can use the second-screen app to switch between channels of the ARD network and to initiate the playback of a video clip from the VoD catalogue on the main TV screen.

Screenshots of the ARD-EPG applications on the TV and on the second screen.

Audio Fingerprinting, Audio Mining and Content Optimisation

About Kate

About Kate is a TV series of 14 episodes aired on ARTE. It was complemented with a dedicated second-screen app (available for Android and iOS). The app offered synchronised add-on content and social media features. The synchronisation with TV content is accomplished by means of the Audio Fingerprinting enabler. Additional information is aggregated with the help of the Audio Mining technology. Find out more about the app under http://kate.arte.tv/de.

Other existing solutions

There are multiple second-screen applications available for iOS, Android and web browsers. In the following, we list some of them categorised by their main purposes.

Apps that allow to get in touch with other viewers

A major category of applications provide the facility to get involved with other viewers e.g. to discuss about a programme. These are also referred to as social TV applications. Examples are GetGlue, Umami, ConnecTV, miso, IntoNow, Yap TV, Zeebox, Couchfunk, Zapitano, McCheckin.

Apps that ease the discovery of content

Furthermore, there are applications that help users to find content of their interest – linear or non-linear, e.g. boxfish, nextGuide, Fayve or Tweek TV.

Apps that allow viewers to get involved with a TV show

Another category of services offered by broadcasters allows getting involved with a TV show. One example is the 4now application. It provides viewers of the British Channel 4 with the possibility to participate in polls or quizzes around a show. In a field trial, the German ARD group offered a web application Tatort+ accompanying the crime thriller Tatort, which enables users to interactively investigate on who is the murderer.

Discover what is beyond all that

The examples presented in “Second-Screen Framework” section demonstrate a feature that goes beyond the functionalities provided by prevalent second-screen applications (cf. section “Other existing solutions”), i.e. the ability to communicate between a TV app and a companion app. One might argue that there are other solutions out there that enable the communication between a TV and second screen, e.g. UPnP/DLNA, Samsung AllShare or Apple AirPlay to name a view. But a major limitation of these approaches is that they are either subject to vendor-specific restrictions or that they are only supported by a limited set of devices. Also, the latter solutions do not provide a way to establish a logical communication link between a broadcasting programme application and the second screen.

In contrast, the Second-Screen Framework builds upon widely supported web standards. Apps build on the basis of the framework can target millions of already deployed devices on the market. Since it is fully compliant with the HbbTV it allows novel programme related second-screen experiences. However, this potential has not been unleashed yet by the existing FIcontent applications. The second-screen applications rbbtext and ARD-EPG developed in the FIcontent project simply mirror the TV application to the second screen. The second screen solely acts as some kind of advanced remote control. This is quite handy, but we think that on the basis of the framework a lot more is possible. Investigating these possibilities can be one way to address our challenge.

The HbbTV standard specifies possibilities to link web content to broadcast content. DVB stream events can be used to contextually synchronise an HbbTV application with the broadcast signal. With the help of these capabilities and the Second-Screen Framework functionalities, interesting scenarios can be realised that create a direct relation between a currently broadcast programme and the second-screen application. Additional information can be displayed in sync with the programme content. At the same time, users can influence the content on the TV with their second screen.

A possible application might be a TV quiz shows. Imagine a family of three watching a quiz programme. All of them launched the dedicated quiz app on their personal devices. The quiz master posts questions to the participant of the TV show. Immediately the questions and possible answers appear on the personal devices of the family members. All of them have the possibility to make their choice on the second screen. The app on the TV screen displays who made the right and who made the wrong choices and leader board ranking who gave the most right answers.

The described scenario provides a very basic example for programme related second-screen interaction. It is up to you to think further and develop a solution that allows exiting second-screen experiences. Your concept can be complemented with an approach that allows users that do not have a connected (or HbbTV-compliant) TV to benefit from the additional information on the second screen. As shown with the “About Kate” format and the accompanying second-screen app the Audio Fingerprinting, Audio Mining and Content Optimisation technology can provide powerful instruments to automatically establish the contextual link and aggregate additional information to a show that can be provided to the user via the second screen.

Description of the challenge

We are looking for a new generation of content-accompanying applications for connected and hybrid TV that make use of second-screen devices and our Social Connected TV Platform technology components.

Your competition entry should clearly demonstrate the added value of second-screen applications in the TV context. Your solution should focus on a tablet or smartphone (web) application synchronised with the TV content. Your entry should show how more in-depth information can be provided or how added interactivity can help to involve users more directly.

Ask yourself: How does an additional connected device help create a richer experience rather than just more distraction? What should happen on which device? How can you make things really simple for the user? Where does the integration of social media components create added value?

We are excited to see how you imagine a future TV media experience to look like – when borders between technologies for Web and TV are blurred.

The solution should meet the following requirements. It should:

• follow a novel approach,
• be beneficial to the end-users (e.g. simplify control and navigation),
• be deployable with reasonable effort,
• be presented in a comprehensive and convincing way.

Furthermore, the level of maturity of the implementation of the solution will be taken into account.

The FIcontent partners will provide access to the Social Connected TV Platform API and its documentation. Furthermore they will offer their support to participants regarding the use of the Social Connected TV Platform technologies (cf. “Background to the challenge” for more information). 

If you are interested in getting access to the Social Connected TV Platform API to help you enter this challenge, please contact:

Christoph Ziegler
Platforms for Broadcast Services
Institut fuer Rundfunktechnik GmbH
Floriansmuehlstraße 60, 80939 Munich, Germany
Phone: +49 (0) 89 323 99-380
Email: ziegler@irt.de