Demo Lab 1_adam Fleming_apadmi

Anatomy of a Mobile Audio Streaming Application Adam Fleming CTO Apadmi

Apadmi Background





Small Team of Mobile Experts Developing Mobile Apps for over 10 Years Developing Platform Technology for Symbian Mobile Consultancy & Development Service





Technical Workshops & Training Feasibility Studies, Prototyping Requirements Analysis Application Development, Test


See www.apadmi.com

NPR Player Implementation
Implemented as Native C++ Application





Background playback support required Low-level access into streaming APIs Performance critical Integration with other parts of system
Notably telephony

A Streaming Media Application

UI

Content Browsing

Stream Metadata Playback Status

Control

Audio Streaming API

Streaming Engine

Network API Network Stream

Audio Data

Content Provider
NPR – National Public Radio
(www.npr.org)


Radio Program Syndication
Mostly spoken word (News, Talk etc)


Also Print Format News
Mix of local and wider-interest items
Already have a API providing text, graphical and audio content (www.npr.org/api/index)

Streaming Service Interfacing
NPR supports a simple RESTful HTTPbased interface
Static content is provided as XML
parsed into HTML for display in embedded browser


Streams either live or fixed-length
MP3 data stream


Will look at the network classes a little later

Data Bearers
Cellular Data/WiFi
Speed
Wide range of actually achievable speeds
Differing degrade behaviour


Access
Operator networks, Private networks, Firewalls


Access Point Behaviour
Connection persistence
Authentication


Cost

System Provided API - Network Socket Server

WiFi

Cellular Data

RSocketServ

Shortlink Data

RConnection

RHttpSession

RHostResolver

RSocket

http://developer.symbian.org/wiki/index.php/Sockets_(Fundamentals_of_Symbian_C++) http://developer.symbian.org/wiki/index.php/Symbian_OS_Communications_Programming/11._HTTP

Network – Connecting to an IAP RSocketServ iSocketServ; RConnection iConnection; TUint32 iIap; … // Set up the RConnection User::LeaveIfError(iSocketServ.Connect()); User::LeaveIfError(iConnection.Open(iSocketServ)); … // Set up the Iap TCommDbConnPref connPref; connPref.SetIapId(iIap); connPref.SetDialogPreference(ECommDbDialogPrefDoNotPrompt); connPref.SetDirection(ECommDbConnectionDirectionOutgoing); … // Try to connect iConnection.Start( connPref, iStatus );

Network – HTTP Connection

RHttpSession

RHttpTransaction

MHTTPSessionEventCallback

MHTTPTransactionCallback


Refer to more detailed explanations on Wiki http://developer.symbian.org/wiki/index.php/Symbian_OS_Communications_Programming/11._HTTP

System Provided API - Network
Open RHttpSession
Request a transaction from the session
Set the properties, filters, headers and body of the request
Submit the request
Receive callback through MHTTPTransactionCallback::MHFRunL()
THTTPEvent contains state of transaction

A Streaming Media Application

UI

Content Browsing

Stream Metadata Playback Status

Control

Audio Streaming API

Streaming Engine

Network API Network Stream

Audio Data

Playback Mechanism Considerations
User-interface requirements
Media Playback
“trick modes”
Background Playback
Degradation Behaviour


Requirements for platform integration
Incoming/Outgoing calls

Playback Control API class CNPRPlaybackControl { … void OpenUrlL(CNPRPlaylist* aPlaylist, TBool aFixedLengthStream); void SetPlayList(const CNPRPlaylist* aPlaylist); void CloseUrl(); void PlayL(); void PauseL(); void StopL(); void TogglePlayPause(); virtual TInt SupportMetadata() const; virtual void SetVolume(TInt aVolume); virtual TInt Volume(); … } class CNPRPlayList { … const TPtrC8 URL(TInt aIndex) const; … }

A Streaming Media Application

UI

Content Browsing

Stream Metadata Playback Status

Control

Audio Streaming API

Streaming Engine

Network API Network Stream

Audio Data

Content Browsing API
Provided by NPR
HTTP Requests which return fixed-form XML
Responses parsed and used to generate HTML
Displayed in embedded browser control
Includes an API to allow user to specify current location to retrieve local stories, stations etc

Location - Classes class Location Classes

RPositionServ er CActiv e

Position Serv er

RPositioner

CPositionGetter

TPosition

http://developer.symbian.org/wiki/index.php/Creating_LocationAware_Applications#Positioning_methods_supported_by_the_S60_platform

Location - Setup #include #include … RPositionServer RPositioner TPositionInfo

iLocationServer; iPositioner; iPositionInfo;

… { // open the server sessions User::LeaveIfError(iLocationServer.Connect()); // open positioner using default module User::LeaveIfError(iPositioner.Open(iLocationServer)); // set our application as location requestor User::LeaveIfError(iPositioner.SetRequestor(CRequestor::ERequestorService, CRequestor::EFormatApplication, *iAppName)); }

Location - Use void CGpsPositionRetriever::GetPosition() { … // request position updates iPositioner.NotifyPositionUpdate(iPositionInfo, iStatus); SetActive(); … } void CGpsPositionRetriever::RunL() { … TInt error = iStatus.Int(); if (error == KErrNone) { TPosition pos; iPositionInfo.GetPosition(pos); iObserver.HandleGPSRetrievalL(pos); } }

A Streaming Media Application

UI

Content Browsing

Stream Metadata Playback Status

Control

Audio Streaming API

Streaming Engine

Network API Network Stream

Audio Data

Audio Streaming
The most complex part of the project
Timeliness is critical
Need to smooth out changes in network speed
Must be able to run in background whilst user doing something else
Must not interfere with UI responsiveness

System Provided API – Audio Streaming

Device Drivers

MMF

Application

MDF

CMdaAudio*Stream

Hardware Codec Implementation

Audio Streaming - Classes
CMdaAudioOutputStream
MMdaAudioOutputStreamCallback
TMdaAudioDataSettings
Also
CMdaAudioInputStream
MMdaAudioInputStreamCallback

CMdaAudioOutputStream class CMdaAudioOutputStream : public CBase, public MMMFClientUtility { public: IMPORT_C static CMdaAudioOutputStream* NewL(MMdaAudioOutputStreamCallback& aCallBack, TInt aPriority, TMdaPriorityPreference aPref = EMdaPriorityPreferenceTimeAndQuality); virtual void SetAudioPropertiesL(TInt aSampleRate, TInt aChannels); virtual void Open(TMdaPackage* aSettings); virtual TInt MaxVolume(); virtual TInt Volume(); virtual void SetVolume(const TInt aNewVolume); virtual void WriteL(const TDesC8& aData); virtual void Stop(); virtual const TTimeIntervalMicroSeconds& Position(); IMPORT_C void SetBalanceL(TInt aBalance = KMMFBalanceCenter); IMPORT_C TInt GetBalanceL() const; IMPORT_C TInt GetBytes(); }

MMdaAudioOutputStreamCallback class MMdaAudioOutputStreamCallback { public: virtual void MaoscOpenComplete(TInt aError) = 0; virtual void MaoscBufferCopied(TInt aError, const TDesC8& aBuffer) = 0; virtual void MaoscPlayComplete(TInt aError) = 0; }

System Provided API – Audio Streaming sd Class M odel Pl ayer

CMdaAudi oOutputStream MMF

Open()

Set Up Stream()

MaoscOpenCompl ete()

Wri te()

Wri teData()

MaoscBufferCopi ed()

Wri te() Wri teData()

MaoscBufferCopi ed()

Stop()

Stop Pl ayback()

Done()

MaoscPl aybackCompl ete()

A Streaming Media Application

UI

Content Browsing

Stream Metadata Playback Status

Control

Audio Streaming API

Streaming Engine

Network API Network Stream

Audio Data

Audio Engine – Architecture
Main decision – single threaded, or multithreaded
Consider implementations of both
Highlight strengths and weaknesses

Important Point
The choice between single or multiple threads is not a choice between active objects or threads
Active objects are fundamental to the way in which clients and servers communicate
Callbacks are just wrapped up active objects
If a thread uses any API which uses callbacks, it will need an active scheduler

Single Threaded Architecture





“Simple” Lowest use of system resources Prone to delays caused by system loading Possible for UI to interfere with audio playback

Architectural Description
Conceptually very simple, everything is built into the same application and application logic simply responds to requests and notifications
Single thread semaphore used by all servers which app has a session with – Windows, Socket, Location, etc
Need to ensure that all event handling code completes quickly
Need to consider priority of events

Downfalls of Single-threading
Unbounded time from event completion to RunL
Long running event handlers
Some of these are out of your control
Displaying softkey menu for example


Very frequent events
UI events when a key is held down


Concrete example
Scrolling through a news story
Either scrolling is not smooth or playback stutters

Multi-Threaded Architecture
Dual-threaded
More complex
Heavier loading on system
Less susceptible to delays due to system loading
UI actions should not impact playback performance
Requires mechanism for inter-thread communication

Architectural Description
UI Thread looks after the display and interactions with the user
Audio Streaming Thread responsible for reading data from the network, and sending it on to the Audio Streaming APIs
Need a way to allow the UI thread to control the Audio Streaming Thread
Need a way to allow the Audio Streaming Thread to communicate information back to the UI

Message Queues
Lightweight way to allow small amounts of data to be passed safely between threads
Simpler to use than setting up shared memory, mutexes, signalling semaphores etc
Fire-and-Forget style messaging, ideal for event notifications
Can carry small data types of fixed length http://developer.symbian.org/wiki/index.php/Threads,_Processes,_and_IPC_(Fundament als_of_Symbian_C++)#Message_Queues

Message Queues // Creation : RMsgQueue iStreamMsgQueue; User::LeaveIfError(iStreamMsgQueue->CreateLocal(5)); // Request Notification iStreamMsgQueue->NotifyDataAvailable(iStatus); SetActive(); // retreive data and do something with it. TMsgType msg; iStreamMsgQueue->Receive(msg);

Architecture Continued
Threads communicate by passing messages through the message queue
Both threads in same process, so can pass local pointers as well as TClass instances
Clearer design
Each thread contains a single state-machine

Dual Thread Example sd Class Model UI Thread

Playback Thread

User

Browse Stations()

Download Station Info()

Display Stations() Select Station() Select Station()

Startup() Notify Startup()

Download Packet() Browse News Stories() Play Packet()

Download Story() Read Story()

View Story()

Download Packet()

Advantages
Playback thread is now allocated timeslices independent of the UI
App is still susceptible to system loading, but not to the same degree
Threads are handled independently
UI and playback priorities explicitly set

Gotchas…
Need to know which thread you’re in
Server handles are thread-local


Callbacks are active-objects too
Still need an AS in your thread


Passing data between threads needs protection
Not an issue here, but needs to be remembered

A final word of warning…
Streaming Audio API is well established and understood
But it doesn’t always behave in the way that you would expect
For MP3, MaoscPlaybackComplete does not get called with KErrUnderflow at the end of playback
But it does for all other media types
Probably a defect in MP3 codec implementation

A Streaming Media Application

UI

Content Browsing

Stream Metadata Playback Status

Control

Audio Streaming API

Streaming Engine

Network API Network Stream

Audio Data

Questions?