Sound Sence

SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones Authored By :Hong Lu, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury and Andrew T. Campbell Department of Computer Science Dartmouth College Presentation Given By:Gaurang Dudhat Stevens Institute Of Technology

Outline of Presentation  Introduction  Design considerations  Sound sense Architecture  Sound sense Algorithms 

Implementation

 Evaluation  Application  Related Work  Conclusion

Introduction •

Perhaps the most ubiquitous and unexploited sensor on mobile phones is the microphone- a powerful sensor that is capable of making sophisticated inferences about human activity, location, and social events from sound.

•

In this paper author exploit this untapped sensor not in the context of human communications but as an enabler of new sensing applications.

•

A key design goal of SoundSence is the scalability of classification to a large population. Specifically, the contribution of this paper are as follows

o An architecture and a set of algorithms for multistage, hierarchal classification

of sound events on mobile phones. o

Address the scaling problem through the introduction of an adaptive unsupervised learning algorithm to classify significant sound events in individuals user’s environment.

o Impalement the soundsence system architecture algorithms on the apple I

phone.

Design Considerations 1) Scaling Sound Classification •

The soundsence system is designed to specifically attempt to make progress toward addressing this important scalability problem.

•

In, essence soundsence uses different strategies when dealing with different sounds.

•

In the first stage, sound is classified as one of the three coarse sound categories: voice, music, ambient sound.

•

In the second stage, further analysis is applied according to the category of the sound.

•

When soundsence determines a new sound to be significant, it prompts the enduser to either provide a textual description or rejects the sound as unimportant or sensitive in terms of privacy.

Design Considerations 2) Phone Context •

The location of a phone with respect to the body, where a phone is used and the conditions under which it is used is collectively referred to as the phone context.

•

The phone context presents a number of challenges to building a robust sound sensing system because sound can be muffed, for example, when the phone is in the pocket or backpack.

•

A goal of sound sense is to support robust sound processing and classification under different phone context conditions, which vary the volume level.

Design Considerations 3) Privacy Issues and Resource Limitations •

The microphone on a phone is typically designed for capturing the human voice, not ambient sounds, and typically sample at 8 KHz. According to the Nyquist Shannon sampling theorem, the microphone cannot capture information above 4 KHz, and, as a result, important information is lost, for example high frequency component of music.

•

In soundsence, sounds need to be analyzed efficiently such that real-time classification is possible while not overwhelming the CPU and memory of the phone

•

Therefore, the designer has to consider the accuracy and cost trade off. This is a significant challenges when designing classification algorithms that have to efficiently run on the phone, without impacting the main function of the phone. For example voice communication.

Soundsense Architecture

Soundsense Architecture

Soundsense Algorithms  Preprocessing

1) Framing •

The frame width needs to be short enough so that the audio is stable and meanwhile long enough to capture the characteristics signature of the sound.

•

Given the resource constraints of the we use independent non-overlapping frames of 64ms. This frame width is slightly larger than what is typically used in other forms of audio processing where the width typically ranges between 25-46 ms.

Soundsense Algorithms

2) Frame Admission Control •

Frame admission control is required since frames may contain audio content that is not interesting for example white noise or is not able to be classified.

•

These frame can occur at any time due to phone context; for example, the phone may be at a location that is virtually silent for example library, home during night.

•

Frame admission is done on the basis of energy level and spectral entropy. Low energy level indicates silence or undesirable phone context, which prevents meaningful classification.

•

To compute spectral entropy; we need to perform these three step. 1)apply hanning window to the frame, which suppresses the boundaries and thus reduces the known effect of FFT spectral leakage

Soundsense Algorithms 2) Calculate the FFT spectrum of the frame 3) Normalize the spectrum, treat it as a probability density function and finally obtain the spectral entropy Hf, by

Acoustic events captured by the phone’s microphone should have reasonable high RMS values, which means the volume of the sound sample is not too low.

Soundsense Algorithms  Coarse category Classification

1) Feature Extraction •

Zero crossing rate, Low energy frame rate, Spectral Flux, Spectral Roll off, Spectral Centroid, Bandwidth, relative Spectral entropy. 2) Multi level classification

•

The Markov models are trained from the output sequence of the decision tree, which are the category assignments of the sound sample.

•

The models are trained to learn the pair wise transition probabilities.

Soundsense Algorithms

Soundsense Algorithms

3) Finite Intra-Category Classification •

The purpose of finer intra-category classification is to allow further analysis of sound events.

•

Much of the previous work on audio signal processing is performed using audio input containing data only from one audio category.

•

Once the category of the sound event is identified by category classification, detailed analysis can be done to provide type specific information of the input signal, according to the requirement of the application.

Implementation  The sound sense prototype system is implemented as a self contained piece of

software that runs on the apple I phone.  Current version is approximately 5,500 lines of code and is a mixture of C,

C++ and objective C. Objective c is necessary to build an apple I Phone application which allows us to access hardware and construct a GUI  The PCM formatted data is placed in a three-buffer circular queue, with each

buffer holding an entire frame.  If there is a lack of an acoustic event, the system enters into a long duty cycle

state in which only one frame in every ten frames is processed. If the frame is accepted by the frame admission control, which means an event has been detected, then processing become continuous.

Implementation

Implementation

Evaluation  CPU and Memory Benchmarks •

Measure the elapsed time for processing a frame to be around 20 to 30 ms, depending on the particular path through the processing workflow.

•

Memory consumption is potentially more dynamic and depends on how many bins are in use by the unsupervised adaptive classifier.

•

This result indicate that our software preserves enough resources for the 3rd party applications or further sound sense extensions, such as more intracategory classifiers.

Evaluation  Classification performance

1) Course category classifier Explore the two thing

 The effectiveness of the decision tree subcomponent  The advantages of the secondary Markov model layer for smoothing of the coarse category classifier.

Evaluation First is confusion matrix for the decision tree classifier Second is confusion matrix for the decision tree classifier with Markov model smoothing

Evaluation 2) Finer Intra-category Classifier •

Currently Implement only a single Intra-category classifier-the gender classifier

•

This classifier is fairly simple in comparison to other example found in the literature

•

According to data 72% classification accuracy.

Evaluation 3) Unsupervised adaptive ambient sound learning

Applications Audio Daily Diary on Oppotunistic Sensing

Applications

Applications Music Detector based on participatory Sensing •

The ability to recognize a board array of sound categories opens up interesting

•

Application spaces for example within the domain of participatory sensing.

•

Built on sound sense on the iPhone.

•

Used the sound category of music and a deployment within Hanover, a small new England town where Dartmouth collage is located.

•

The goal of the application is to provide students with a way to discover events that are associated with music being played.

Applications

Related Work  There has been significant work on audio analysis and signal processing.  The basic problem of sound classification has been as an active area of

research including some of the challenges overcome with sound sense.  Existing work that considers problems such as sound reorganization or audio

scene reorganization do not prove their techniques on resource limited hardware.  Also benefited from audio processing research that considers problems other

than sound classification. For example, work on speech recognition, speaker identification and music genre classification.

Conclusion  Sound sense, an audio event classification system specifically designed for resource limited mobile phones.  The hierarchical classification architecure that is light-weight and scalable yet

capable of recognizing a broad set of sound events.  The ambient sound learning algorithm adaptively learns a unique set acoustic

events for each individual user, and provides a powerful and scalable framework for modeling personalized context. 

Sound sense carries out all the sensing and classification tasks exclusively on the mobile phone without undermining the main functions of the phone.

 The flexibility and scalability of Sound Sense makes it suitable for a wide

range of people-centric sensing applications and present two simple proof-ofconcept applications in this paper.