MPEG-4 Audio Synchronization

Transcription

1 MPEG-4 Audio Synchronization Masayuki Nishiguchi, Shusuke Takahashi, Akira Inoue Oct 22, 2014 Sony Corporation

2 Agenda Use case Synchronization Scheme Extraction tool (Normative) Similarity Calculation Tool (Informative) Performance evaluation Conclusion

3 Audio Synchronization Use case of Second Screen Application Receiver Audio Signal of Main Media Stream Transmitter Main Media Stream Of Main Media Stream Main Device (1 st screen) Synchronization using Audio Feature! Sub Media Stream Internet (IP) foreign language audio tracks audio commentary closed caption information audio/visual contents recorded from various angles high quality audio/visual contents advertisement 1 st screen Sub Device (2 nd screen) 2 nd screen

4 Synchronization Scheme Transmitter Main Media Stream (Synchronous in time) Sub Media Stream Audio Signal Extraction of Main Media Stream MUX Receiver (Sub device) Main Device DeMUX & Playback Sub Device Audio Signal of Main Media Stream Audio Playback device (such as speaker) Main Media Presentation noise 1 st screen Audio Recording Device (such as microphone) Multiplexed Data Stream - Sub Media Stream - of Main Media Stream of Main Media Stream (Transmitted) DeMUX Sub Media Stream Extraction 2 nd screen of Main Media Stream(Extracted) Similarity Calculation Synchronization Information Timing Adjustment & Playback Sub Media Presentation

5 Extraction tool (Normative)

6 Block Diagram of Extraction tool Audio Signal (fs=8khz) Framing Pre Emphasis Filter bank Autocorrelation Integration Peak Detection Frame Rate Conversion

7 Overall Signal Flow split the audio signals into 5 equally spaced frequency bands in log frequency domain Confidence Measure for this band is less than threshold Integrate together into single Auto Correlation Input Signal (After Pre emphasis filter) Integrated Auto Correlation Band split signal Auto Correlation converted into a 128-bit length feature vector Prominent peak : 1 Otherwise : 0

8 Block Diagram of Extraction tool Audio Signal (fs=8khz) Framing Pre Emphasis Filter bank Autocorrelation Integration Peak Detection Frame Rate Conversion

9 Framing Input Input frame interval: 8msec (64sample) Input frame length: 32msec (256samples), Hamming Window Feature Extraction Frame Rate Conversion Output frame interval: 8msec or 32msec (audio_sync_feature_time_resolution) Output

10 Block Diagram of Extraction tool Audio Signal (fs=8khz) Framing Pre Emphasis Filter bank Autocorrelation Integration Peak Detection Frame Rate Conversion

11 Filter Bank For each audio frame, a pre-emphasis filter is applied to emphasize the high frequency, then band pass filtering is applied in order to split the audio signals into 5 equally spaced frequency bands in log frequency domain.

12 Block Diagram of Extraction tool Audio Signal (fs=8khz) Framing Pre Emphasis Filter bank Autocorrelation Integration Peak Detection Frame Rate Conversion

13 Auto-correlation For each band, Auto-correlation is calculated using:, 0, 0 The Auto-correlation is normalized using: 0 0, 0 : input frame length, : index of frequency band : index of lag for autocorrelation : order of auto-correlation and is set to 128, : index of the input audio signal. : number of frequency bands and is set 5 For each frequency band, confidence measure is calculated based on the auto-correlation value. max, 0

14 Block Diagram of Extraction tool Audio Signal (fs=8khz) Framing Pre Emphasis Filter bank Autocorrelation Integration Peak Detection Frame Rate Conversion

15 Integration The normalized auto-correlation function values derived from each sub-band are summed together into a single integrated auto-correlation function. where is defined as following 0, 0.3 1, 0.3, 0

16 Block Diagram of Extraction tool Audio Signal (fs=8khz) Framing Pre Emphasis Filter bank Autocorrelation Integration Peak Detection Frame Rate Conversion

17 Peak Detection The integrated auto-correlation function is converted into a 128-bit length feature vector and each bit position corresponds to the lag of the autocorrelation function. (lag)

18 Similarity Calculation Tool (Informative)

19 Block Diagram Similarity Calculation Tool (Informative) Sequence #1 Sequence #2 Frame Rate Conversion Frame Rate Conversion Block Extraction Block Extraction Block Similarity Calculation Time difference between audio signals

20 Block Diagram Similarity Calculation Tool (Informative) Sequence #1 Sequence #2 Frame Rate Conversion Frame Rate Conversion Block Extraction Block Extraction Block Similarity Calculation Time difference between audio signals

21 Block Extraction The blocks are generated by concatenating the consecutive audio features N N sequence #1 Block Similarity Calculation is performed between two blocks of audio features. N sequence #2,,,.,, 0 N N,,,.,, 0 N N

22 Block Diagram Similarity Calculation Tool (Informative) Sequence #1 Sequence #2 Frame Rate Conversion Frame Rate Conversion Block Extraction Block Extraction Block Similarity Calculation Time difference between audio signals

23 Block Similarity Calculation Block Similarity between and is calculated as follows: J A,B Example A B τ time

24 Time Difference Estimation For each time difference, a score is calculated by using the block similarity as follows: Score 1 min, max, 0 argmax Score, The time difference which has the largest score is regarded as the time difference between two audio feature sequences:, J, j τ 1 N g τ 0 τ 2 Each box represents block similarity J, i -τ 0 N g -τ 2 the summations for each time difference is performed along the arrows. N f Score(τ 0 ) Score(τ 1 ) Score(τ 2 )

25 Time Difference Estimation Example delay i j

26 Performance evaluation Capturing the 1 st screen content and additive noise sound at the 2 nd screen. The noise contaminated 1 st screen content files Line-out of the 1 st screen and the 2 nd screen are captured as a single stereo wave file. Time difference between the L-ch and the R-ch in the file is measured

27 1 st Screen content and additive noise files Filename of 1 st screen content files 1st_betty 5.1 The 1 st Screen Content Files Description down mix (according to ARIB STD-B32) version of CO_11_Betty3b_output 1st_speech 2 Speech (German Male, SQAM track 54) 1st_music 2 Music (Wind ensemble, SQAM track 67) Additive Noise Sound Files Filename of additive noise sound files Description File4 noise_pinknoise File5 noise_speech Speech (English Female, SQAM track 49) File6 noise_music Music (Eddie Rabbitt, SQAM track 70)

28 Result Time Difference between 1 st Screen and 2 nd Screen Line-Out Signals (sec) Filename of 1 st screen content files 1st_betty 1st_speech 1st_music Filename of additive noise sound files Signal level of the 1 st screen content (db) noise_pinknoise N/A N/A N/A N/A noise_speech N/A N/A N/A noise_music N/A N/A N/A noise_pinknoise N/A N/A N/A noise_speech N/A N/A noise_music noise_pinknoise N/A N/A N/A N/A noise_speech N/A N/A noise_music N/A N/A The figures with orange background is approximately within 1frme length (32ms). Synchronization is successful!

29 Result (cont.) Synchronization robustness against interference noise 24 Allowable SNR for synchronization measured at 2 nd screen(db) Additive noise 1st screen content pinknoise speech music pinknoise speech music pinknoise speech music betty speech music

30 MPEG-4 Audio Object Type (ISO/IEC :2009) Object Type ID Audio Object Type gain control [ ] Remark 0 Null [..] [ ] 43 SAOC 44 LD MPEG Surround 45 SAOC-DE 46 Audio Sync (reserved)

31 Demonstration 1 st screen(blue walkman): Instrument only 2 nd screen(my note PC): Vocal only Same song Noise(white walkman): Female speech

32 Conclusion MPEG-4 Audio Synchronization standard defines: Extraction tool and syntax of the feature stream(normative) Feature Similarity Calculation Tool (Informative) The Audio Object Type (AOT=46) Audio Sync to allow transmission of audio feature for synchronization as elementary stream The MPEG-4 synchronization mechanism works with highly noisy environment and proven that the scheme is useful under practical conditions.

33 End