Measuring Radio Network Performance

Transcription

1 Measuring Radio Network Performance Gunnar Heikkilä AWARE Advanced Wireless Algorithm Research & Experiments Radio Network Performance, Ericsson Research EN/FAD Düsseldorf (outside)

2 Düsseldorf (inside) GSM Association IREG QoS ~30 participants from operators and manufacturers Meetings in Düsseldorf 4-6 times/year Specifies how to make performance measurements Not a standard; more like recommendations

3 Anywhere, Anytime... Antennas and Base Stations

4 Cell Planning Coverage Maps

5 Interference Analysis 3D Visualization

6 System Metrica: Unsuccessful Calls Unsuccessful Call Attempts 3.0% 2.5% 2.0% 1.5% % No Service % Blocked 1.0% 0.5% 0.0% Netw ork 1 Netw ork 2 Netw ork 3 Netw ork 4 System Metrica: Dropped Calls % Dropped Calls 9.0% 8.0% 7.6% 7.0% 6.0% 5.6% 5.8% 5.0% 4.0% 3.0% 2.5% 2.0% 1.0% 0.0% Network 1 Network 2 Network 3 Network 4

7 System Performance Reports System Performance Reports

8 Field Measurements Field Measurements

9 S t ö r a r e What Should Be Measured? Coverage Interference Speech Quality Data Quality

10 What is the True Speech Quality? Subjectively Measured by Humans Different Measuring Techniques DRT (Diagnostic Rhyme Test) MOS (Mean Opinion Score) DMOS (Degradation MOS) DAM (Diag. Accept. Measure) A/B-Tests Speech Quality is a Statistical Measure Speech Quality Standards

11 P.800: MOS and DMOS Scales Grade Quality Distortion Level 5 Excellent Imperceptible 4 Good Just perceptible but not annoying 3 Fair Perceptible and slightly annoying 2 Poor Annoying but not objectionable 1 Bad Very annoying and objectionable Speech Dependencies Speaker pitch and formant structure (male - female - children) High dependency Sentence contents Medium to high dependency Language Relatively low dependency Relevant measure Statistical average

12 P PESQ Objective Methods 2 x sweep_jump Transmitted speech 2 x Jump Received speech

13 Examples Of Bad Speech Quality Audio levels Noise Background noise Bit errors Erased frames Echo Discontinuous transmission Time warping Delay PESQ Structure

14 Level And Time Alignment IRS Filtering

15 Level And Time Alignment Synchronisation Dependency Speech Quality Estimate Correct 5 ms error 10 ms 20 ms Sentence

16 Perceptual Modelling The Bark Scale

17 Perceptual Modelling The Sone Scale

18 Perceptual Modelling Norm Aggregating

19 Perceptual Modelling Cognitive Modelling

20 Final PESQ Score System Overview Air Interface Channel Errors + Channel Coder Speech Coder Channel Decoder Speech Decoder Original Speech Received Speech Sending Side Receiving Side

21 What Affects Speech Quality? Most important Codec capabilities Radio channel disturbancies Also relevant Mobile type Background noise Speech contents Echo problems Modelling Principle Channel Decoder Parameters & Transforms Speech Decoder Multivariate Modelling Model Received Speech Recordings & Listenings Receiving Side

22 The Full-Rate Model BER Average bit error rate during 2.5 seconds FER Frame erasure rate during 2.5 seconds TFER Square root of FER (max 0.66) LFER Longest erased frame sequence (max 30) SQI *BER *TFER *LFER (min -20) Handover adjustment are made for each handover Sentence Dependency PSQM SQI Speech Quality Estimate Sentence

23 Sentence Dependency, cont PSQM SQI Speech Quality Estimate Sentence Selected Sentences PSQM SQI Speech Quality Estimate Sentence

24 Sentence Averaging PSQM 10-SENTENCE AVERAGE Speech Quality Estimate Sentence Field Measurements

25 C/I Example: Three Frequencies SQI, Signal Strength and C/I. SQI RxLevel ci_recording 30 C/I [db] Time [s]

26 S t ö r a r e GSM Burst Structure TDMA-ram Tidlucka Burst träningssekvens , 1, 0, 0, 1, 0, 1, 0

27 Algorithm Structure I C Carrier detection yes Bit detection Graphical presentation no Channel estimation Correlation + Σ - Carrier estimation Training sequences Ramp model Correlation Curves

28 Algorithm Performance Graphical Presentation

29 S t ö r a r e Channels: S t ö r a r e Channels: Channels: Channels:

30 Candidate 1 Interference on channel 15 Candidate 3 Channels: S t ö r a r e Candidate Channels: Channels: Channels: Streaming Quality

31 GPRS Coding Schemes Block Error Probability blep 0.8 Block errorprobability C/I [db]

32 Throughput for a 3-slot 3 Mobile raw_tp Throughput [kbit/s] C/I [db] EGPRS Ack/Nack Window

33 Effect of the RLC Protocol Throughput [kbit/s] C/I [db] prot_tp_20_6_3 3-slot mobile 20 blocks poll interval 120 ms roundtrip delay Block Error Probability: Live Recording rec_blep Block errorprobability Time [s] 1/1 reuse network Hopping on 15 frequencies 50 km/h

34 Block Error Probability: 2.5 Sec. Filter rec_blep_filt Block errorprobability Time [s] 1/1 reuse network Hopping on 15 frequencies 50 km/h Throughput for a 3-slot 3 Mobile rec_raw_filt Throughput [kbit/s] Time [s]

35 RLC Throughput rec_prot_filt Throughput [kbit/s] Time [s] 3-slot mobile 20 blocks poll interval 120 ms roundtrip delay What is Echo? 4-wire transmission network Hybrid 2-wire subscriber line PSTN subscriber Echo PSTN: Public Switched Telephone Network

36 Echo Tolerance Echo Return Loss (ERL) db Tolerable Intolerable Echo Tolerance Curve Delay (msec) Analog Networks Digital Networks Satellite Requirement and Real Hybrid Data 45 db needed (theoretically) for typical GSM delay Can be relaxed to 35 db considering background noise around the mobile user

37 Trunk Echo Canceller Digital cellular subscriber Echo Canceller in Pool Digital cellular subscriber Digital cellular system MSC ETC ETC ETC ETC EC EC EC EC PSTN Digital cellular system MSC ETC ETC ETC ETC PSTN ECP ETC: EC: ECP: PSTN: MSC: Exchange Terminal Circuit Echo Canceller Ericsson Echo Canceller in Pool Public Switched Telephone Network Mobile Switching Center Echo Effects on Speech Quality No standard-defined speech quality tests ITU-T tests (G.165, G.168) are rather minimum functionality tests Objective methods are difficult to define The echo path and the conversation characteristics are changing from subscriber to subscriber Subjective speech quality tests are necessary ITU-T P.800 & P.831 for correct methods Only simple objective parameters can be measured today Echo delay Echo amplitude

38 DTX in Cellular Networks DTX Principle: Send only when there is something important to send (i.e. when talking) Estimate the background noise, and send estimates with 0.5 s intervals during silence in conversation DTX Benefits: Longer battery life in mobile Lower total interference in network DTX Drawbacks: Wrong silence is produced if estimate is bad, which can be extremely disturbing DTX and Speech Quality The impact of silence distortions are not well understood or handled by todays algorithms Most algorithms excludes (or weights down) silent parts when estimating the speech quality For time-coverage reasons, the transmitted speech does not contain (much) silence at all, thus avoiding the DTX problem But the users on the street are experiencing the full DTX effect, thus the measurements are not representative

39 Music Quality Madonna, original Madonna, 12.2 kbit/s, no channel errors Madonna, 4.75 kbit/s, no channel errors Madonna, 4.75 kbit/s, bad channel Skip The Audio?

40 Skip the Audio 2... QoS Attributes vs. Quality Real Time Non Real Time Traffic class Conversational class Streaming class Interactive class Background class Maximum bit rate X X X X Delivery order X X X X Maximum SDU size X X X X SDU format information X X SDU error ratio X X X X Residual bit error ratio X X X X Delivery of erroneous SDUs X X X X Transfer delay X X Guaranteed bit rate X X Traffic handling priority X Allocation/Retention priority X X X X

41 Keep the Stock Exchange Happy...