Measuring Radio Network Performance Gunnar Heikkilä AWARE Advanced Wireless Algorithm Research & Experiments Radio Network Performance, Ericsson Research EN/FAD 109 0015 Düsseldorf (outside)
Düsseldorf (inside) GSM Association IREG QoS ~30 participants from operators and manufacturers Meetings in Düsseldorf 4-6 times/year Specifies how to make performance measurements Not a standard; more like recommendations
Anywhere, Anytime... Antennas and Base Stations
Cell Planning Coverage Maps
Interference Analysis 3D Visualization
System Metrica: Unsuccessful Calls Unsuccessful Call Attempts 3.0% 2.5% 2.0% 1.5% % No Service % Blocked 1.0% 0.5% 0.0% Netw ork 1 Netw ork 2 Netw ork 3 Netw ork 4 System Metrica: Dropped Calls % Dropped Calls 9.0% 8.0% 7.6% 7.0% 6.0% 5.6% 5.8% 5.0% 4.0% 3.0% 2.5% 2.0% 1.0% 0.0% Network 1 Network 2 Network 3 Network 4
System Performance Reports System Performance Reports
Field Measurements Field Measurements
S t ö r a r e What Should Be Measured? Coverage Interference Speech Quality Data Quality
What is the True Speech Quality? Subjectively Measured by Humans Different Measuring Techniques DRT (Diagnostic Rhyme Test) MOS (Mean Opinion Score) DMOS (Degradation MOS) DAM (Diag. Accept. Measure) A/B-Tests Speech Quality is a Statistical Measure Speech Quality Standards
P.800: MOS and DMOS Scales Grade Quality Distortion Level 5 Excellent Imperceptible 4 Good Just perceptible but not annoying 3 Fair Perceptible and slightly annoying 2 Poor Annoying but not objectionable 1 Bad Very annoying and objectionable Speech Dependencies Speaker pitch and formant structure (male - female - children) High dependency Sentence contents Medium to high dependency Language Relatively low dependency Relevant measure Statistical average
P.862 - PESQ Objective Methods 2 x 104 1 0 sweep_jump -1-2 0 1 2 3 4 5 6 Transmitted speech 2 x 104 1 0-1 Jump -2 0 1 2 3 4 5 6 Received speech
Examples Of Bad Speech Quality Audio levels Noise Background noise Bit errors Erased frames Echo Discontinuous transmission Time warping Delay PESQ Structure
Level And Time Alignment IRS Filtering
Level And Time Alignment Synchronisation Dependency Speech Quality Estimate 30 25 20 15 10 5 0-5 -10 Correct 5 ms error 10 ms 20 ms -15-20 0 50 100 150 200 250 300 350 400 Sentence
Perceptual Modelling The Bark Scale
Perceptual Modelling The Sone Scale
Perceptual Modelling Norm Aggregating
Perceptual Modelling Cognitive Modelling
Final PESQ Score System Overview Air Interface Channel Errors + Channel Coder Speech Coder Channel Decoder Speech Decoder Original Speech Received Speech Sending Side Receiving Side
What Affects Speech Quality? Most important Codec capabilities Radio channel disturbancies Also relevant Mobile type Background noise Speech contents Echo problems Modelling Principle Channel Decoder Parameters & Transforms Speech Decoder Multivariate Modelling Model Received Speech Recordings & Listenings Receiving Side
The Full-Rate Model BER Average bit error rate during 2.5 seconds FER Frame erasure rate during 2.5 seconds TFER Square root of FER (max 0.66) LFER Longest erased frame sequence (max 30) SQI 20.67-57.2*BER - 29.3*TFER - 0.11*LFER (min -20) Handover adjustment are made for each handover Sentence Dependency 40 30 PSQM SQI Speech Quality Estimate 20 10 0-10 -20 700 750 800 850 900 950 1000 1050 1100 Sentence
Sentence Dependency, cont. 34 32 PSQM SQI Speech Quality Estimate 30 28 26 24 22 700 710 720 730 740 Sentence Selected Sentences 40 30 PSQM SQI Speech Quality Estimate 20 10 0-10 -20 0 50 100 150 200 250 300 350 400 Sentence
Sentence Averaging 40 30 PSQM 10-SENTENCE AVERAGE Speech Quality Estimate 20 10 0-10 -20 0 50 100 150 200 250 300 350 400 Sentence Field Measurements
C/I Example: Three Frequencies SQI, Signal Strength and C/I. SQI RxLevel 60 40 20 0 0 5 10 15 20 25 30 35 40 ci_recording 30 C/I [db] 20 10 0 0 5 10 15 20 25 30 35 40 Time [s]
S t ö r a r e GSM Burst Structure TDMA-ram 0 1 2 3 4 5 6 7 Tidlucka Burst träningssekvens 3 58 58 3 8.25 26 0, 1, 0, 0, 1, 0, 1, 0
Algorithm Structure I C Carrier detection yes Bit detection Graphical presentation no Channel estimation Correlation + Σ - Carrier estimation Training sequences Ramp model Correlation Curves
Algorithm Performance Graphical Presentation
S t ö r a r e Channels: S t ö r a r e 15 22 46 49 Channels: 10 15 26 Channels: 7 15 23 Channels: 15 19 24 52
Candidate 1 Interference on channel 15 Candidate 3 Channels: S t ö r a r e 15 22 46 49 Candidate 2-50 0 50 100 150 Channels: 10 15 26 Channels: 7 15 23 Channels: 15 19 24 52 Streaming Quality
GPRS Coding Schemes Block Error Probability 1 0.9 blep 0.8 Block errorprobability 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 5 10 15 20 25 30 C/I [db]
Throughput for a 3-slot 3 Mobile 60 50 raw_tp Throughput [kbit/s] 40 30 20 10 0 0 5 10 15 20 25 30 C/I [db] EGPRS Ack/Nack Window
Effect of the RLC Protocol Throughput [kbit/s] 60 50 40 30 20 10 0 0 5 10 15 20 25 30 C/I [db] prot_tp_20_6_3 3-slot mobile 20 blocks poll interval 120 ms roundtrip delay Block Error Probability: Live Recording 1 0.9 rec_blep Block errorprobability 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 20 40 60 80 100 120 Time [s] 1/1 reuse network Hopping on 15 frequencies 50 km/h
Block Error Probability: 2.5 Sec. Filter 1 0.9 0.8 rec_blep_filt Block errorprobability 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 20 40 60 80 100 120 Time [s] 1/1 reuse network Hopping on 15 frequencies 50 km/h Throughput for a 3-slot 3 Mobile 60 50 rec_raw_filt Throughput [kbit/s] 40 30 20 10 0 0 20 40 60 80 100 120 Time [s]
RLC Throughput 60 50 rec_prot_filt Throughput [kbit/s] 40 30 20 10 0 0 20 40 60 80 100 120 Time [s] 3-slot mobile 20 blocks poll interval 120 ms roundtrip delay What is Echo? 4-wire transmission network Hybrid 2-wire subscriber line PSTN subscriber Echo PSTN: Public Switched Telephone Network
Echo Tolerance Echo Return Loss (ERL) db Tolerable Intolerable Echo Tolerance Curve Delay (msec) Analog Networks Digital Networks 200 300 Satellite Requirement and Real Hybrid Data 45 db needed (theoretically) for typical GSM delay Can be relaxed to 35 db considering background noise around the mobile user
Trunk Echo Canceller Digital cellular subscriber Echo Canceller in Pool Digital cellular subscriber Digital cellular system MSC ETC ETC ETC ETC EC EC EC EC PSTN Digital cellular system MSC ETC ETC ETC ETC PSTN ECP ETC: EC: ECP: PSTN: MSC: Exchange Terminal Circuit Echo Canceller Ericsson Echo Canceller in Pool Public Switched Telephone Network Mobile Switching Center Echo Effects on Speech Quality No standard-defined speech quality tests ITU-T tests (G.165, G.168) are rather minimum functionality tests Objective methods are difficult to define The echo path and the conversation characteristics are changing from subscriber to subscriber Subjective speech quality tests are necessary ITU-T P.800 & P.831 for correct methods Only simple objective parameters can be measured today Echo delay Echo amplitude
DTX in Cellular Networks DTX Principle: Send only when there is something important to send (i.e. when talking) Estimate the background noise, and send estimates with 0.5 s intervals during silence in conversation DTX Benefits: Longer battery life in mobile Lower total interference in network DTX Drawbacks: Wrong silence is produced if estimate is bad, which can be extremely disturbing DTX and Speech Quality The impact of silence distortions are not well understood or handled by todays algorithms Most algorithms excludes (or weights down) silent parts when estimating the speech quality For time-coverage reasons, the transmitted speech does not contain (much) silence at all, thus avoiding the DTX problem But the users on the street are experiencing the full DTX effect, thus the measurements are not representative
Music Quality Madonna, original Madonna, 12.2 kbit/s, no channel errors Madonna, 4.75 kbit/s, no channel errors Madonna, 4.75 kbit/s, bad channel Skip The Audio?
Skip the Audio 2... QoS Attributes vs. Quality Real Time Non Real Time Traffic class Conversational class Streaming class Interactive class Background class Maximum bit rate X X X X Delivery order X X X X Maximum SDU size X X X X SDU format information X X SDU error ratio X X X X Residual bit error ratio X X X X Delivery of erroneous SDUs X X X X Transfer delay X X Guaranteed bit rate X X Traffic handling priority X Allocation/Retention priority X X X X
Keep the Stock Exchange Happy...