A Turbo Tuorial by Jakob Dahl Andersen COM Cener Technical Universiy of Denmark hp:\\www.com.du.dk/saff/jda/pub.hml Conens. Inroducion........................................................ 3 2. Turbo Codes....................................................... 5 2. Encoding................................................... 5 2.2 Firs Decoding............................................... 8 2.3 Puing Turbo on he Turbo Codes................................ 9 2.4 Performance Example........................................ 3. APP Decoding...................................................... 3 4. Final Remarks..................................................... 9 5. Seleced Lieraure.................................................. 2
2
. Inroducion The heory of error correcing codes has presened a large number of code consrucions wih corresponding decoding algorihms. However, for applicaions where very srong error correcing capabiliies are required hese consrucions all resul in far oo complex decoder soluions. The way o comba his is o use concaenaed coding, where wo (or more) consiuen codes are used afer each oher or in parallel - usually wih some kind of inerleaving. The consiuen codes are decoded wih heir Figure Concaenaed coding. respecive decoders, bu he final decoded resul is usually sub-opimal. This means ha beer resuls migh be achieved wih a more complicaed decoding algorihm - like he brue-force rying of all possible codewords. However, concaenaed coding offers a nice rade of beween error correcing capabiliies and decoder complexiy. Concaenaed coding is illusraed in Figure. Here we see he informaion frame illusraed as a square - assuming block inerleaving - and we see he pariy from he verical encoding and he pariy from he horizonal encoding. For serial concaenaion he pariy bis from one of he consiuen codes are encoded wih he second code and we have pariy of pariy. If he codes are working in parallel, we do no have his addiional pariy. The idea of concaenaed coding fis well wih Shannon s channel coding heorem, saing ha as long as we say on he righ side of he channel capaciy we can correc everyhing - if he code is long enough. This also means ha if he code is very long, i does no have o be opimal. The lengh in iself gives good error correcing capabiliies, and concaenaed coding is jus a way of consrucing - and especially decoding - very long codes. 3
2. Turbo Codes 2. Encoding Figure 2 Turbo encoder The basic idea of urbo codes is o use wo convoluional codes in parallel wih some kind of inerleaving in beween. Convoluional codes can be used o encode a coninuous sream of daa, bu in his case we assume ha daa is configured in finie blocks - corresponding o he inerleaver size. The frames can be erminaed - i.e. he encoders are forced o a known sae afer he informaion block. The erminaion ail is hen appended o he encoded informaion and used in he decoder. The sysem is illusraed in Figure 2. We can regard he urbo code as a large block code. The performance depends on he weigh disribuion - no only he minimum disance bu he number of words wih low weigh. Therefore, we wan inpu paerns giving low weigh words from he firs encoder o be inerleaved o paerns giving words wih high weigh for he second encoder. Convoluional codes have usually been encoded in heir feed-forward form, like 2 2 (G,G2)=(+D,+D+D ). However, for hese codes a single, i.e. he sequence...000000..., will give a codeword which is exacly he generaor vecors and he weigh of his codeword will in general be very low. I is clear ha a single will propagae hrough any inerleaver as a single, so he conclusion is ha if we use he codes in he feed-forward form in he urbo scheme he resuling code will have a large number of codewords wih very low weigh. The rick is o use he codes in heir recursive sysemaic form where we divide wih one of 2) 2 he generaor vecors. Our example gives (,G2/G)=(,(+D+D /(+D )). This operaion does no change he se of encoded sequences, bu he mapping of inpu sequences o ou- 4
pu sequences is differen. We say ha he code is he same, meaning ha he disance properies are unchanged, bu he encoding is differen. In Figure 3 we have shown an encoder on he recursive sysemaic form. The oupu sequence we go from he feed-forward encoder wih a single is now obained wih he 2 inpu +D =G. More imporan is he fac ha a single gives a codeword of semi-infinie weigh, so wih he recursive sysemaic encoders we may have a chance o find an inerleaver where informaion paerns giving low weigh words from he firs encoder are inerleaved o paerns giving words wih high weigh from he second encoder. The mos criical inpu paerns are now paerns of weigh 2. For he example code he informaion sequence...000... will give an oupu of weigh 5. Figure 3 Recursive sysemaic encoder Noice ha he fac ha he codes are sysemaic is jus a coincidence, alhough i urns ou o be very convenien for several reasons. One of hese is ha he bi error rae (BER) afer decoding of a sysemaic code can no exceed he BER on he channel. Imagine ha he received pariy symbols were compleely random, hen he decoder would of course sick o he received version of he informaion. If he pariy symbols a leas make some sense we would gain informaion on he average and he BER afer decoding will be below he BER on he channel. One hing is imporan concerning he sysemaic propery, hough. If we ransmi he sysemaic par from boh encoders, his would jus be a repeiion, and we know ha we can consruc beer codes han repeiion codes. The informaion par should only be ransmied from one of he consiuen codes, so if we use consiuen codes wih rae /2 he final rae of he urbo code becomes /3. If more redundancy is needed, we mus selec consiuen codes wih lower raes. Likewise we can use puncuring afer he consiuen encoders o increase he rae of he urbo codes. Now comes he quesion of he inerleaving. A firs choice would be a simple block inerleaver, i.e. o wrie by row and read by column. However, wo inpu words of low 5
weigh would give some very unforunae paerns in his inerleaver. The paern is shown in Figure 4 for our example code. We see ha his is exacly wo imes he criical woinpu word for he horizonal encoder and wo imes he criical wo-inpu paern for he verical encoder as well. The resul is a code word of low weigh (6 for he example code) - no he lowes possible, bu since he paern appears a every posiion in he inerleaver we would have a large number of hese words. This ime he rick is o use a pseudo-random inerleaver, i.e. o read he informaion bis o he second encoder in a random (bu fixed) order. The paern from Figure 4 may sill appear, bu no nearly as ofen. On he oher hand we now have he possibiliy ha a criical wo-inpu paern is inerleaved o anoher criical wo-inpu paern. The probabiliy ha a specific wo-inpu.............. 0 0 0 0 0...... 0 0 0...... 0 0 0 0 0...... 0 0 0...... 0 0 0 0 0.............. Figure 4 Criical paern in block inerleaver paern is inerleaved o anoher (or he same) specific wo-inpu paern is 2/N, where N is he size of he inerleaver. Since he firs paern could appear a any of he N posiions in he block, we mus expec his unforunae mach o appear 2 imes in a pseudo-random inerleaver of any lengh. Sill he pseudo random inerleaver is superior o he block inerleaver, and he pseudo-random inerleaving is sandard for he urbo codes. I is possible o find inerleavers ha are slighly beer han he pseudo-random ones, some papers on his opic are included in he lieraure lis. We will end his secion by showing a more deailed drawing of a urbo encoder, Figure 5. Here we see he wo recursive sysemaic encoders, his ime for he code 4 2 3 4 (,(+D )/(+D+D +D +D )). Noice ha he sysemaic bi is removed from one of hem. A he inpu of he consiuen encoders we see a swich. This is used o force he encoders o he all-zero sae - i.e. o erminae he rellis. The complee incoming frame is kep in a 6
buffer from where i is read ou wih wo differen ses of addresses - one for he original sequence and one for he inerleaved one. This way oupu and oupu 2 correspond o he same frame and can be merged before ransmission. Figure 5 Turbo encoder example 2.2 Firs Decoding Decoding of error correcing codes is basically a comparison of he probabiliies for differen codewords - or wih convoluional codes, differen pahs in he rellis. When we alk abou probabiliies, i is always he probabiliy of some even given a cerain amoun of informaion abou his even. This is especially clear when we alk abou probabiliies of somehing ha has already happened - which is always he case in coding heory. Wha we mean when we alk abou he probabiliy ha x was sen, p(x), is he probabiliy ha x was sen given he amoun of informaion ha we have abou he even. Usually ha is only he received noisy version of x - and of course knowledge of he coding scheme, ransmission link ec. In some cases we have some knowledge of he ransmied signal - before we decode he received one. Tha may be informaion ha some messages are more likely o occur han ohers or informaion from oher ransmied sequences. We call his informaion a priori informaion and have he corresponding a priori probabiliies. Similar we alk abou a poseriori probabiliies when we have included boh he a priori informaion probabiliies and he informaion gained by he decoding. For urbo codes we have wo encoded sequences. Clearly we mus sar by decoding one of hem o ge a firs esimae of he informaion sequence. This esimae should hen be used as a priori informaion in he decoding of he second encoded sequence. This requires ha 7
he decoder is able o use a sof decision inpu and o produce some kind of sof oupu. The decoding is skeched in Figure 6. Figure 6 Firs decoding sage The sandard decoder for urbo codes is he A Poseriori Probabiliy decoding (APP) (someimes referred o as he Maximum A Poseriori decoding algorihm (MAP)). The APP decoder, described in Secion 3, does indeed calculae he a poseriori probabiliies for each informaion bis. We will represen he sof inpu/oupu as log-likelihood raios, i.e. a signed number where negaive numbers indicae ha zero is he mos likely value of he bi. As seen from Formula he log-likelihood raio of he a poseriori probabiliies can easily be divided ino wo componens - he log- likelihood raio of he a priori probabiliies of he bi d and he informaion gained by he curren observaion. This means ha when we gain addiional informaion abou he informaion bis - like wih he second decoding - we simply add a (negaive or posiive) componen o he log-likelihood raio. (d ) log Prd,observaion Prd 0,observaion log Pr ap d Pr ap d 0 log Pr ap d Pr ap d 0 log Probservaiond Probservaiond 0 (d ) () 8
2.3 Puing Turbo on he Turbo Codes. When we have a pariy equaion, i involves a number of informaion bis. Le us look a one of he simples possible pariy equaions - a sum of wo informaion bis: P=I +I. I is 2 clear ha if boh P and I are very reliable we ge a reliable esimae of I, on he oher 2 hand if I is very unreliable we do no ge much informaion abou I. If we now imagine 2 ha boh I and I are unreliable when we decoded he firs sequence, bu ha I is in- 2 2 volved in some pariy equaions wih very reliable bis in he second encoded sequence - hen we migh reurn o he pariy equaions from he firs sequence for a second ieraion wih his new and much more reliable esimae of I. This way we could coninue o 2 decode he wo encoded sequences and ierae owards he final decision. However, i is no ha easy since we mus be very careful no o use our informaion more han once. Luckily we see from Formula, ha i is easy o subrac he a priori informaion - which came from he oher decoder - from he decoder oupu. This will preven mos of he unwaned posiive feed-back. We may sill have loops in he decision process, hough, i.e. we migh see ha I influences I in he firs decoder, ha I 2 2 influences I in he second decoder and finally ha I influences I in he nex ieraion in 3 3 he firs decoder. This way he new improved esimae of I will be based on informaion ha came from I in he firs place. Use of he sysem in pracice has shown ha if we subrac he log-likelihood raio of he a priori informaion afer each consiuen decoder and make a number of decoding ieraions we ge a sysem ha is working remarkably well - for many applicaions i acually ouperforms he previously known sysems. Sill, we mus conclude ha he final resul afer urbo decoding is a sub-opimal decoding due o he loops in he decision process. For low signal-o-noise raios we may even see ha he decoding does no converge o anyhing close o he ransmied codeword. The urbo decoder is shown in Figure 7. 9
Figure 7 Turbo decoder 2.4 Performance Example We will show an example of he performance wih urbo codes. We use he sysem illus- 4) 2 3 4 raed in Figure 5, i.e. he code (,(+D /(+D+D +D +D ) for boh encoders bu he infor- maion sequence is only ransmied from he firs one. This means ha he over-all rae is /3. The block lengh is 0384 bis and we use a pseudo-random inerleaver. Afer each frame he encoders are forced o he zero sae. The corresponding erminaion ail - 4 informaion bis and 4 pariy bis for each encoder, a oal of 6 bis - is appended o he ransmied frame and used in he decoder. In principle he erminaion reduces he rae, bu for large frames his has no pracical influence. In his case he rae is reduced from 0.3333 o 0.3332. The performance curves for Bi Error Rae (BER) and Frame Error Rae (FER) are shown in Figure 8. Due o he sub-opimal decoding he performance curves consis of wo pars. For low signal-o-noise raios he main problem is lack of convergence in he ieraed decoding process, resuling in frames wih a large number of errors. In his region we are far from opimal decoding. This means ha we may benefi from more ieraions. As we see from he figure here is a considerable gain by going from 8 o 8 ieraions, and wih more ieraions he performance migh be even beer. 0
For high signal-o-noise raios he decoding is almos opimal, and he main problem is codewords of low weigh. This region is usually referred o as he error-floor since he improvemen for increasing signal-o-noise raio is very small. In spie of he name i is no a rue floor, since he BER and FER is consanly decreasing - alhough no nearly as fas as for he low signal-o-noise raios. Noice ha when he signal o noise raio is high a small number of ieraions is sufficien. Figure 8 Simulaion resuls.
3. APP Decoding The A Poseriori Probabiliy (APP) algorihm does in fac calculae he a poseriori probabiliies of he ransmied informaion bis for a convoluional code. In his presenaion we will resric ourselves o convoluional codes wih rae /n. The convoluional encoder wih memory M (Figure 3) may be seen as a Markov source M wih 2 saes S, inpu d and oupu X. The oupu X and he new sae S are funcions of he inpu d and he previous sae S. - If he oupu X is ransmied hrough a Discree Memoryless Channel wih whie Gauss ian noise. The probabiliy of receiving Y when X was sen is PrY X n j 2 (y j x j )2 2 e 2 (2) where x is he j-h bi of he ransmied word X, and y he corresponding received j 2 value. The signal o noise raio is E s/n 0=/2. In principle knowledge of he signal-o- noise raio is needed for he APP algorihm. However, i may be chosen o a fixed value - depending on he operaion poin of he sysem - wih only a small degradaion of he performance. j Assume ha we receive he sequence Y = Y,Y,...Y. The a poseriori probabiliies of L 2 L he sae ransiions (i.e. branches) are found as: PrS m,s my L PrS m,s m,y L,...L PrY L (3) L Pr { Y } is a consan for a given received sequence and since we consider rae /n codes only, here is one specific informaion bi associaed wih each sae ransiion. We herefore define 2
(i,m )Prd i,s m,y L (4) The final log-likelihood raio becomes (d ) log Prd,observaion Prd 0,observaion (,m ) m (0,m ) m (5) In order o calculae (i,m ) we define he following probabiliy funcions (m), (m), and (i,m ) as (m)prs m,y (6) (m)pry L S m (7) (i,m )Prd i,y S m (8) Compared o he Vierbi algorihm (m) corresponds o he sae merics, while (i,m ) corresponds o he branch merics. (m) can be seen as backwards sae merics. For he noaion we will also need he funcion giving he new encoder sae S when S - = m and d =i newsae(i,m ) and he funcion giving he old encoder sae S when S =m and d =i - oldsae(i,m) Since he encoder is a Markov process and he channel is memoryless, we have PrY L S m,y PrY L S m () and 3
(i,m )PrS m,y Prd i,y S m PrY L S newsae(i,m ) (2) (m ) (i,m ) (newsae(i,m )) If we assume ha he frames are erminaed o sae 0, we have (0)=, and (m)= 0, 0 0 M m=,2,...2. We can calculae as a forward recursion (m) Prd i,s oldsae(i,m),y i0, PrS oldsae(i,m),y Prd i,y S oldsae(i,m) i0, (3) i0, (oldsae(i,m)) (i,oldsae(i,m)) M A he end of he frame we have (0)=, and (m)=0, m=,2,...2. We can calculae as a backward recursion (m) Prd i,y L S m i0, L L Prd i,y S mpry L 2 S newsae(i,m) i0, (4) i0, (i,m) (newsae(i,m)) If he frames are no erminaed we have no knowledge of he iniial and final saes. In -M his case we mus use 0(m)= L(m)=2. Since (m)prs m,y becomes very small wih increasing some rescaling mus be used. In principle he funcion (m) should be used (m) PrS my PrS m,y PrY (5) where Pr{Y } is found as he sum of (m) over all saes, meaning ha he '(m) values always add up o one. However, since he oupu is he log-likelihood raio he acual 4
rescaling is no imporan as long as underflows are avoided. Similar he funcion (m) needs rescaling. The algorihm skeched here requires ha (m) is sored for he complee frame since we have o awai he end of he frame before we can calculae (m). We can insead use a sliding window approach wih period T and raining period Tr. Firs (m) is calculaed and sored for =0 o T-. The calculaion of (m) is iniiaed a ime =T+Tr- wih iniial condiions T+Tr- -M (m)=2. The firs Tr values (m) is discarded bu afer he raining pe- riod, i.e. for =T- down o 0, we assume ha (m) is correc and ready for he calculaion of (i,m ). Afer he firs window we coninue wih he nex one unil we reach he end of he frame where we use he rue final condiions for (m). L Of course, his approach is an approximaion bu if he raining period is carefully chosen he performance degradaion can be very small. Since we have only one oupu associaed wih each ransiion, we can calculae (i,m ) as (i,m )Pr apriori d ipry d i,s m (6) For urbo codes he a priori informaion ypically arrives as a log-likelihood raio. Luckily we see from he calculaion of (m) and (m) ha (i,m ) is always used in pairs - (0,m ) and (,m ). This means we can muliply (i,m ) wih a consan k Pr apriori d 0 (7) and ge (,m ) Pr apriori d Pr apriori d 0 PrY d,s m (8) (0,m )PrY d 0,S m (9) 5
For an acual implemenaion he values of (m), (m) and (i,m ) may be represened as he negaive logarihm o he acual probabiliies. This is also common pracice for Vierbi decoders where he branch and sae merics are -log o he corresponding probabiliies. Wih he logarihmic presenaion muliplicaion becomes addiion and addiion becomes an E-operaion, where x E ylog(e x e y )min(x,y)log(e yx ) (20) This funcion can be reduced o finding he minimum and adding a small correcion facor. As seen from Formula 8 he incoming log-likelihood raio, can be used direcly in he calculaion of -log() as he log-likelihood raio of he a priori probabiliies. 6
7
4. Final Remarks This uorial was mean as a firs glimpse on he urbo codes and he ieraed decoding principle. Hopefully, we have shed some ligh on he opic, if here are sill some dark spos - ry reading i again! Of course here are a lo of deails no explained here, a lo of variaion o he urbo coding scheme and a lo of hings ha may need a proof. Some of hese can be found in he papers on he lieraure lis. 8
9
5. Seleced Lieraure [] Jakob Dahl Andersen, Turbo Codes Exended wih Ouer BCH Code, Elecronics Leers, vol. 32 No. 22, Oc. 996. [2] J. Dahl Andersen and V. V. Zyablov, Inerleaver Design for Turbo Coding, Proc. In. Symposium on Turbo Codes, Bres, Sep. 997. [3] Jakob Dahl Andersen, Selecion of Componen Codes for Turbo Coding based on Convergence Properies, Annales des Telecommunicaion, Special issue on ieraed decoding, June 999. [4] L. R. Bahl, J. Cocke, F. Jelinek and R. Raviv, Opimal Decoding of Linear Codes for Minimizing Symbol Error Rae, IEEE Trans. Inform. Theory, vol IT- 20, pp 284-287, March 974. [5] S. Benedeo and G. Monorsi, Performance Evaluaion of Turbo-codes, Elecronics Leers, vol. 3, No. 3, Feb. 995. [6] S. Benedeo and G. Monorsi, Serial Concaenaion of Block And Convoluional Codes, Elecronics Leers, Vol. 32, No. 0, May 996. [7] C. Berrou, A. Glavieux and P. Thiimajshima, Near Shannon Limi Error-correcing Coding and Decoding : Turbo-codes(), Proc. ICC 93, pp. 064-070, May 993. [8] C. Berrou and A. Glavieux, Near Opimum Error Correcing Coding and Decoding: Turbo Codes, IEEE rans. on Communicaions, Vol. 44, No. 0, Oc. 996. [9] R. J. McEliece, E. R. Rodemich and J.-F. Cheng, The Turbo Decision Algorihm, Presened a he 33rd Alleron Conference on Communicaion, Conrol and Compuing, Oc. 995. 20
[0] L. C. Perez, J. Seghers and D. J. Cosello, Jr, A Disance Specrum Inerpreaion of Turbo Codes, IEEE Trans. on Inform. Theory, Vol. 42, No. 6, Nov. 996. [] Seven S. Pierobon, Implemenaion and Performance of a Serial MAP Decoder for use in an Ieraive Turbo Decoder, Proc. In. Symposium on Informaion Theory, Whisler, Canada, Sep. 995. 2