CODING FOR CHANNELS WITH FEEDBACK

CODING FOR CHANNELS WITH FEEDBACK

THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE

CODING FOR CHANNELS WITH FEEDBACK by JamesM.Ooi The Cambridge Analytic Group SPRINGER SCIENCE+BUSINESS MEDIA, LLC

ISBN 978-1-4613-7619-4 DOI 10.1007/978-1-4615-5719-7 ISBN 978-1-4615-5719-7 (ebook) Library of Congress Cataloging-in-Publication Data A C.I.P. Catalogue record for this book is available from the Library of Congress. Copyright 1998 by Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 1998 Softcover reprint ofthe hardcover Ist edition 1998 AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC Printed on acid-free paper.

DEDICATED TO MY FATHER: BOON SENG 001 APRIL 21,1940 - OCTOBER 29, 1997

Contents Preface Acknowledgments Notation and Abbreviations 1 Introduction 1.1 Feedback in Communication. 1.2 Overview. References............. xiii xvii xix 1 3 5 7 2 DMCs: An Introduction to the Framework 9 2.1 Introduction.............. 9 2.2 Variable-Length Codes and Their Properties: Rate, Reliability, and Complexity................ 15 2.3 A Feedback Coding Scheme for DMCf's. 18 2.3.1 Encoding. 19 2.3.2 Decoding... 25 2.4 Reliability................ 26 2.5 Removing the Perfect-Detection Assumption 30 2.5.1 Reliability of the Modified Scheme. 31 2.6 Complexity... 33 2.6.1 Time and Space Complexity for the Transmitter and Receiver. 33 2.6.2 Uniform Versus Non-Uniform Complexity......... 35 2.7 Variations... 36 2.7.1 Scheme Using a Fixed-Length Precoder and Source Coder 36 2.8 Discussion... 38 2.8.1 Structuring Computation...... 38 2.8.2 Variability of Transmission Length. 2.8.3 Number of Iterations..... 2.8.4 Feedback Delay....... 2.8.5 Inaccurate Channel Modeling 2.8.6 Simulation... 2.8.7 Summary and Future Directions Appendix to Chapter 2....... 2.A Proof of Subsystem Property 1..... 40 40 40 40 41 42 43 43

x CONTENTS 2.B Proof of Subsystem Property 2 2.C Proof of Subsystem Property 3 2.D Proof of Lemma 2.4.1... 2.E Proof of Lemma 2.4.2..... 2.F Proof of Subsystem Property 4 2.G Modified Scheme Reliability.. 2.H Precoding with Linear Complexity. 2.1 Fixed-Length Precoding and Source Coding Subsystems. 2.1.1 Fixed-Length Precoder...... 2.1.2 Fixed-Length Source Coder... 2.1 Analysis of Variable-Length Interleaving. References................... 46 50 51 52 52 53 55 55 55 57 57 60 3 Channels with Memory 61 3.1 Introduction............. 61 3.2 Generalization of the Coding Scheme 63 3.3 A Coding Scheme for DFSCf's. 65 3.3.1 Precoder... 68 3.3.2 Source Coder... 69 3.3.3 Synchronization Sequence and Detector 70 3.3.4 Termination Coder-Modified Schalkwijk-Barron Coder 70 3.4 Complexity... 70 3.5 Discussion... 72 3.5.1 Choosing the Synchronization and msb Sequences 72 3.5.2 Beyond Feedback-Free Capacity....... 72 3.5.3 Computational Delays............ 74 3.5.4 Fixed-Length Precoding and Source Coding. 75 3.5.5 Summary and Future Directions....... 75 Appendix to Chapter 3................... 76 3.A Ergodic Finite-Order Markov Input to Indecomposable DFSC f 's 76 3.B Efficiency Property of the Precoder... 77 3.C Interaction between Precoder and Source Coder......... 80 3.D A Fast Algorithm for Conditional Shannon-Fano Source Coding. 85 3.E Synchronization-Sequence Detection. 87 3.F Lemmas Relating to Entropy Rates. 94 References................ 97 4 Unknown Channels 99 4.1 Introduction.. 99 4.2 Formulation of Variable-Length Coding for Unknown Channels. 100 4.3 A Universal Communication Scheme for a Class ofufsc's. 101 4.3.1 Precoder and Source Coder 103 4.3.2 Synchronization Subsystem............. 105

CONTENTS xi 4.3.3 Termination Codes. 106 4.4 Complexity... 106 4.5 Uniform Convergence... 107 4.6 Discussion... 109 4.6.1 Choice of input distribution 109 4.6.2 Computational Delays and Maximum Quality 109 4.6.3 Summary and Future Directions..... 110 Appendix to Chapter 4................... III 4.A Properties of the Conditional Lempel-Ziv Coder... III 4.B Universal Detection of the Synchronization Sequence 115 References......................... 120 5 Multiple-Access Channels 121 5.1 Introduction............................. 121 5.2 Formulation of Variable-Length Coding for Two-User Multiple Access Channels............................ 125 5.3 Low-Complexity Coding Scheme for Two-User Multiple-Access Channels....... 126 5.3.1 Encoders... 128 5.4 Performance Analysis 132 5.4.1 Reliability. 132 5.4.2 Complexity. 135 5.5 Discussion... 135 5.5.1 Achieving Other Rate Pairs 135 5.5.2 Variations......... 135 5.5.3 Beyond the Feedback-Free Capacity Region. 136 5.5.4 Summary and Future Directions 136 Appendix to Chapter 5........... 138 5.A Fixed-Length Source Coder for Tx-2. 138 References................. 139 6 Channels with Partial and Noisy Feedback 141 6.1 Introduction..................... 141 6.2 Feedback Rate Reduction... 142 6.2.1 An Approach Based on Slepian-Wolf Coding 142 6.2.2 A Reduced-Feedback-Rate Coding Scheme for BSC's. 144 6.2.3 Performance Analysis of the Modified Scheme... 147 6.2.4 Comments on Slepian-Wolf Coded Feedback for General DMC's 152 6.3 A Concatenated Coding Framework for Combining FEC Codes with Feedback Codes.................... 153 6.3.1 Main Idea................... 153 6.3.2 Superchannels Induced by Strong Inner Codes. 154 6.3.3 Superchannels Induced by Weak Inner Codes. 156

xii CONTENTS 6.3.4 Computational Comparison with Feedback-Free Linear Complexity Concatenated Coding Scheme.... 159 6.4 Feedback Coding on the Erasure Channel and ARQ. 160 6.5 Coping with Noisy Feedback............. 162 6.6 Discussion...................... 164 6.6.1 Multiple-Access Channels with Partial Feedback. 164 6.6.2 An Alternative View of Coding with Partial Feedback.. 165 6.6.3 Summary and Future Directions. 166 References............................... 168 7 Conclusions 169

Preface Since the introduction of the coding theorem in 1948, the main obstacle to realizing the performance promised by this theorem has been computational complexity. While researchers realized in the 50's and 60's that the presence of feedback on a communication link enables one to construct low-complexity coding schemes, the flurry of excitement over the first low-complexity capacity-achieving feedback coding schemes subsided over the next two decades. But there has been a recent resurgence of interest in feedback communication systems as the intense efforts in feedback-free communication for even the simplest channel models have been only partly successful. This monograph, which is based on my doctoral research at the Massachusetts Institute of Technology, presents both algorithms for feedback coding and performance analyses of these algorithms, including analyses of perhaps the most important performance criterion: computational complexity. The algorithms are developed within a single framework, which I term the compressed-error-cancellation framework. In this framework, data are sent via a sequence of messages: the first message contains the original data; each subsequent message contains a source-coded description of the channel distortions introduced on the message preceding it. The ideas behind this extraordinarily simple and elegant framework first appeared 25 years ago as a tool for proving theoretical results about feedback communication systems. Yet, despite the usual emphasis of information theory on elegant theoretical results, a substantial fraction of practicing information theorists are unaware of these ideas. By exhibiting the usefulness and flexibility of this framework for deriving low-complexity solutions to a wide variety of feedback communication problems, I hope awareness of these ideas will increase. Indeed, one of the research contributions of this monograph is to show that the compressed-error-cancellation framework leads to practical solutions and is applicable to a wide variety of problems. It was previously thought that the compressed-errorcancellation framework led only to coding schemes with high computational complexity; furthermore, the framework had only been applied to discrete memoryless channels with feedback and certain arbitrarily varying channels with feedback. I show in this monograph that the framework leads to coding schemes with the lowest possible asymptotic order of growth of computations and can be applied to channels with memory, unknown channnels, multiple-access channels, all with complete noiseless feedback, as well as to channels with partial and noisy feedback. I believe that practitioners, researchers, and students alike can appreciate this monograph, which requires only working knowledge of information theory and probability theory. For practitioners, high-performance algorithms are explicitly developed

xiv Preface for some of the most challenging communication problems. For researchers, new insights are presented into challenging research questions on feedback communication, and many directions for future research are suggested. For the student, a rather different perspective on information theory is provided that ties source coding and channel coding explicitly together. The monograph is organized into seven chapters. The first and last chapters provide introductory and summary material, respectively. The middle chapters are divided by the type of communication scenario to be addressed within our unified framework, ranging from the simplest scenario to more complex. Specifically, Chapter 2 provides an introduction to the basic framework in the context of applying the basic framework to developing low-complexity coding schemes for discrete memory less channels with complete, noiseless feedback. The coding problem with feedback is formulated, and a coding scheme exploiting low-complexity lossless source coding algorithms is developed. Notions of computational complexity are discussed, and the associated encoder and decoder are shown to use a number of computations growing only linearly with the number of channel inputs used (linear complexity). The associated error exponent is shown to be optimal in an appropriate sense and implies that capacity is achievable. Simulation results are reported to confirm the analytically predicted behavior. In Chapter 3, channels with memory and noiseless feedback are considered. Channels with memory are becoming increasingly important models for contemporary communication systems, especially wireless systems. In particular, for the sake of mathematical tractability, I focus on the class of channels with memory known as discrete finitestate channels with complete, noiseless feedback, which form an extremely flexible class of channels with memory. The framework is used to develop linear-complexity coding schemes performing analogously in terms of rate and reliability to the schemes developed for discrete memory less channels. Chapter 4 explores the use of the framework for coding over unknown channels with feedback. Unknown channels model the scenario in which the channel varies with time slowly in an unknown way. The availability of feedback introduces the possibility of channel-dependent communication performance. In particular, this chapter introduces the notion of universal communication, whereby communication over an unknown channel with feedback takes place and results in an effective transmission rate that varies with the quality of the realized channel. A linear-complexity universal communication scheme is developed and analyzed within the framework. The asymptotic rate and reliability characteristics of this universal scheme are shown to be similar to those of the schemes developed for known channels. In Chapter 5, an extension of the compressed-error-cancellation framework is developed for discrete memory less multiple-access channels with complete, noiseless feedback, which are important models for contemporary communication networks. This extended framework leads to linear-complexity coding schemes achieving rates on the frontier of the feedback-free capacity region. In Chapter 6 the compressed-error-cancellation framework is applied to the prac-

Coding for Channels with Feedback xv tically important problem of coding for channels with noisy and partial feedback. The scheme developed for discrete memory less channels with feedback is modified to incorporate Slepian-Wolf coded feedback, resulting in a linear-complexity, capacityachieving coding scheme with partial, noiseless feedback. This modified scheme, of which the popular automatic-repeat-request protocol is shown to be a special case, is then used as an outer code in a concatenated coding arrangement; with a forward errorcorrecting (FEC) code used as the inner code, a framework emerges for integrating FEC coding with feedback coding, leading to a broad class of coding schemes using various amounts of noiseless feedback. Preliminary investigations on partial-feedback multiple-access scenarios and noisy feedback scenarios are also discussed.

Acknowledgments It is my privilege to mention here a number of individuals whose efforts made this monograph possible. My thesis supervisor at MIT, Professor Gregory Wornell, provided excellent technical and literary guidance that greatly improved the content and and presentation of my doctoral thesis upon which this monograph is based; I sincerely thank him for his efforts. The other members of my thesis committee, Professors Amos Lapidoth and Robert Gallager, inspired important developments in my doctoral thesis. My colleagues, Jeff Ludwig and Shawn Verbout, influenced my thinking immeasurably on a variety of subjects related to this monograph. Finally, I thank my partner in life, Amy Yin, for her love and support while completing my doctoral thesis and this monograph. I must also mention that this monograph could not have been completed without the resources of the Digital Signal Processing Group in the MIT Research Laboratory of Electronics and the generous financial support given by the Defense Advanced Research Projects Agency, the Office of Naval Research, AT&T, Lucent Technologies, and the MIT Electrical Engineering and Computer Science Department.

Notation and Abbreviations An important notational convention of which we must be aware at this point is that if X is a discrete random variable, then p x automatically denotes its probability mass function (pmt), and E[X] automatically denotes its expected value. Standard adjustments to this notation are used for conditional and joint pmfs and expectations. Remaining notational conventions and abbreviations are introduced as needed throughout the thesis (often in footnotes). For reference, we summarize here some important conventions and abbreviations, which should be assumed to hold unless otherwise stated Abbreviations: Bernoulli-t: ~ Bernoulli with probability of equaling one being t: BSC ~ binary symmetric channel BSCf ~ BSC with complete noiseless feedback BSCpf ~ BSC with partial noiseless feedback cdf ~ cumulative distribution function CSWCF ~ concatenated Slepian-Wolf coded feedback DFSC ~ discrete finite-state channel DFSCf ~ DFSC with complete noiseless feedback DFSCpf ~ DFSC with partial noiseless feedback DMC ~ discrete memoryless channel DMCf ~ DMC with complete noiseless feedback DMCpf ~ DMC with partial noiseless feedback DMMAC ~ discrete memoryless multiple-access channel DMMACf ~ DMMAC with complete noiseless feedback DMMACpf ~ DMMAC with partial noiseless feedback FEC ~ forward error-correcting i.i.d. ~ independent and identically distributed MABC ~ multiple-access broadcast channel pmf ~ probability mass function SWCF ~ Slepian-Wolf coded feedback UFSC ~ unknown DFSC UFSCf ~ UFSC with complete noiseless feedback UFSCpf ~ UFSC with partial noiseless feedback

xx Notation and Abbreviations Notational Conventions: o -t marks the end of a proof (0.1) 'il -t marks the end of a proof of a lemma introduced within a larger proof (0.2) x[ J -t square brackets have no general meaning; they are used as an alternative or in addition to super- and subscripts. (0.3) On(g(n)) -t a function in the set {f(n) limn... oo f(n)/g(n) = O} (0.4) On(g(n)) -t a function in the set {f(n) liminfn... oo f(n)/g(n) ~ and limsupn... oo f(n)/g(n) < co} (0.5) 6 n(g(n)) -t a function in the set {f(n) liminfn... oo f(n)/g(n) > and Z -t the set of integers lr -t the set of real numbers N -t the set of natural numbers ({O, 1,2,... }) IA I -t cardinality of the set A coa -t convex hull of the subset A of Euclidean space An -t n-fold Cartesian product of A with itself At -t the set of variable-length tuples with elements in A: At = U~=lAn a! -t (as,'..,at) a';' -t (as, ash,". ) an -t a~ f( a) -t length of variable-length tuple a: f(a)=nifaean i=l limsupn... oo f(n)/g(n) < co} (0.6) x[nj -t nth M-ary expansion digit of x E [0, IJ, where M is determined from (0.7) (0.8) (0.9) (0.10) (0.11) (0.12) (0.13) (0.14) (0.15) (0.16) (0.17) (0.18) context. When two expansions exist, the one ending with zeros is taken. (0.19) [tj ( ) x[.j -t x[.]>, x[tj x[tj -t x[tj [lj x t Y -t X converges to y from below r x 1 -t ceiling of x: rxl = min{z: z E Z,z ~ x} log x -t base-2 logarithm of x In x -t natural logarithm of x (0.20) (0.21) (0.22) (0.23) (0.24) (0.25)

Coding for Channels with Feedback xxi exp2 {X} -t 2 x exp{ x} -t ex (0.26) (0.27) Pr{ A} -t probability of an event A (0.28) Pr{AI23} -t probability of an event A conditioned on the event 23 (0.29) px -t probability mass function (pmf) for X: px(x) = Pr{X = x} (0.30) PXIY -t pmffor X conditioned on Y: PXly(xly) = Pr{X = xly = y} (0.31) E[X] -t expected value of the random variable X: E[X] = LXPx(x) (0.32) var(x) -t variance of the random variable X: var(x) = L(x - E[X])2pX(X) std(x) -t standard deviation of the random variable X: std(x) = vvar(x) H 2 ( f) -t binary entropy function: H2(f) = -dog f - (1 - f) log(l - f) H(X) -t entropy of random variable X: H(X) = - LPx(x) logpx(x) H (X, Y) -t joint entropy of random variables X and Y: H(X,Y) = - LPX,y(x,y}logpx,Y(x,y) X,Y H(XIY} -t conditional entropy of X given Y: H(XIY) = H(X, Y) - H(Y} J(Xj Y) -t mutual information between X and Y: J(Xj Y} = H(X) + H(Y} - H(X, Y) H 00 (Xoo) -t entropy rate of random process Xoo: Hoo(Xoo) = lim.!.h(xn) n-+oo n Hoo(X oo, y oo ) -t joint entropy rate of random processes X OO and y oo : Hoo (Xoo, y oo ) = lim.!.h(xn, yn) n-+oo n Hoo(XooIY oo ) -t conditional entropy rate XOO conditioned on y oo : Hoo(XooIY oo ) = Hoo(Xoo, y oo ) _ H(Yoo) D2 (f II a) -t Kullback-Leibler distance between two Bernoulli pmfs with parameters f and a: f 1-f D2(f II a) = dog - + (1- f}log -1- a -a (0.33) (0.34) (0.35) (0.36) (0.37) (0.38) (0.39) (0.40) (0.41) (0.42) (0.43)

xxii Notation and Abbreviations D(P II q) """* Kullback-Leibler distance between pmfs p and q: D(p II q) = l>(x) log :~:~., (0.44)