Decrypted Secrets
Friedrich L. Bauer Decrypted Secrets Methods and Maxims of Cryptology Fourth, Revised and Extended Edition With 191 Figures, 29 Tables, and 16 Color Plates 123
Dr. rer. nat. Dr. ès sc. h.c. Dr. rer. nat. h.c. mult. Friedrich L. Bauer Professor Emeritus of Mathematics and Computer Science Munich Institute of Technology Department of Computer Science Boltzmannstr. 3 85748 Garching, Germany ACM Computing Classification (1998): E.3, D.4.6, K.6.5, E.4 Mathematics Subject Classification (1991): 94A60, 68P25 Library of Congress Control Number: 2006933429 ISBN-10 3-540-24502-2 Springer Berlin Heidelberg New York ISBN-13 978-3-540-24502-5 Springer Berlin Heidelberg New York ISBN 3-540-42674-4 3rd ed. Springer Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com Springer-Verlag Berlin Heidelberg 1997, 2000, 2002, 2007 The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover Design: Design & Concept E. Smejkal, Heidelberg Color Photos: Reinhard Krause, Deutsches Museum München Typesetting: By the author in TEX Production: LE-TEX, Jelonek, Schmidt & Vöckler GbR, Leipzig Printed on acid-free paper 33/3100 YL 5 4 3 2 1 0
Preface Towards the end of the 1960s, under the influence of the rapid development of microelectronics, electromechanical cryptological machines began to be replaced by electronic data encryption devices using large-scale integrated circuits. This promised more secure encryption at lower prices. Then, in 1976, Diffie and Hellman opened up the new cryptological field of public-key systems. Cryptography, hitherto cloaked in obscurity, was emerging into the public domain. Additionally, ENIGMA revelations awoke the public interest. Computer science was a flourishing new field, too, and computer scientists became interested in several aspects of cryptology. But many of them were not well enough informed about the centuries-long history of cryptology and the high level it had attained. I saw some people starting to reinvent the wheel, and others who had an incredibly naive belief in safe encryption, and I became worried about the commercial and scientific development of professional cryptology among computer scientists and about the unstable situation with respect to official security services. This prompted me to offer lectures on this subject at the Munich Institute of Technology. The first series of lectures in the winter term 1977/78, backed by the comprehensive and reliable book The Codebreakers (1967) by David Kahn, was held under the code name Special Problems of Information Theory and therefore attracted neither too many students nor too many suspicious people from outside the university. Next time, in the summer term of 1981, my lectures on the subject were announced under the open title Cryptology. This was seemingly the first publicly announced lecture series under this title at a German, if not indeed a Continental European, university. The series of lectures was repeated a few times, and in 1986/87 lecture notes were printed which finally developed into Part I of this book. Active interest on the side of the students led to a seminar on cryptanalytic methods in the summer term of 1988, from which Part II of the present book originated. The 1993 first edition (in German) of my book Kryptologie, although written mainly for computer science students, found lively interest also outside the field. It was reviewed favorably by some leading science journalists, and the publisher followed the study book edition with a 1995 hardcover edition under the title Entzifferte Geheimnisse [Decrypted Secrets], which gave me the opportunity to round out some subjects. Reviews in American journals recommended also an English version, which led in 1997 to the present book. It has become customary among cryptologists to explain how they became acquainted with the field. In my case, this was independent of the Second World War. In fact, I was never a member of any official service and I
VI Preface consider this my greatest advantage, since I am not bound by any pledge of secrecy. On the other hand, keeping eyes and ears open and reading between the lines, I learned a lot from conversations (where my scientific metier was a good starting point), although I never know exactly whether I am allowed to know what I happen to know. It all started in 1951, when I told my former professor of formal logic at Munich University, Wilhelm Britzelmayr, of my invention of an error-correcting code for teletype lines 1. This caused him to make a wrong association, and he gave me a copy of Sacco s book, which had just appeared 2. I was lucky, for it was the best book I could have encountered at that time although I didn t know that then. I devoured the book. Noticing this, my dear friend and colleague Paul August Mann, who was aware of my acquaintance with Shannon s redundancy-decreasing encoding, gave me a copy of the now-famous paper by Luigi Sacco (1883 1970) Claude Shannon called Communication Theory of Secrecy Systems 3 (which in those days as a Bell Systems Technical Report was almost unavailable in Germany). I was fascinated by this background to Shannon s information theory, which I was already familiar with. This imprinted my interest in cryptology as a subfield of coding theory and formal languages theory, fields that held my academic interest for many years to come. Strange accidents or maybe sharper observation then brought me into contact with more and more people once close to cryptology, starting with Willi Jensen (Flensburg) in 1955, Karl Stein (Munich) in 1955, Hans Rohrbach, my colleague at Mainz University, in 1959, as well as Helmut Grunsky, Gisbert Hasenjäger, and Ernst Witt. In 1957, I became acquainted with Erich Hüttenhain (Bad Godesberg), but our discussions on the suitability of certain computers for cryptological work were in the circumstances limited by certain restrictions. Among the American and British colleagues in numerical analysis and computer science I had closer contact with, some had been involved with cryptology in the Second World War; but no one spoke about that, particularly not before 1974, the year when Winterbotham s book The Ultra Secret appeared. In 1976, I heard B. Randall and I. J. Good reveal some details about the Colossi in a symposium in Los Alamos. As a scienceoriented civilian member of the cryptology academia, my interest in cryptology was then and still is centered on computerized cryptanalysis. Other aspects of signals intelligence ( SIGINT ), for example, traffic analysis and direction finding, are beyond the scope of this book; the same holds for physical devices that screen electromechanical radiation emitted by cipher machines. 1 DBP No. 892767, application date January 21, 1951. 2 Général Luigi Sacco, Manuel de Cryptographie. Payot, Paris 1951. 3 Bell Systems Technical Journal 28, Oct. 1949, pp. 656 715.
Preface VII Cryptology is a discipline with an international touch and a particular terminology. It may therefore be helpful sometimes to give in this book some explanations of terms that originated in a language other than English. The first part of this book presents cryptographic methods. The second part covers cryptanalysis, above all the facts that are important for judging cryptographic methods and for saving the user from unexpected pitfalls. This follows from Kerckhoffs maxim: Only a cryptanalyst can judge the security of a cryptosystem. A theoretical course on cryptographic methods alone seems to me to be bloodless. But a course on cryptanalysis is problematic: Either it is not conclusive enough, in which case it is useless, or it is conclusive, but touches a sensitive area. There is little clearance in between. I have tried to cover at least all the essential facts that are in the open literature or can be deduced from it. No censorship took place. Certain difficulties are caused by the fact that governmental restrictions during and after World War II, such as the need to know rule and other gimmicks, misled even people who had been close to the centers of cryptanalysis. Examples include the concept of Banburismus and the concept of a cilli. The word Banburismus the name was coined in Britain was mentioned in 1985 by Deavours and Kruh in their book, but the method was only vaguely described. Likewise, the description Kahn gave in 1991 in his book is rather incomplete. On the other hand, in Kozaczuk s book of 1979 (English edition of 1984), Rejewski gave a description of Różycki s clock method, which turned out to be the same but most of the readers could not know of this connection. Then, in 1993, while giving a few more details on the method, Good (in Codebreakers ) confirmed that Banburism was an elaboration of... the clock method... [of]... Różycki. He also wrote that this elaboration was invented at least mainly by Turing, and referred to a sequential Bayesian process as the method of scoring. For lack of declassified concrete examples, the exposition in Sect. 19.4.2 of the present book, based on the recently published postwar notes of Alexander and of Mahon and articles by Erskine and by Noskwith in the recent book Action This Day, cannot yet be a fully satisfactory one. And as to cillies, even Gordon Welchman admitted that he had misinterpreted the origin of the word, thinking of silly. Other publications gave other speculations, see Sect. 19.7, fn. 29. Ralph Erskine, in Action This Day, based on the recently declassified Cryptanalytic Report on the Yellow Machine, 71-4 (NACP HCC Box 1009, Nr. 3175), gives the following summary of the method: Discovered by Dilly Knox in late January 1940, cillies reduced enormously the work involved in using the Zygalski sheets, and after 1 May, when the Zygalski sheets became useless, they became a vital part of breaking Enigma by hand during most of 1940. They were still valuable in 1943. Cillies resulted from a combination of two different mistakes in a multi-part message by some Enigma operators. The first was their practice of leaving the rotors untouched when they reached the end of some part of the message.
Since the letter count of each message part was included in the preamble, the message key of the preceding part could be calculated within fine limits. The second error was the use of non-random message keys stereotyped keyboard touches and 3-letter-acronyms. In combination, and in conjunction with the different turnover points of rotors I to V, they allowed one to determine which rotors could, and which could not, be in any given position in the machine. Although Banburismus and cillies were highly important in the war, it is hard to understand why Derek Taunt in 1993 was prevented by the British censor from telling the true story about cillies. Possibly, the same happened to Jack Good about Banburismus. *** My intellectual delight in cryptology found an application in the collection Informatik of the Deutsches Museum in Munich which I built up in 1984 1988, where there is a section on cryptological devices and machines. My thanks go to the Deutsches Museum for providing color plates of some of the pieces on exhibit there. And thanks go to my former students and co-workers in Munich, Manfred Broy, Herbert Ehler, and Anton Gerold for continuing support over the years, moreover to Hugh Casement for linguistic titbits, and to my late brotherin-law Alston S. Householder for enlightenment on my English. Karl Stein and Otto Leiberich gave me details on the ENIGMA story, and I had fruitful discussions and exchanges of letters with Ralph Erskine, Heinz Ulbricht, Tony Sale, Frode Weierud, Kjell-Ove Widman, Otto J. Horak, Gilbert Bloch, Arne Fransén, and Fritz-Rudolf Güntsch. Great help was given to me by Kirk H. Kirchhofer from Crypto AG, Zug (Switzerland). Hildegard Bauer-Vogg supplied translations of difficult Latin texts, Martin Bauer, Ulrich Bauer and Bernhard Bauer made calculations and drawings. Thanks go to all of them. The English version was greatly improved by J. Andrew Ross, with whom working was a pleasure. In particular, my sincere thanks go to David Kahn who encouraged me ( The book is an excellent one and deserves the widest circulation ) and made quite a number of proposals for improvements of the text. For the present edition, additional material that has been made public recently has been included, among others on Bletchley Park, the British attack on Tunny, Colossus and Max Newman s pioneering work. Moreover, my particular thanks go to Ralph Erskine who indefatigably provided me with a lot of additional information and checked some of the dates and wordings. In this respect, my thanks also go to Jack Copeland, Heinz Ulbricht, and Augusto Buonafalce. Finally, I have to thank once more Hans Wössner for a well functioning cooperation of long standing, and the new copy editor Ronan Nugent for very careful work. The publisher is to be thanked for the fine presentation of the book. And I shall be grateful to readers who are kind enough to let me know of errors and omissions. Grafrath, Spring 2006 F. L. Bauer
Contents Part I: Cryptography The People... 1 1 Introductory Synopsis...9 1.1 Cryptography and Steganography...9 1.2 Semagrams... 10 1.3 Open Code: Masking... 13 1.4 Cues... 17 1.5 Open Code: Veiling by Nulls... 19 1.6 Open Code: Veiling by Grilles... 23 1.7 Classification of Cryptographic Methods... 24 2 Aims and Methods of Cryptography... 26 2.1 The Nature of Cryptography... 26 2.2 Encryption... 32 2.3 Cryptosystems... 34 2.4 Polyphony... 36 2.5 Character Sets... 39 2.6 Keys... 41 3 Encryption Steps: Simple Substitution... 44 3.1 Case V (1) W (Unipartite Simple Substitutions)... 44 3.2 Special Case V V (Permutations)... 46 3.3 Case V (1) W m (Multipartite Simple Substitutions)... 53 3.4 The General Case V (1) W (m), Straddling... 55 4 Encryption Steps: Polygraphic Substitution and Coding. 58 4.1 Case V 2 W (m) (Digraphic Substitutions)... 58 4.2 Special Cases of Playfair and Delastelle: Tomographic Methods... 64 4.3 Case V 3 W (m) (Trigraphic Substitutions)... 68 4.4 The General Case V (n) W (m) : Codes... 68 5 Encryption Steps: Linear Substitution...80 5.1 Self-reciprocal Linear Substitutions... 82 5.2 Homogeneous Linear Substitutions... 82 5.3 Binary Linear Substitutions... 86 5.4 General Linear Substitutions... 86 5.5 Decomposed Linear Substitutions... 87
X Contents 5.6 Decimated Alphabets... 90 5.7 Linear Substitutions with Decimal and Binary Numbers... 91 6 Encryption Steps: Transposition... 93 6.1 Simplest Methods... 93 6.2 Columnar Transpositions... 98 6.3 Anagrams... 102 7 Polyalphabetic Encryption: Families of Alphabets...106 7.1 Iterated Substitutions...106 7.2 Cyclically Shifted and Rotated Alphabets...107 7.3 Rotor Crypto Machines...110 7.4 Shifted Standard Alphabets: Vigenère and Beaufort...127 7.5 Unrelated Alphabets...131 8 Polyalphabetic Encryption: Keys...139 8.1 Early Methods with Periodic Keys...139 8.2 Double Key... 141 8.3 Vernam Encryption...142 8.4 Quasi-nonperiodic Keys...144 8.5 Machines that Generate Their Own Key Sequences...145 8.6 Off-Line Forming of Key Sequences...156 8.7 Nonperiodic Keys...158 8.8 Individual, One-Time Keys...161 8.9 Key Negotiation and Key Management...165 9 Composition of Classes of Methods...169 9.1 Group Property...169 9.2 Superencryption...171 9.3 Similarity of Encryption Methods...173 9.4 Shannon s Pastry Dough Mixing...174 9.5 Confusion and Diffusion by Arithmetical Operations...180 9.6 DES and IDEA R...184 10 Open Encryption Key Systems...193 10.1 Symmetric and Asymmetric Encryption Methods...194 10.2 One-Way Functions...196 10.3 RSA Method...203 10.4 Cryptanalytic Attack upon RSA...205 10.5 Secrecy Versus Authentication...208 10.6 Security of Public Key Systems...210 11 Encryption Security...211 11.1 Cryptographic Faults...211 11.2 Maxims of Cryptology...220 11.3 Shannon s Yardsticks...225 11.4 Cryptology and Human Rights...226
Contents XI Part II: Cryptanalysis The Machinery...233 12 Exhausting Combinatorial Complexity...237 12.1 Monoalphabetic Simple Encryptions...238 12.2 Monoalphabetic Polygraphic Encryptions...239 12.3 Polyalphabetic Encryptions...241 12.4 General Remarks on Combinatorial Complexity...244 12.5 Cryptanalysis by Exhaustion...244 12.6 Unicity Distance...246 12.7 Practical Execution of Exhaustion...248 12.8 Mechanizing the Exhaustion...251 13 Anatomy of Language: Patterns...252 13.1 Invariance of Repetition Patterns...252 13.2 Exclusion of Encryption Methods...254 13.3 Pattern Finding...255 13.4 Finding of Polygraphic Patterns... 259 13.5 The Method of the Probable Word...259 13.6 Automatic Exhaustion of the Instantiations of a Pattern... 264 13.7 Pangrams...266 14 Polyalphabetic Case: Probable Words...268 14.1 Non-Coincidence Exhaustion of Probable Word Position...268 14.2 Binary Non-Coincidence Exhaustion...271 14.3 The De Viaris Attack...272 14.4 Zig-Zag Exhaustion of Probable Word Position...280 14.5 The Method of Isomorphs... 281 14.6 A clever brute force method: EINSing...287 14.7 Covert Plaintext-Cryptotext Compromise...288 15 Anatomy of Language: Frequencies... 290 15.1 Exclusion of Encryption Methods...290 15.2 Invariance of Partitions...291 15.3 Intuitive Method: Frequency Profile...293 15.4 Frequency Ordering...294 15.5 Cliques and Matching of Partitions...297 15.6 Optimal Matching...303 15.7 Frequency of Multigrams... 305 15.8 The Combined Method of Frequency Matching...310 15.9 Frequency Matching for Polygraphic Substitutions... 316 15.10 Free-Style Methods...317 15.11 Unicity Distance Revisited...318 16 Kappa and Chi... 320 16.1 Definition and Invariance of Kappa... 320 16.2 Definition and Invariance of Chi... 323 16.3 The Kappa-Chi Theorem... 325 16.4 The Kappa-Phi Theorem... 326 16.5 Symmetric Functions of Character Frequencies... 328
XII Contents 17 Periodicity Examination... 330 17.1 The Kappa Test of Friedman... 331 17.2 Kappa Test for Multigrams... 332 17.3 Cryptanalysis by Machines: Searching for a period... 333 17.4 Kasiski Examination... 339 17.5 Building a Depth and Phi Test of Kullback... 345 17.6 Estimating the Period Length... 348 18 Alignment of Accompanying Alphabets... 350 18.1 Matching the Profile... 350 18.2 Aligning Against Known Alphabet... 354 18.3 Chi Test: Mutual Alignment of Accompanying Alphabets... 358 18.4 Reconstruction of the Primary Alphabet... 363 18.5 Kerckhoffs Symmetry of Position... 365 18.6 Stripping off Superencryption: Difference Method... 370 18.7 Decryption of Code... 373 18.8 Reconstruction of the Password... 373 19 Compromises... 375 19.1 Kerckhoffs Superimposition... 375 19.2 Superimposition for Encryptions with a Key Group... 377 19.3 COLOSSUS... 401 19.4 Adjustment in depth of Messages... 412 19.5 Cryptotext-Cryptotext Compromises... 419 19.6 Cryptotext-Cryptotext Compromise: ENIGMA Indicator Doubling 431 19.7 Plaintext-Cryptotext Compromise: Feedback Cycle... 448 20 Linear Basis Analysis... 459 20.1 Reduction of Linear Polygraphic Substitutions... 459 20.2 Reconstruction of the Key... 460 20.3 Reconstruction of a Linear Shift Register... 461 21 Anagramming... 464 21.1 Transposition... 464 21.2 Double Columnar Transposition... 467 21.3 Multiple Anagramming... 467 22 Concluding Remarks... 470 22.1 Success in Breaking... 471 22.2 Mode of Operation of the Unauthorized Decryptor... 476 22.3 Illusory Security... 482 22.4 Importance of Cryptology... 484 Appendix: Axiomatic Information Theory... 487 Bibliography... 497 Index... 501 Photo Credits... 525
List of Color Plates 4 Plate A Plate B Plate C The disk of Phaistos Brass cipher disks The Cryptograph of Wheatstone Plate D The US Army cylinder device M-94 Plate E Plate F The US strip device M-138-T4 The cipher machine of Kryha Plate G The Hagelin Cryptographer C-36 Plate H Plate I Plate K Plate L Plate M The US Army M-209, Hagelin licensed The cipher machine ENIGMA with four rotors Rotors of the ENIGMA The British rotor machine TYPEX Uhr box of the German Wehrmacht Plate N Cipher teletype machine Lorenz SZ 42 Plate O Plate P Plate Q Russian one-time pad Modern crypto board CRAY Supercomputers 4 In the middle of the book, following page 232.
Leone Battista Alberti (1404 1472) Father of Western Cryptology (David Kahn)