Decrypted Secrets. Friedrich L. Bauer. Methods and Maxims of Cryptology. Fourth, Revised and Extended Edition

Transcription

1 Decrypted Secrets

2 Friedrich L. Bauer Decrypted Secrets Methods and Maxims of Cryptology Fourth, Revised and Extended Edition With 191 Figures, 29 Tables, and 16 Color Plates 123

3 Dr. rer. nat. Dr. ès sc. h.c. Dr. rer. nat. h.c. mult. Friedrich L. Bauer Professor Emeritus of Mathematics and Computer Science Munich Institute of Technology Department of Computer Science Boltzmannstr Garching, Germany ACM Computing Classification (1998): E.3, D.4.6, K.6.5, E.4 Mathematics Subject Classification (1991): 94A60, 68P25 Library of Congress Control Number: ISBN Springer Berlin Heidelberg New York ISBN Springer Berlin Heidelberg New York ISBN rd ed. Springer Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com Springer-Verlag Berlin Heidelberg 1997, 2000, 2002, 2007 The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover Design: Design & Concept E. Smejkal, Heidelberg Color Photos: Reinhard Krause, Deutsches Museum München Typesetting: By the author in TEX Production: LE-TEX, Jelonek, Schmidt & Vöckler GbR, Leipzig Printed on acid-free paper 33/3100 YL

4 Preface Towards the end of the 1960s, under the influence of the rapid development of microelectronics, electromechanical cryptological machines began to be replaced by electronic data encryption devices using large-scale integrated circuits. This promised more secure encryption at lower prices. Then, in 1976, Diffie and Hellman opened up the new cryptological field of public-key systems. Cryptography, hitherto cloaked in obscurity, was emerging into the public domain. Additionally, ENIGMA revelations awoke the public interest. Computer science was a flourishing new field, too, and computer scientists became interested in several aspects of cryptology. But many of them were not well enough informed about the centuries-long history of cryptology and the high level it had attained. I saw some people starting to reinvent the wheel, and others who had an incredibly naive belief in safe encryption, and I became worried about the commercial and scientific development of professional cryptology among computer scientists and about the unstable situation with respect to official security services. This prompted me to offer lectures on this subject at the Munich Institute of Technology. The first series of lectures in the winter term 1977/78, backed by the comprehensive and reliable book The Codebreakers (1967) by David Kahn, was held under the code name Special Problems of Information Theory and therefore attracted neither too many students nor too many suspicious people from outside the university. Next time, in the summer term of 1981, my lectures on the subject were announced under the open title Cryptology. This was seemingly the first publicly announced lecture series under this title at a German, if not indeed a Continental European, university. The series of lectures was repeated a few times, and in 1986/87 lecture notes were printed which finally developed into Part I of this book. Active interest on the side of the students led to a seminar on cryptanalytic methods in the summer term of 1988, from which Part II of the present book originated. The 1993 first edition (in German) of my book Kryptologie, although written mainly for computer science students, found lively interest also outside the field. It was reviewed favorably by some leading science journalists, and the publisher followed the study book edition with a 1995 hardcover edition under the title Entzifferte Geheimnisse [Decrypted Secrets], which gave me the opportunity to round out some subjects. Reviews in American journals recommended also an English version, which led in 1997 to the present book. It has become customary among cryptologists to explain how they became acquainted with the field. In my case, this was independent of the Second World War. In fact, I was never a member of any official service and I

5 VI Preface consider this my greatest advantage, since I am not bound by any pledge of secrecy. On the other hand, keeping eyes and ears open and reading between the lines, I learned a lot from conversations (where my scientific metier was a good starting point), although I never know exactly whether I am allowed to know what I happen to know. It all started in 1951, when I told my former professor of formal logic at Munich University, Wilhelm Britzelmayr, of my invention of an error-correcting code for teletype lines 1. This caused him to make a wrong association, and he gave me a copy of Sacco s book, which had just appeared 2. I was lucky, for it was the best book I could have encountered at that time although I didn t know that then. I devoured the book. Noticing this, my dear friend and colleague Paul August Mann, who was aware of my acquaintance with Shannon s redundancy-decreasing encoding, gave me a copy of the now-famous paper by Luigi Sacco ( ) Claude Shannon called Communication Theory of Secrecy Systems 3 (which in those days as a Bell Systems Technical Report was almost unavailable in Germany). I was fascinated by this background to Shannon s information theory, which I was already familiar with. This imprinted my interest in cryptology as a subfield of coding theory and formal languages theory, fields that held my academic interest for many years to come. Strange accidents or maybe sharper observation then brought me into contact with more and more people once close to cryptology, starting with Willi Jensen (Flensburg) in 1955, Karl Stein (Munich) in 1955, Hans Rohrbach, my colleague at Mainz University, in 1959, as well as Helmut Grunsky, Gisbert Hasenjäger, and Ernst Witt. In 1957, I became acquainted with Erich Hüttenhain (Bad Godesberg), but our discussions on the suitability of certain computers for cryptological work were in the circumstances limited by certain restrictions. Among the American and British colleagues in numerical analysis and computer science I had closer contact with, some had been involved with cryptology in the Second World War; but no one spoke about that, particularly not before 1974, the year when Winterbotham s book The Ultra Secret appeared. In 1976, I heard B. Randall and I. J. Good reveal some details about the Colossi in a symposium in Los Alamos. As a scienceoriented civilian member of the cryptology academia, my interest in cryptology was then and still is centered on computerized cryptanalysis. Other aspects of signals intelligence ( SIGINT ), for example, traffic analysis and direction finding, are beyond the scope of this book; the same holds for physical devices that screen electromechanical radiation emitted by cipher machines. 1 DBP No , application date January 21, Général Luigi Sacco, Manuel de Cryptographie. Payot, Paris Bell Systems Technical Journal 28, Oct. 1949, pp

6 Preface VII Cryptology is a discipline with an international touch and a particular terminology. It may therefore be helpful sometimes to give in this book some explanations of terms that originated in a language other than English. The first part of this book presents cryptographic methods. The second part covers cryptanalysis, above all the facts that are important for judging cryptographic methods and for saving the user from unexpected pitfalls. This follows from Kerckhoffs maxim: Only a cryptanalyst can judge the security of a cryptosystem. A theoretical course on cryptographic methods alone seems to me to be bloodless. But a course on cryptanalysis is problematic: Either it is not conclusive enough, in which case it is useless, or it is conclusive, but touches a sensitive area. There is little clearance in between. I have tried to cover at least all the essential facts that are in the open literature or can be deduced from it. No censorship took place. Certain difficulties are caused by the fact that governmental restrictions during and after World War II, such as the need to know rule and other gimmicks, misled even people who had been close to the centers of cryptanalysis. Examples include the concept of Banburismus and the concept of a cilli. The word Banburismus the name was coined in Britain was mentioned in 1985 by Deavours and Kruh in their book, but the method was only vaguely described. Likewise, the description Kahn gave in 1991 in his book is rather incomplete. On the other hand, in Kozaczuk s book of 1979 (English edition of 1984), Rejewski gave a description of Różycki s clock method, which turned out to be the same but most of the readers could not know of this connection. Then, in 1993, while giving a few more details on the method, Good (in Codebreakers ) confirmed that Banburism was an elaboration of... the clock method... [of]... Różycki. He also wrote that this elaboration was invented at least mainly by Turing, and referred to a sequential Bayesian process as the method of scoring. For lack of declassified concrete examples, the exposition in Sect of the present book, based on the recently published postwar notes of Alexander and of Mahon and articles by Erskine and by Noskwith in the recent book Action This Day, cannot yet be a fully satisfactory one. And as to cillies, even Gordon Welchman admitted that he had misinterpreted the origin of the word, thinking of silly. Other publications gave other speculations, see Sect. 19.7, fn. 29. Ralph Erskine, in Action This Day, based on the recently declassified Cryptanalytic Report on the Yellow Machine, 71-4 (NACP HCC Box 1009, Nr. 3175), gives the following summary of the method: Discovered by Dilly Knox in late January 1940, cillies reduced enormously the work involved in using the Zygalski sheets, and after 1 May, when the Zygalski sheets became useless, they became a vital part of breaking Enigma by hand during most of They were still valuable in Cillies resulted from a combination of two different mistakes in a multi-part message by some Enigma operators. The first was their practice of leaving the rotors untouched when they reached the end of some part of the message.

7 Since the letter count of each message part was included in the preamble, the message key of the preceding part could be calculated within fine limits. The second error was the use of non-random message keys stereotyped keyboard touches and 3-letter-acronyms. In combination, and in conjunction with the different turnover points of rotors I to V, they allowed one to determine which rotors could, and which could not, be in any given position in the machine. Although Banburismus and cillies were highly important in the war, it is hard to understand why Derek Taunt in 1993 was prevented by the British censor from telling the true story about cillies. Possibly, the same happened to Jack Good about Banburismus. *** My intellectual delight in cryptology found an application in the collection Informatik of the Deutsches Museum in Munich which I built up in , where there is a section on cryptological devices and machines. My thanks go to the Deutsches Museum for providing color plates of some of the pieces on exhibit there. And thanks go to my former students and co-workers in Munich, Manfred Broy, Herbert Ehler, and Anton Gerold for continuing support over the years, moreover to Hugh Casement for linguistic titbits, and to my late brotherin-law Alston S. Householder for enlightenment on my English. Karl Stein and Otto Leiberich gave me details on the ENIGMA story, and I had fruitful discussions and exchanges of letters with Ralph Erskine, Heinz Ulbricht, Tony Sale, Frode Weierud, Kjell-Ove Widman, Otto J. Horak, Gilbert Bloch, Arne Fransén, and Fritz-Rudolf Güntsch. Great help was given to me by Kirk H. Kirchhofer from Crypto AG, Zug (Switzerland). Hildegard Bauer-Vogg supplied translations of difficult Latin texts, Martin Bauer, Ulrich Bauer and Bernhard Bauer made calculations and drawings. Thanks go to all of them. The English version was greatly improved by J. Andrew Ross, with whom working was a pleasure. In particular, my sincere thanks go to David Kahn who encouraged me ( The book is an excellent one and deserves the widest circulation ) and made quite a number of proposals for improvements of the text. For the present edition, additional material that has been made public recently has been included, among others on Bletchley Park, the British attack on Tunny, Colossus and Max Newman s pioneering work. Moreover, my particular thanks go to Ralph Erskine who indefatigably provided me with a lot of additional information and checked some of the dates and wordings. In this respect, my thanks also go to Jack Copeland, Heinz Ulbricht, and Augusto Buonafalce. Finally, I have to thank once more Hans Wössner for a well functioning cooperation of long standing, and the new copy editor Ronan Nugent for very careful work. The publisher is to be thanked for the fine presentation of the book. And I shall be grateful to readers who are kind enough to let me know of errors and omissions. Grafrath, Spring 2006 F. L. Bauer

8 Contents Part I: Cryptography The People Introductory Synopsis Cryptography and Steganography Semagrams Open Code: Masking Cues Open Code: Veiling by Nulls Open Code: Veiling by Grilles Classification of Cryptographic Methods Aims and Methods of Cryptography The Nature of Cryptography Encryption Cryptosystems Polyphony Character Sets Keys Encryption Steps: Simple Substitution Case V (1) W (Unipartite Simple Substitutions) Special Case V V (Permutations) Case V (1) W m (Multipartite Simple Substitutions) The General Case V (1) W (m), Straddling Encryption Steps: Polygraphic Substitution and Coding Case V 2 W (m) (Digraphic Substitutions) Special Cases of Playfair and Delastelle: Tomographic Methods Case V 3 W (m) (Trigraphic Substitutions) The General Case V (n) W (m) : Codes Encryption Steps: Linear Substitution Self-reciprocal Linear Substitutions Homogeneous Linear Substitutions Binary Linear Substitutions General Linear Substitutions Decomposed Linear Substitutions... 87

9 X Contents 5.6 Decimated Alphabets Linear Substitutions with Decimal and Binary Numbers Encryption Steps: Transposition Simplest Methods Columnar Transpositions Anagrams Polyalphabetic Encryption: Families of Alphabets Iterated Substitutions Cyclically Shifted and Rotated Alphabets Rotor Crypto Machines Shifted Standard Alphabets: Vigenère and Beaufort Unrelated Alphabets Polyalphabetic Encryption: Keys Early Methods with Periodic Keys Double Key Vernam Encryption Quasi-nonperiodic Keys Machines that Generate Their Own Key Sequences Off-Line Forming of Key Sequences Nonperiodic Keys Individual, One-Time Keys Key Negotiation and Key Management Composition of Classes of Methods Group Property Superencryption Similarity of Encryption Methods Shannon s Pastry Dough Mixing Confusion and Diffusion by Arithmetical Operations DES and IDEA R Open Encryption Key Systems Symmetric and Asymmetric Encryption Methods One-Way Functions RSA Method Cryptanalytic Attack upon RSA Secrecy Versus Authentication Security of Public Key Systems Encryption Security Cryptographic Faults Maxims of Cryptology Shannon s Yardsticks Cryptology and Human Rights...226

10 Contents XI Part II: Cryptanalysis The Machinery Exhausting Combinatorial Complexity Monoalphabetic Simple Encryptions Monoalphabetic Polygraphic Encryptions Polyalphabetic Encryptions General Remarks on Combinatorial Complexity Cryptanalysis by Exhaustion Unicity Distance Practical Execution of Exhaustion Mechanizing the Exhaustion Anatomy of Language: Patterns Invariance of Repetition Patterns Exclusion of Encryption Methods Pattern Finding Finding of Polygraphic Patterns The Method of the Probable Word Automatic Exhaustion of the Instantiations of a Pattern Pangrams Polyalphabetic Case: Probable Words Non-Coincidence Exhaustion of Probable Word Position Binary Non-Coincidence Exhaustion The De Viaris Attack Zig-Zag Exhaustion of Probable Word Position The Method of Isomorphs A clever brute force method: EINSing Covert Plaintext-Cryptotext Compromise Anatomy of Language: Frequencies Exclusion of Encryption Methods Invariance of Partitions Intuitive Method: Frequency Profile Frequency Ordering Cliques and Matching of Partitions Optimal Matching Frequency of Multigrams The Combined Method of Frequency Matching Frequency Matching for Polygraphic Substitutions Free-Style Methods Unicity Distance Revisited Kappa and Chi Definition and Invariance of Kappa Definition and Invariance of Chi The Kappa-Chi Theorem The Kappa-Phi Theorem Symmetric Functions of Character Frequencies

11 XII Contents 17 Periodicity Examination The Kappa Test of Friedman Kappa Test for Multigrams Cryptanalysis by Machines: Searching for a period Kasiski Examination Building a Depth and Phi Test of Kullback Estimating the Period Length Alignment of Accompanying Alphabets Matching the Profile Aligning Against Known Alphabet Chi Test: Mutual Alignment of Accompanying Alphabets Reconstruction of the Primary Alphabet Kerckhoffs Symmetry of Position Stripping off Superencryption: Difference Method Decryption of Code Reconstruction of the Password Compromises Kerckhoffs Superimposition Superimposition for Encryptions with a Key Group COLOSSUS Adjustment in depth of Messages Cryptotext-Cryptotext Compromises Cryptotext-Cryptotext Compromise: ENIGMA Indicator Doubling Plaintext-Cryptotext Compromise: Feedback Cycle Linear Basis Analysis Reduction of Linear Polygraphic Substitutions Reconstruction of the Key Reconstruction of a Linear Shift Register Anagramming Transposition Double Columnar Transposition Multiple Anagramming Concluding Remarks Success in Breaking Mode of Operation of the Unauthorized Decryptor Illusory Security Importance of Cryptology Appendix: Axiomatic Information Theory Bibliography Index Photo Credits

12 List of Color Plates 4 Plate A Plate B Plate C The disk of Phaistos Brass cipher disks The Cryptograph of Wheatstone Plate D The US Army cylinder device M-94 Plate E Plate F The US strip device M-138-T4 The cipher machine of Kryha Plate G The Hagelin Cryptographer C-36 Plate H Plate I Plate K Plate L Plate M The US Army M-209, Hagelin licensed The cipher machine ENIGMA with four rotors Rotors of the ENIGMA The British rotor machine TYPEX Uhr box of the German Wehrmacht Plate N Cipher teletype machine Lorenz SZ 42 Plate O Plate P Plate Q Russian one-time pad Modern crypto board CRAY Supercomputers 4 In the middle of the book, following page 232.

13 Leone Battista Alberti ( ) Father of Western Cryptology (David Kahn)