United States cryptologic security failures in WWII

Signals intelligence and codebreaking played an important role in WWII. British and American codebreakers solved many important Axis crypto systems, such as the German Enigma machine and the Japanese Navy’s code JN25. 

Historians have not only acknowledged these Allied successes but they’ve probably exaggerated their importance in the actual campaigns of the war.

Unfortunately the work of the Axis codebreakers hasn’t received similar attention. As I’ve mentioned in my piece Acknowledging failures of crypto security all the participants suffered setbacks from weak/compromised codes and they all had some successes with enemy systems. 

Britain, the Soviet Union and the United States did not have impenetrable codes. In the course of WWII all three suffered setbacks from their compromised communications.

After having dealt with the British side let’s have a look at the Americans and their worst failures. 

First it’s time for a short history lesson on the organizations responsible for making and breaking codes.

Herbert Yardley, MI-8 and the American Black chamber

The first dedicated codebreaking unit of the US military was organized during WWI by Herbert Osborne Yardley. Yardley had worked as a telegrapher and then as a State Department code clerk. During WWI he demonstrated the insecurity of US diplomatic codes by solving, on his own, a message sent from Colonel House to President Woodrow Wilson. This attracted the attention of his superiors and Colonel Ralph H. Van Deman, head of the Military Intelligence Division, made him a first lieutenant and assigned him head of the MI-8 department, responsible for codes and ciphers.

The MI-8 unit solved several foreign codes and their success led the War Department and the State Department to jointly fund Yardley’s activities in the postwar era. The codebreaking department was moved to New York and called the ‘Black chamber’. Their main effort in the 1920’s was against Japanese diplomatic codes and in this area they were able to prove their worth. Yardley’s group not only solved regular Japanese diplomatic traffic but scored a great victory during the Washington Naval Conference by discovering Japan’s minimum acceptable battleship requirements. This allowed the US diplomats to get the Japanese representatives to agree to a battleship ratio of 5-5-3 for USA-UK-Japan.

‘Gentlemen do not read each other's mail’ and the Great Depression

Yardley’s unit had performed well in the 1920’s but the new Secretary of State Henry L. Stimson was shocked when he learned of the existence of the ‘Black chamber’ and famously stated ‘Gentlemen do not read each other's mail’. Without government funds Yardley’s group disbanded in October 1929. It was at this time that the world experienced an economic downturn, called the Great Depression.

Finding himself without a job and with his investments having lost their worth Yardley was forced to write about his secret activities in order to get money. He wrote the history of the ‘Black chamber’ for the ‘Saturday Evening Post’ and later published the book ‘The American Black Chamber’ which became a best-seller not only in the US but around the world and especially in Japan.

By publishing a summary of his codebreaking activities Yardley compromised the solution of foreign codes and in that sense hurt US national interests. Many foreign governments changed their codes after the publication of the book.  The US government wanted to prosecute Yardley but since he hadn’t broken an existing law the Espionage Act was amended to prohibit the disclosure of foreign codes or anything sent in code.

US Army and Navy agencies

The decision by the US government to stop codebreaking activities and the limited funds available for military spending had a negative impact on the performance of US signals intelligence in the 1930’s.

Despite these problems two small groups under the Army and the Navy continued to work on the solution of foreign codes, with considerable success in the case of Japanese systems.

In the field of crypto security the new systems introduced from the 1930’s up to 1941 were the M-138-A strip cipher and the SIGABA cipher machine.

 SIS- Signal Intelligence Service

The US Army concentrated its cryptologic functions in the newly established Signal Intelligence Service, whose cipher research department was headed by William F. Friedman. The SIS was part of the Signal Corps and officially they were responsible only for preparing and testing the security of US Army codes. However in practice the military authorities were not willing to stop all codebreaking work so the department continued to attack foreign codes.

The work of the SIS was made easier by the appointment of Major General Joseph O. Mauborgne as Chief Signal Officer in the period 1937-41. Mauborgne was no stranger to cryptology and while in office he expanded the SIS and made the unit report directly to him.

In the 1930’s Friedman hired a small group of young mathematicians (Rowlett, Sinkov, Kullback) and the department concentrated on mathematical research in order to solve foreign codes. Their main success up to the Pearl Harbor attack was the solution of the Purple cipher machine, used by Japan’s Foreign Ministry. This success enabled them to read the diplomatic messages sent from Tokyo to its embassies abroad.

Their other great success was the development of the SIGABA cipher machine, a device that was vastly more secure than any other cipher machine of that time period.

Navy’s OP-20-G

The US Navy had its own cryptologic unit, the OP-20-G - Office of Chief Of Naval Operations (OPNAV), 20th Division of the Office of Naval Communications, G Section / Communications Security.

This unit was headed by Commander Laurence F. Safford and was responsible not only for codebreaking but also direction finding, traffic analysis and the production of codes for USN use.

The naval codebreakers worked mostly on Japanese naval codes and they were able to solve these from the 1920’s up to 1940.

In the field of crypto security they adopted the Army’s idea for SIGABA and named their modified version ECM MARK II. In 1941 Army and Navy agreed to jointly produce the modified SIGABA/Converter M-134-C/ECM MARK II as their high level crypto system.

Overview of US crypto systems in 1941

In the 1930’s the US cryptologic agencies were hampered in their operations by the lack of adequate resources. Still in that timeframe they were able to introduce the M-138-A strip cipher and the SIGABA. The M-138-A was quite hard to solve, provided it was used properly, while the SIGABA was the most secure cipher machine of WWII.

The main problem for US crypto security was the continued use of a large number of outdated systems. Without the necessary funding both military and civilian authorities were forced to use old codes and ciphers, that were not only insecure from a security standpoint but were also suspected to have been physically compromised during their long time in service!

Military systems

US Army and Navy used the M-94 cylinder even though it offered very limited security and had been introduced in 1921. The Army and USAAF would continue to rely on it even as late as 1943.

For administrative traffic the services used the War Department Telegraph Code 1919 edition - SIGRIM. This was a 5-letter codebook used without additional encipherment.

Systems considered to be quite secure were the Military Intelligence Code and the War Department Confidential Code (both letter codes). They were enciphered with substitution tables.

The Military Intelligence Code No5 had been printed in 1918, Military Intelligence Code No9 in 1919, Military Intelligence Code No10 in 1927, Military Intelligence Code No11 in 1933 and Military Intelligence Code No12 in 1935.

The War Department Confidential Code No1 was introduced in the 1930’s. It was not a new codebook but rather the old Military Intelligence Code No5, provided with a new title page and supplement.

It seems that the War Department Confidential Code No2 also followed this system. According to a 1943 message of the Japanese military attaché in Hungary the War Department Confidential Code No2 was the same as the Military Intelligence Code No12.

Diplomatic systems

The State Department had a small unit responsible for the production of codes in the Division of Communications and Records. This unit was controlled by David Aden Salmon.

The basic cryptosystems were codebooks. These were the enciphered codes A1 (introduced in 1919), B1 (introduced in 1920), C1 (introduced in 1927) and D1 (introduced in 1928). The unenciphered Gray code (introduced in 1918) was used for low level traffic.

In the late 1930’s two new systems were introduced, the Brown codebook and the M-138-A strip cipher. Although there was a survey proposing the introduction of cipher machines this idea was rejected for financial reasons.

Voice communications

The Bell Labs A-3 speech scrambler was used by the military authorities and on the civilian link Washington-London. Even though it wasn’t considered 100% secure it was the only speech privacy system in widespread use.

Failures of crypto security during WWII

During WWII military high level communications were secure, thanks to the advanced SIGABA machine, however the other cryptologic systems used by the US military and civilian authorities had vulnerabilities and the Axis powers were able to compromise almost all of them.

Notable cases

Fellers code

Colonel Bonner Fellers, US military attaché in Cairo during 1940-2, sent back to Washington detailed reports concerning the conflict in North Africa. In his reports he mentioned morale, the transfer of British forces, evaluation of equipment and tactics, location of specific units and often gave accurate statistical data on the number of British tanks and planes by type and working order. In some cases his messages betrayed upcoming operations.

Fellers used the Military Intelligence Code No11, together with substitution tables. The Italian codebreakers had a unit called Sezione Prelevamento (Extraction Section). This unit entered embassies and consulates and copied cipher material. In 1941 they were able to enter the US embassy in Rome and they copied the MI Code No11. A copy was sent to their German Allies, specifically the German High Command's deciphering department – OKW/Chi. The Germans got a copy of the substitution tables from their Hungarian allies and from December 1941 they were able to solve messages. Once the substitution tables changed they could solve the new ones since they had the codebook and they could take advantage of the standardized form of the reports. Messages were solved till 29 June 1942 and they provided Rommel with so much valuable information that he referred to Fellers as his ‘good source’.

A-3 speech scrambler

The Bell Labs A-3 speech scrambler was used by the military and on the civilian link Washington-London. Two different German teams solved this system and they were able to decode the conversations in real-time.  Traffic was successfully recorded from late 1941 up to late 1944. Through this operation they got political, economic and military intelligence. Their greatest success was the interception, on 29 July 1943, of a conversation between Roosevelt and Churchill which revealed negotiations with the newly established Badoglio government in Italy. This convinced the Germans that the Italians were trying to exit the war and thus they stepped up their plans to occupy the whole country.

M-94 strip cipher

The strip cipher M-94 was the basic US military cryptosystem in the interwar period and continued to be used widely by the Army and USAAF even as late as mid 1943. As a system it offered limited security and in early 1942 the German Army’s codebreakers solved it. According to the war diary of Inspectorate 7/VI it was solved in May ’42 and traffic from several networks read till July 1943. In German reports the M-94 was called ACr2. Some of the networks read had the indicators CDAF, URSAL, USABU, SENOB as can be seen from the following report taken from the war diary of Inspectorate 7/VI, month of September 1942:

The M-94 was used for administrative traffic and by military units at the division level. It was replaced in 1943 by the M-209 cipher machine.

M-209 cipher machine

The US authorities had the SIGABA for high level traffic but lacked a secure device for mid-level traffic. The Swedish inventor Boris Hagelin had developed a small enciphering device called the C-38, which was an improvement of his earlier design C-36 (sold in the 1930’s to the French Army). When he offered this device to the US government it was tested by the SIS and after minor modifications it was produced from 1942 till the end of the war as the M-209. Roughly 140.000 devices were built. The M-209 was used by the US Army at division level and by the USAAF for administrative traffic.

Its first use in the field was during the Tunisia Campaign of 1942-43. According to the war diary of Inspectorate 7/VI the German Army’s codebreakers investigated this traffic in the first half of 1943, ascertained that it was a Hagelin type device and found ways of solving it by using two messages ‘in depth’ (enciphered with the same internal and external settings). By retrieving the internal settings they were able to decode the entire day’s traffic. Their designation for the M-209 was AM-1 (Amerikanische Maschine 1).

Reports based on decoded M-209 messages were first issued in July 1943, as can be seen from the following paragraph taken from the war diary of Inspectorate 7/VI, month of July ‘43:

The M-209 continued to be solved till March ’45, with the following list showing date/indicator/frequency/department that solved relative settings/ department that solved absolute settings/date solved.

It’s interesting to note that in 1944 the Germans even built a cryptanalytic device for speeding up the solution of M-209 messages.

Slidex

The Slidex code was a simple bigram substitution table. It was used extensively by the US forces in 1944 during the liberation of Western Europe. Slidex offered very limited security but was well liked by troops because it was easy to use.

The Germans had no problem in solving the messages and reconstructing the Slidex tables. In late 1944 their solution of Slidex traffic from military police units gave them an advantage during the battle of the Bulge. This episode shows that even the compromise of low level codes can sometimes have strategic consequences.

However the limitations of Slidex had not gone undetected and in January ’45 the Signal Division recommended that Slidex be replaced within the U.S. forces.

War Department Telegraph Code

The War Department Telegraph Code was used for administrative traffic. The 1919 edition - SIGRIM was used at the start of the war till 1943-44 when the new version WDTC 1942 edition - SIGARM was introduced.

According to TICOM reports both versions were solved by the Germans. The war diary of Inspectorate 7/VI shows that a codebook A1 (later called AC1) was solved since 1942. It’s not clear to me whether this A1 code was the WDTC 1919 edition or the A1 also used by the State Department.

The next version WDTC 1942 edition was called TELWA by the Germans due to its indicator. It was solved cryptanalytically, in part by taking advantage of a parity check in its values.

It doesn’t seem like TELWA survived the war. A US report from May ’45 said that it would be replaced by a new system and existing copies had to be destroyed.

ACAN - Army Command and Administrative Network

The radio network of the US military was called ACAN - Army Command and Administrative Network. During the war it was greatly expanded in order to cover the worldwide deployment of US forces and satisfy their needs for secure and reliable communications.

German radio intelligence agencies intercepted traffic of the ACAN network and were even able to follow the activation and deployment of military units from the US interior to the theatres of war.

In this case they took advantage of the limitations of commercial equipment used by ACAN. In the period 1941-43 the network had to be built up fast so some types of equipment that did not fully meet specifications were accepted. The Signal Corps wanted modern radio-teletype equipment of the Baudot type with automatic enciphering and deciphering capability but up to 1943 no device satisfied these requirements.

While new equipment was in development the Signal Corps had to make use of a commercial design. The IBM corporation sold the Radiotype a 6-unit teletype machine that was a commercial success. In 1942 the Signal Corps introduced the Radiotype in military networks and the official history ‘Signal Corps - The Test’ says:

‘The International Business Machines Corporation had worked out an imperfect solution involving equipment that the firm called radiotype, using, unfortunately, not the standard five-unit teletypewriter code but a special six-unit code. Like a narrow gauge railroad adjoining a standard line, this special code necessitated much hand labor at conversion points where standard teletypewriter texts had to be shifted onto radiotype circuits, and vice versa. Moreover, the standard automatic cipher machines could not function with the six-unit system. Notwithstanding these inconveniences, the Signal Corps early in the war began making use of radiotype, leased from IBM. It was another step in the right direction, toward automatic, high-speed, heavy-duty communications for the Army.’

The same source says that the use of radiotype was extensive during the war: ‘Radiotype would continue to be used considerably. Not till September 1943 would the Signal Corps stop its procurement in favor of radioteletype and not until May 1945 would the Army take its last radiotype out of service (on the WAR-Accra circuit). Then the triumph of radioteletype would be complete.’

The main problem with Radiotype was that due to its 6-unit code it could not be connected to a standard cipher machine since these used the Baudot 5-code system. While it was used Radiotype allowed the Germans to intercept valuable traffic up to mid-late 1943 when the new SIGCUM and SIGTOT teletype devices were introduced.

Diplomatic and OSS systems

The State Department’s A1 and C1 codebooks were read by German codebreakers through cryptanalysis. The low level Gray code had been solved in the 1930s and the new Brown code was received from the Japanese in 1941 together with the A1.

The M-138-A strip cipher was the State Department’s high level system and it was used extensively in the period 1941-44. Although we still don’t know the full story the information available points to a serious compromise both of the circular traffic (Washington to all embassies) and special traffic (Washington to specific embassy) in the period 1941-44. In this area there was cooperation between Germany, Japan and Finland. The German success was made possible thanks to alphabet strips and key lists they received from the Japanese in 1941 and these were passed on by the Germans to their Finnish allies in 1942. The Finnish codebreakers solved several diplomatic links in that year and in 1943 started sharing their findings with the Japanese.

German and Finnish codebreakers cooperated in the solution of the strips during the war, with visits of personnel to each country. The Axis codebreakers took advantage of mistakes in the use of the strip cipher by the State Department’s cipher unit. Apart from diplomatic messages their success against State Department systems also allowed them to read messages of other US agencies such as the OSS -Office of Strategic Services Bern station, Military Attaché in Switzerland. Office of War Information representative in Switzerland and also the Foreign Economic Administration, War Shipping Administration, Office of Lend-Lease Administration and the War Refugee Board.

Naval Cypher No3

Naval Cypher No3 was a British 4-figure code enciphered with subtractor tables. It was used in the Atlantic by the US, UK and Canadian Navies in their convoy operations and its compromise by the codebreakers of the B-Dienst led to heavy losses for the Allied merchant ships.

Since it was a Royal Navy system its failure should be attributed to the British side, however the Americans contributed by withholding their secure cipher machine. In theory the ECM MARK II could have been given to the Royal Navy to equip the units operating in the North Atlantic but this was not done for reasons of national security, since only US personnel were allowed to use this advanced machine and the US authorities did not want to divulge its operating principle even to their closest ally!

Other systems: 

The Germans easily solved several editions of the Division Field Code used by frontline troops, the Bomber code used by USAAF units during the combined bomber offensive, the Combined Assault Code used during naval landings and the Aircraft movement code used by the USAAF ferry service.

A more interesting case is the AN/GSQ-1 speech scrambler. One of these devices was retrieved from a crashed US fighter plane and the codebreakers of OKW/Chi were able to find a solution, although the chief cryptanalyst Huettenhain was doubtful of whether the key could have been solved in the few hours that the missions took place.   

Responding to failures of crypto security

In the course of WWII US Army and Navy cryptologists worked hard to secure existing codes and design new ones. Special teams of codebreakers tried to solve US systems by every way possible. Based on their findings changes were made in operational procedures.

The Army created dedicated radio units called SIAM - Signal Information and Monitoring, whose sole mission was to monitor the radio traffic of US units for violations of signal procedures and cipher security.

A case that shows how seriously cipher security was taken by the US was the Colmar incident. A truck carrying the SIGABA machine of the 28^th Infantry division was lost in Colmar, France in February 1945. The vehicle had been stolen by someone while the crew where sleeping indoors. Immediately a huge search was organized to retrieve the vehicle and the cipher material and it was eventually found on 9 March ’45 in a wooded area but with the safe missing. During the same day a French unit found the safe submerged in the Gressen River. After the safe was examined it did not show any signs of being opened nor was material missing. In addition the rotors of the machine were set up in the correct arrangement for 5 February. Even though the examination showed that the material had not been compromised a decision was made to rewire the rotors for all the SIGABA machines in use!

Denying cases of compromise

Based on the information presented so far one would expect that any sign of compromised codes would have led to an exhaustive search for the truth. As with all things in life the reality was more complicated.

Although the Americans placed a high value on crypto security there were some embarrassing cases during the war where they obstinately refused to admit (and in some cases still do) that their systems were compromised.

Fellers code

In 1942 when the codebreakers of Bletchley Park decoded German Enigma messages from the ME theatre, they were surprised to find information that could only have come from the US diplomatic mission in Egypt. This convinced them that a cipher used between Washington-Cairo was being read by the Germans but they found it very, very hard to convince the Americans.

When the US authorities were informed of the affair they refused to believe that they were the source of the leak and instead suspected the Brits of having solved US codes. In his NSA interviews the legendary codebreaker Frank Rowlett says about this affair ‘G2’s reaction was as follows: This could not happen in G2. It could not be Fellers.’

The code was finally changed on 29 June 1942. According to Rowlett a SIGABA machine was sent to Egypt to replace the MI code.

State Department strip cipher

During WWII there was exchange of information on State Department codes between Germans, Finns and Japanese. The Finnish codebreakers had solved several links in 1942 and in early 1943 they shared their results with the Japanese. These messages were in turn decoded by the British codebreakers and clearly revealed the compromise of State Department systems.

What was the US response when they were told of this? Did they quickly institute several security changes? Let’s have a look:

The belief of the US officials that their systems were essentially secure meant the Germans and Finns would continue to solve the strip cipher till late 1944!

Office of Strategic Services - Bern station

Several sources state that the messages of the OSS station in Berne were read during the war. It is possible that this only happened when they used State Department codes (for convenience) or when some of their reports of general political nature were given to the US embassy for transmission to the Secretary of State.

It is definitely amusing that the Americans were warned twice to change their codes by none other than Admiral Canaris, head of the military intelligence service Abwehr (through the agent Halina Szymańska) and by General Walter Schellenberg of the Sicherheitsdienst.

The American response was: ‘Dulles made little secret of what he was doing, but he was good at separating valuable informants from Nazi plants, and his codes were never broken. He recognized Schellenberg’s ploy.’

The Brits were smarter than that and their report ZIP/D-S/G.9 of 10^th April 1943 says:

Conclusion

The USA entered WWII in 1941 with the secure cipher machine SIGABA, the M-138-A strip cipher that offered adequate security but was burdensome to use and a large number of outdated and insecure crypto systems. In the course of the war modern cipher machines were designed and built to replace the old systems and securely cover all types of traffic.

In 1942 the M-209 device was used in the field and in 1943 the cipher teleprinters Converter M-228 - SIGCUM and SIGTOT were introduced in communications networks. In the summer of ’43 a new speech privacy device called SIGSALY became operational and the first system was used on the link Pentagon-London.  At the end of the year the CCM - Combined Cipher Machine was used in the Atlantic and in 1944-45 the British relied on the CCM as much as they did on their own Typex.

By the end of war the US had several types of cipher machines in use, all offering a very high degree of security. Even older systems like the M-138-A and the codebooks were used in such a way that solution was very difficult if not impossible (daily change of key settings, use of one time pads).

However the success of the US in securing its codes should not hide the failures of crypto security that took place during the war. Especially the Fellers case, the A-3 speech scrambler and the State Department’s strip cipher revealed to the enemy valuable intelligence. At the same time several cases of compromise took too long to resolve due to the belief of US officials that their codes were impregnable.

It’s up to historians to cover these historical events in more detail.  

Sources: ‘The Codebreakers’, ‘Hitler’s spies’, NSA interviews of Frank Rowlett (NSA oral histories 1974 and 1976), various TICOM reports, ‘European Axis Signal Intelligence in World War II’, Cryptologia article: ‘The Sigcum story: cryptographic failure, cryptologic success’, Cryptologia article: ‘The ECM MARK II: design, history, and cryptology’, SRH-366 ‘The history of Army strip cipher devices’, Cryptologia article: ‘The Slidex RT Code ‘, ‘Achievements of the Signal Security Agency in World War II’, ‘United States Diplomatic Codes and Ciphers: 1775-1938’, State Department history: ‘Division of cryptography’, War Diary of Inspectorate 7/VI, FMS P-038 ‘German radio intelligence’, ‘Delusions of intelligence’  USS PAMPANITO, British archives - HW 40/132 , Wikipedia, Japanese JMA message SAC/JMA 69, ‘The history of codes and ciphers in the United States during the period between the world wars part ii. 1930-1939’

Pics: M-94 and M-138-A pics from SRH-366, paragraph on State department security survey from ‘Division of cryptography’, solution list for M-209 taken from TICOM DF-120, report on TELWA found in NARA-RG 457- box 797, ACAN pics from FMS P-038, paragraph on AN/GSQ-1 taken from TICOM I-31, strip cipher compromise messages from British archives HW 40/132.

Acknowledgments: I have to thank Rene Stein of the National Cryptologic Museum for the Frank Rowlett files and TICOM report I-31.