Australian codebreakers of WWII

The very interesting book ‘Breaking Japanese Diplomatic Codes David Sissons and D Special Section during the Second World War’ is available for download from the Australian National University’s website.

The summary says:

During the Second World War, Australia maintained a super-secret organisation, the Diplomatic (or `D’) Special Section, dedicated to breaking Japanese diplomatic codes. The Section has remained officially secret as successive Australian Governments have consistently refused to admit that Australia ever intercepted diplomatic communications, even in war-time.

This book recounts the history of the Special Section and describes its code-breaking activities. It was a small but very select organisation, whose `technical’ members came from the worlds of Classics and Mathematics. It concentrated on lower-grade Japanese diplomatic codes and cyphers, such as J-19 (FUJI), LA and GEAM. However, towards the end of the war it also worked on some Soviet messages, evidently contributing to the effort to track down intelligence leakages from Australia to the Soviet Union.

This volume has been produced primarily as a result of painstaking efforts by David Sissons, who served in the Section for a brief period in 1945. From the 1980s through to his death in 2006, Sissons devoted much of his time as an academic in the Department of International Relations at ANU to compiling as much information as possible about the history and activities of the Section through correspondence with his former colleagues and through locating a report on Japanese diplomatic codes and cyphers which had been written by members of the Section in 1946. Selections of this correspondence, along with the 1946 report, are reproduced in this volume. They comprise a unique historical record, immensely useful to scholars and practitioners concerned with the science of cryptography as well as historians of the cryptological aspects of the war in the Pacific.

Typex cipher machines for the Polish Foreign Ministry

In 1926, the British Government set up an Inter-Departmental Cypher Committee to investigate the possibility of replacing the codebooks then used by the armed forces, the Foreign Office, the Colonial Office and the India Office with a cipher machine. It was understood that a cipher machine would be inherently more secure and much faster to use in encoding and decoding messages. Despite spending a considerable amount of money and evaluating various models by 1933 the committee had failed to find a suitable machine. Yet the need for such a device continued to exist and the Royal Air Force decided to independently fund such a project. The person in charge of their programme was Wing Commander Lywood, a member of their Signals Division. Lywood decided to focus on modifying an existing cipher machine and the one chosen was the commercially successful Enigma. Two more rotor positions were added in the scrambler unit and the machine was modified so that it could automatically print the enciphered text. This was done so these machines could be used in the DTN-Defence Teleprinter Network.

The new machine was called Typex (originally RAF Enigma with TypeX attachments). In terms of security it was similar to a commercial Enigma but had the additional security measure of multiple notches per rotor. This meant that during encipherment the rotors moved more often than in the standard Enigma machines. 

In the period 1939-45 the Typex was one of the main high level British crypto systems. According to documents found in British national archives HW 40/221 ‘Poland: reports and correspondence relating to the security of Polish communications’, it seems that the Polish government in exile learned about Typex and was interested in buying a small number of machines in 1944.

During WWII the Polish foreign ministry relied on enciphered codebooks for its secret communications. Perhaps they were interested in Typex because they considered their own systems insecure. Whatever the reason it doesn’t seem like they were given any machines since the report says ‘the supply position in respect of Type X is such that it is probably impossible to meet their requirements for the time being’

It is interesting to note that the same report says ‘provided the Type X machines supplied were not fitted with Plugboard and provided also we wired for them and supplied the necessary drums, the advantages to be gained by meeting their request would outweigh the disadvantages’.

Hmmm…..

Soviet pre-arranged form reports

The war between Nazi Germany and the Soviet Union was the largest land campaign of WWII, with millions of troops fighting in the vast areas of Eastern Europe. In this conflict both sides used every weapon available to them, from various models of tanks and self propelled guns to fighter and bomber aircraft. However an aspect of the war that has not received a lot of attention from historians is the use of signals intelligence and codebreaking by the Germans and the Soviets.

Codebreaking and signals intelligence played a major role in the German war effort. The German Army had 3 signal intelligence regiments (KONA units) assigned to the three Army groups in the East (Army Group North, South and Centre). In addition from 1942 another one was added to monitor Partisan traffic. The Luftwaffe had similar units assigned to the 3 Air Fleets (Luftflotten) providing aerial support to the Army Groups. Both the Army and the Luftwaffe also established central cryptanalytic departments (Horchleitstelle Ost and LN Regt 353) for the Eastern front in East Prussia. During the war this effort paid off as the German codebreakers could solve Soviet low, mid and high level cryptosystems. They also intercepted the internal radio teletype network carrying economic and military traffic and used traffic analysis and direction finding in order to identify the Soviet order of battle.

An important source of information on the Soviet military was their pre-arranged form reports sent at regular intervals by all units to their higher headquarters. These messages used a pre-arranged format to communicate strength, serviceability and loss statistics. By reading these messages the Germans were able to monitor the strength, losses and reinforcements of Soviet formations.

Luftwaffe Chi Stelle effort

Several TICOM sources give information on the exploitation of these pre-arranged reports by the codebreakers of the Luftwaffe. According to IF-187 Seabourne Report, Vol. XII. ‘Technical Operations in the East, Luftwaffe SIS’ (available from site Ticom Archive) pages 5-8 the reports had information on the condition of Soviet airfields, stocks of planes, ammunition, rations and fuel.

TICOM report I-107 ‘Preliminary Interrogation Report on Obltn. Chlubek and Lt. Rasch, both of III/LN. RGT. 353’, p4 says that the pre-arranged reports were extremely valuable to the Luftwaffe.

 

Army’ s General der Nachrichten Aufklaerung effort

According to FMS P-038 'German Radio intelligence', p115-7 pre-arranged reports sent by Soviet Army units contained information on personnel strength, losses, number of vehicles, guns, ammunition gasoline supplies and similar statistical data.

 

 

By analyzing this information the Germans were not only able to monitor the strength and equipment situation of enemy units but also make deductions about overall Soviet strategy.

Heartbleed bug and OSS codes

The recently discovered Heartbleed bug is considered to be one of the worst compromises of internet security, so check to see if the websites that you’re using have fixed it and change your passwords.

I have added information and pics in Allen Dulles and the compromise of OSS codes in WWII.

Soviet cryptologic security failures in WWII – A sneak peak

I’ve already covered the cryptologic failures of the United States and Britain in WWII but I still haven’t covered the Soviet Union. According to Soviet/Russian sources their codes were impenetrable and the Germans were never able to compromise their high level communications links. Is that true?

Well I’m still researching this case and I haven’t copied all the available documents. Once I do I will write a detailed essay on Soviet codes.

For now here is a sneak peak:

Update

I have added information and pics in The US TELWA code.

Some thoughts on Soviet tank reliability in WWII

The Eastern front was the largest land campaign of WWII and millions of soldiers fought and died there in the period 1941-45. Although infantry dominated the fighting both sides used a large number of tanks and armored vehicles and these played a big role in breakthrough operations. Most historians focus on the ‘paper’ characteristics of tanks and the production statistics however a very important aspect of complex weapon systems is their reliability and kill/loss ratio. In the East the Germans were always outnumbered but the exchange ratios were in their favor. I’ve often wondered of how much that has to do with poor reliability of Soviet equipment.

Here is something I read recently from ‘Moscow to Stalingrad: Decision in the East’ by Earl F. Ziemke, in page 363:

Active as it was, the Soviet armor was apparently not giving fully satisfactory performance at this stage, and in early August, it became the subject of the following Stalin order:

‘Our armored forces and their units frequently suffer greater losses through mechanical breakdowns than they do in battle. For example, at Stalingrad Front in six days twelve of our tank brigades lost 326 out of their 400 tanks. Of those about 260 owed to mechanical problems. Many of the tanks were abandoned on the battlefield. Similar instances can be observed on other fronts. Since such a high incidence of mechanical defects is implausible, the Supreme Headquarters sees in it covert sabotage and wrecking by certain elements in the tank crews who try to exploit small mechanical troubles to avoid battle.’

Henceforth, every tank leaving the battlefield for alleged mechanical reasons was to be gone over by technicians, and if sabotage was suspected, the crews were to be put into tank punishment companies or "degraded to the infantry" and put into infantry punishment companies.'"

Were the problems really caused by sabotage and wreckers? Apparently not, since captured T-34 tanks used by the Germans in summer 1944 had the following problems:

‘Regardless of our limited experience, it can be stated that the Russian tanks are not suitable for long road marches and high speeds. It has turned out that the highest speed that can be achieved is 10 to 12 km/hr. It is also necessary on marches to halt every half hour for at least 15 to 20 minutes to let the machine cool down. Difficulties and breakdowns of the steering clutches have occurred with all the new Beute-Panzer. In difficult terrain, on the march, and during the attack, in which the Panzer must be frequently steered and turned, within a short time the steering clutches overheat and are coated with oil. The result is that the clutches don't grip and the Panzer is no longer maneuverable. After they have cooled, the clutches must be rinsed with a lot of fuel.’

Also T-34 tanks captured by the Americans in Korea (built in 1945) continued to suffer from the same issues. According to Zaloga’s ‘T-34-85 Medium Tank’, p21-22

An analysis of a T-34-85 captured in Korea by the American tank producer Chrysler, conducted in 1951, provides a good assessment of the T-34- 85……………………. The study, found the following negative features about the tank:…………………………………. Wholly inadequate engine intake air cleaners could be expected to allow early engine failure due to dust intake and the resulting abrasive wear. Several hundred miles in very dusty operation would probably be accompanied by severe engine power loss.' The report was also critical of the lack of a turret basket, poor fire fighting equipment, poor electrical weatherproofing, lack of an auxiliary generator to keep the batteries charged, and lack of a means to heat engine oil for cold weather starts. The report noted that although Soviet manufacturing techniques were adequate for the job, there were many instances where poor or unskilled workmanship undermined the design, and where overworked machines led to course feeds, severe chatter or tearing of machined surfaces, a consequence no doubt of the extreme pressures placed on plants to ensure maximum output. For example, in the tank inspected (manufactured in 1945) the soldering job on the radiator was so poor that it effectively lost half of its capacity.

It’s also worth noting that even in 1941 German reports on captured Soviet T-26 and BT tanks pointed out serious productions issues. For the T-26 tank: ‘The Pz.Kpfw.Zug created by the division is no longer operational. One Panzer is completely burnt out due to an engine fire. Both of the other Panzers have engine and transmission problems. Repetitive repairs were unsuccessful. The Panzers always broke down after being driven several hundred meters on good roads. As reported by technical personnel, both of the engines in the Panzers are unusable because they were incorrectly run in.’

And for the BT tank: ‘B. T. (Christi): The main cause of failure is a transmission that is too weak in combination with a strong engine that should provide the tank with high speed, but is over-stressed when driven off road where the lower gears must be used for longer periods. In addition, as in the T 26, problems continuously arise that are due to entire design and poor materials, such as failure of the electrical system, stoppages in fuel delivery, breaks in the oil circulation lines, etc.’

Finally there are the Aberdeen tests on a T-34 tank:

'On the T-34 the transmission is also very poor. When it was being operated, the cogs completely fell to pieces (on all the cogwheels). A chemical analysis of the cogs on the cogwheels showed that their thermal treatment is very poor and does not in any way meet American standards for such mechanisms.’

‘The deficiency of our diesels is the criminally poor air cleaners on the T-34. The Americans consider that only a saboteur could have constructed such a device’

The reliability issues of Soviet tanks during WWII point to serious problems with Soviet industry. The only other explanation is that a huge Nazi/White Guard wrecker movement existed in Soviet factories…

I think that even comrade Stalin would find this idea implausible!

The epic quest for the Carlson-Goldsberry report

During WWII the US State Department used several cryptosystems in order to protect its radio communications from the Axis powers. For low level messages the unenciphered Gray and Brown codebooks were used.  For important messages four different codebooks (A1,B1,C1,D1) enciphered with substitution tables were available.

Their most modern and (in theory) secure system was the M-138-A strip cipher. Unfortunately for the Americans this system was compromised and diplomatic messages were read by the Germans, Finns, Japanese, Italians and Hungarians. The strip cipher carried the most important diplomatic traffic of the United States (at least until late 1944) and by reading these messages the Axis powers gained insights into global US policy.

Germans, Finns and Japanese cooperated on the solution of the strip cipher. The Japanese gave to the Germans alphabet strips and numerical keys that they had copied from a US consulate and these were passed on by the Germans to their Finnish allies. Then in 1943 the Finns started sharing their results with Japan.

The German effort

Unfortunately the information we have today on the compromise of the State Department’s strips cipher is limited. One problem is that the archives of the agencies that worked on this system are not available to researchers. Three different German agencies worked on the US diplomatic M-138-A strip cipher. The German High Command’s deciphering department – OKW/Chi, the Foreign Ministry’s deciphering deparment Pers Z and the Air Ministry’s Research Department - Reichsluftfahrtministerium Forschungsamt.

I know that the NSA has some interesting reports on the codebreaking successes of the Forschungsamt but they have not been declassified yet. Regarding OKW/Chi I don’t know if their archives (or parts of them) survived the war. Finally the files of the Pers Z agency were captured by the Anglo-Americans at the end of the war but the reports I’ve seen from the National Archives and Records Administration are mostly administrative files.

This means that so far our sources on the strip compromise are mainly TICOM reports written postwar.

The Finnish codebreakers and the strip cipher

The Finnish codebreakers also worked on the strip cipher and solved several links in the period 1942-44. In this area there was cooperation with their German counterparts, not only in receiving copies of the Japanese cipher material but also exchanges of personnel and analysis of the strip system.

The fact that the Finns cooperated with the Germans against this cryptosystem means that we can find out more about the German operation through Finnish sources and thus circumvent the lack of German archival sources.

Operation Stella Polaris

In September 1944 Finland signed an armistice with the Soviet Union. The people in charge of the Finnish signal intelligence service anticipated this move and fearing a Soviet takeover of the country had taken measures to relocate the radio service to Sweden. This operation was called Stella Polaris (Polar Star).

In late September roughly 700 people, comprising members of the intelligence services and their families were transported by ship to Sweden. The Finns had come to an agreement with the Swedish intelligence service that their people would be allowed to stay and in return the Swedes would get the Finnish crypto archives and their radio equipment. At the same time colonel Hallamaa, head of the signals intelligence service, gathered funds for the Stella Polaris group by selling the solved codes in the Finnish archives to the Americans, British and Japanese. The Stella Polaris operation was dependent on secrecy. However the open market for Soviet codes made the Swedish government uneasy. In the end most of the Finnish personnel chose to return to Finland, since the feared Soviet takeover did not materialize. 

The American reaction and the Carlson-Goldsberry report

According to the NSA study History of Venona (Ft. George G. Meade: Center for Cryptologic History, 1995) by Robert Louis Benson and Cecil J. Phillips, it was at that time that the Finns revealed to the US authorities that they had solved their diplomatic codes. On 29 September 1944 colonel Hallamaa met with L.Randolph Higgs of the US embassy in Stockholm and told him about their success.

In response two cryptanalysts were sent from the US to evaluate the compromise of US codes in more detail. They were Paavo Carlson of the Army’s Signal Security Agency-SSA and Paul E. Goldsberry of the State Department’s cipher unit. Their report dated 23 November 1944 had details on the solution of US systems.

Unfortunately finding this report has proven to be quite a problem!

Freedom of Information Act requests

After trying to find this report in the US archives i gave up and filed FOIA requests with the State Department, NSA and NARA. The results:

1).  The State Department told me that they no longer have these files as they have been sent to NARA so I should bother them.

2). NARA could not locate this file but they did send me a list of references that I should look up.

3). The NSA informed me that they had expended the free time allowed for research and if I wanted to continue I’d have to pay. I decided not to.

Assistant Secretary Shaw

Apart from the FOIA requests I tried to find information on the people responsible for evaluating the compromise of State Department codes during the war.  A name that came up in relevant reports was Assistant Secretary Shaw. This was Gardiner Howland Shaw, Assistant Secretary of State in the period 1941-44 and in charge of the State Department’s cipher unit. Unfortunately NARA does not have a separate body of records for G. Howland Shaw.

Another lead I followed was the Shaw foundation but their response was that ‘To our knowledge, he left no immediate family members and we have no record of any of his State Dept work.’

The messages from the embassy in Sweden

After failing to find anything either with the FOIA requests or the Shaw search I decided it would be best to try to track down the messages sent from the US embassy Sweden to Washington during the days mentoned in ‘History of Venona’. Unfortunately the State Department messages are indexed according to a complicated system and it is very difficult to find anything:

So I asked NARA again if they could locate the messages of the embassy in Sweden for these specific dates and their response was:

‘We searched the Source Cards, 1940-1944; General Records of the Department of State, Record Group 59 and located index cards which lead us to believe that no record of these sensitive meetings/topics were kept by the State Department.  It is possible, though, that further examination of this series may yield records which may be pertinent to your research.’

They also gave me a reference to an OSS report in the Director’s files but I had already checked that.

Now I have to give credit where credit is due and the NARA people really did some great work in this case! Unfortunately even after all these efforts the Carlson-Goldsberry report continues to elude us…

A small win?

Although I haven’t been able to find the actual report I think that a page found in NARA-RG 457-Entry 9032-box 214-‘M-138-A numerical keys/daily key table/alphabet strips’ is a part of that report or at least contains information from it.

As can be seen in that page it says Department of State- Assistant Secretary, which should be G. Howland Shaw and the date says 23/11/1944, which matches the ‘History of Venona’ date. The file shows the coupling of alphabet strips with a set of keylists. This implies that the State Department did not use separate sets for each embassy but instead had a limited number of strips and keylists that were rearranged during the war.

Update March 2018: The NSA FOIA/MDR office has declassified this report. (Google drive link)