British disapprove of US State department crypto security

It seems that during WWII the British codebreakers did not have much respect for the security of US diplomatic communications:

 What report are they referring to? This one:

Pseudo and inartistic folk songs

Thanks to help from Ralph Erskine another interesting file turns up. Now we all know the evils of communism. Gulag, censorship, propaganda, Party rule etc etc

However there are some things that cannot be tolerated.

 

‘The love of Stepan Razin’ inartistic? You’ve crossed the line there comrade!

German intelligence on operation Overlord – more clues

In 1944 the German High Command knew that the Anglo-Americans would stage a cross channel landing into France. If they succeeded it would be the end of the German Reich. The area they considered to be most endangered was initially Pas de Calais.

However by April/May 1944 they started to move new units into Normandy and they positioned powerful armored divisions close by (21^st Panzer, 12 SS Hitlerjugend, Panzer Lehr).

Why did the Germans place new units in Normandy? Were they expecting the Normandy landings?

I’ve tried to look into all the available sources of information in my essay German intelligence on operation Overlord.

According to report FMS B-675 ‘Army Group B-Intelligence Estimate (1 Jun 1944)’ By Oberst i. G. Anton Staubwasser (head of intelligence for Army Group B in 1944-45) the German High Command became convinced that Normandy would be the site of Allied landings in April/May ’44:

It is important that - for the first time in April/May - Hitler informed OB West, through General O. JODEL, as follows: "The Fuehrer has definite information that Normandy is endangered." It has not become known from, what source this news originated. This message was immediately and also later repeatedly passed on to A Gp B and all armies of the west, that is, approximately 4 weeks before the beginning of the invasion. This is also the reason for the transfer of the 91 Luftwaffe Division, several armored battalions and antitank battalions to the COTENTIN peninsula and for the assembly by OKW of the Pz Lehr Division.

Here is another piece of the puzzle. The following episode is described by Albert Speer in his memoirs ‘Inside the Third Reich’, p479

On June 1, I was at the Berghof about ten o'clock in the morning when one of Hitler's military adjutants told me that the invasion had begun early that morning. "Has the Fuehrer been awakened?" I asked. He shook his head. "No, he receives the news after he has eaten breakfast."

In recent days Hitler had kept on saying that the enemy would probably begin with a feigned attack in order to draw our troops away from the ultimate invasion site. So no one wanted to awaken Hitler and be ranted at for having judged the situation wrongly.

At the situation conference in the Berghof salon a few hours later Hitler seemed more set than ever on his preconceived idea that the enemy was only trying to mislead him. "Do you recall? Among the many reports we've received there was one that exactly predicted the landing site and the day and hour. That only confirms my opinion that this is not the real invasion yet."

Hmm… What report was Hitler referring to?

Gulf war studies

The military conflicts between Saddam’s Iraq and the US coalition in Gulf war I and in 2003 are usually assumed to have been completely one sided, with Iraqi forces abandoning their weapons and surrendering en mass.

The superior performance of Western weapons vs their Soviet equivalents is also easily countered by pointing out that Iraq had outdated versions of Soviet tanks and planes.

The reality was not as simple as people think. Even though Iraq had no chance against the forces of the West there are still reasons to study these conflicts in more detail.

Here are two interesting articles by academic Daryl G. Press:

Lessons From Ground Combat in the Gulf: The Impact of Training and Technology

The Myth of Airpower in the Persian Gulf War and the Future of Airpower

The Iraqi Perspectives Project has produced several very interesting studies of Saddam Hussein’s Iraq based on captured archival material and interviews of important personalities:

Saddam's War: An Iraqi Military Perspective of the Iran-Iraq War

Um Al-Ma’arik (The Mother of All Battles): Operational and Strategic Insights from an Iraqi Perspective

Iraqi Perspectives Project: a view of Operation Iraqi Freedom from Saddam's senior leadership

The German response against the Soviet T-34 and KV tanks

When the Germans invaded the Soviet Union in the summer of 1941 they were surprised to come across tanks that were superior to their own in armor and firepower. The T-34 and KV tanks had 76mm guns that could penetrate the armor of the Panzers even at long ranges while their sloped armor protected them from the German A/T and tank guns.

The appearance of these vehicles forced the Germans to take emergency measures. The immediate effort was concentrated in:

1). Introducing a new A/T gun

2). Up gunning and up armoring their Pz III, Pz IV and Stug vehicles

3). Using the chassis of old vehicles (like the Pz II and Pz 38) coupled with captured Soviet 76mm A/T guns or the new Pak 40 as mobile tank destroyers.

The long term solution was of course to design new tanks that could deal with the T-34 and KV. These were the Tiger and Panther vehicles.

Let’s take a look at the German effort:

Update

I made the following changes in Tank strength and losses – Eastern Front: Added ‘Waffen und Geheimwaffen des deutschen Heeres 1933 – 1945’ as a source and corrected the total numbers of German vehicles (my original numbers were slightly lower than they should be because I didn’t add special tanks like command, recovery and flak vehicles).

I added the following in SOE codes and Referat Vauck, paragraph ‘Key taken from a poem’:

Note that Marks doesn’t say anything about six consecutive letters. On the contrary in his book page 324 he says ‘every poem code message began with a five letter indicator group to show which five words of the poem had been used’.

SOE codes and Referat Vauck

In the 1930’s the British SIS (Secret Intelligence Service) collected information from European targets through two parallel systems. On the one hand regular SIS officers operated as passport control officers in the British consulates. This system gave them diplomatic protection but on the other hand foreign governments could easily identify them and keep them under observation.

As the diplomatic situation deteriorated a parallel system was created that would afford better security. This was the Z organization, created in 1936 and headed by Claude Dansey. The Z organization was supposed to operate independently of British embassies and thus avoid the attention of foreign internal security agencies.

At the start of WWII both networks were unable to perform as intended. As British embassies closed down, the PCO’s lost their networks. The undercover Z organization on the other hand had been compromised from double agents and British intelligence suffered a grievous defeat in the Venlo incident.

Since both groups had neglected to build up ‘stay behind’ networks and supply them with the necessary radio equipment this meant that Britain had practically no reliable intelligence networks available after the fall of France. In this void the need for extreme measures led to the creation of the SOE (Special Operations Executive) organization in 1940. SOE was responsible for intelligence and sabotage operations against the Axis powers but since it had the same mission as SIS countless power struggles ensued between these two agencies.

The wartime performance of SOE was mixed at best. Although they certainly had their successes, countless SOE networks were compromised and their members arrested and executed. In Holland their entire network fell under German control in the famous Englandspiel operation.  In France they lost countless agents and networks. Just the fall of their Prosper network in 1943 led to the arrest of hundreds of resistance members.

SOE was disbanded in 1946 and most of its archives were destroyed postwar with some lost in a fire. Unfortunately the loss of the archives means that many questions about SOE wartime operations can never be answered.

Were some of the failures of SOE in Western Europe connected with their insecure cryptosystems? Leo Marks, head of the SOE cipher section, was constantly worried about the insecurity of their poem code but it took him till late 1943 to introduce the unbreakable letter one time pad. The change was gradual and even in 1944 many insecure systems continued to be used.

Let’s have a look at this whole affair.

SOE cryptosystems

WWII intelligence services had two conflicting requirements when it came to cryptologic systems for their agents.

On the one hand they needed systems that would be easy to use in the field (so that ruled out like complicated/bulky systems like cipher machines).

On the other hand these messages had to be kept secure from enemy codebreakers, since each one contained information that could compromise their entire networks.

Unlike military messages that are usually unimportant on their own the traffic of a spy group contains names, addresses and other sensitive information that can be used by the enemy to untangle the entire group.

The only system that satisfied both requirements was the one time pad system and it was introduced gradually in late 1943. However for most of the war SOE used systems that were both insecure and prone to errors by the user.

Let’s take a look at them:

1). Playfair square

The first crypto system used by SOE was the well known Playfair cipher.

The security afforded by this system was very low and from 1942 it was restricted to internal network communications and its use prohibited for messages sent by radio.

2). Columnar transposition

The main system used by SOE for most of the war was the transposition of the text based on a numerical key.

Simple transposition

First the text is written underneath the key. Then each column is written vertically in the order specified by the key. This results in a ‘scrambled’ text.

For example let’s say our message is the following ‘Gestapo has arrested our radio operator cipher material compromised’ and the ‘key’ is ‘automobile’:

First we write the key and number each letter according to their position in the alphabet. If the same letter is present more than once we number them starting from the one to the left.

1	10	9	7	6	8	2	4	5	3
a	u	t	o	m	o	b	i	l	e

Then we write the plain text below the key:

1	10	9	7	6	8	2	4	5	3
g	e	s	t	a	p	o	h	a	s
a	r	r	e	s	t	e	d	o	u
r	r	a	d	i	o	o	p	e	r
a	t	o	r	c	i	p	h	e	r
m	a	t	e	r	i	a	l	c	o
m	p	r	o	m	i	s	e	d

Now we use the numerical key to rearrange the columns and copy the output:

garammoeopassurrohdphleaoeecdasicrmtedreoptoiiisraotrerrtap

Double transposition

The same procedure is then repeated one more time but with a different key of a different length. For example let’s assume the second ‘key’ is ‘elephant’:

2	5	3	7	4	1	6	8
e	l	e	p	h	a	n	t

2	5	3	7	4	1	6	8
g	a	r	a	m	m	o	e
o	p	a	s	s	u	r	r
o	h	d	p	h	l	e	a
o	e	e	c	d	a	s	i
c	r	m	t	e	d	r	e
o	p	t	o	i	i	i	s
r	a	o	t	r	e	r	r
t	a	p

The scrambled text becomes

muladiegooocortrademtopmshdeirapherpaaoresriraspctoteraiesr. Then the text is broken up in 5-letter groups and null letters are inserted to make the total divisible by 5.

Each message had to contain at least 100 letters and no more than 400-500.

The security of the transposition system depended on the use of different keys for each message. How were these keys selected?

Key taken from a book

In the early years the transposition keys were taken from a book. Both the agent and the receiving station had the same edition of a specific book and the indicator at the start of the message specified the page number, the line and the number of letters to be used for the two tables. Since the indicator had to be sent in letter groups a number to letter conversion table was used to turn the page numbers etc into letters. Before converting the numbers however the agent had to encipher this group by adding (without carrying) his own secret identification number.

This whole operation was time consuming and prone to errors. Moreover the use of a book as a key generator was found to be impractical in the field. Instead a poem or verse was used to create transposition keys.

Key taken from a poem

Each agent had to memorize a specific poem and could then use it to create different transposition keys for each message. After writing down the poem each word was assigned a letter of the alphabet. According to Lorain the user then had to choose at random 6 consecutive letters.

Let’s say our poem is ‘Mary Had a Little Lamb’:

MARY	HAD	A	LITTLE	LAMB	WHOSE	FLEECE	WAS
A	B	C	D	E	F	G	H
WHITE	AS	SNOW	AND	EVERYWHERE	THAT	MARY	WENT
I	J	K	L	M	N	O	P
THE	LAMB	WAS	SURE	TO	GO	IT	FOLLOWED
Q	R	S	T	U	V	W	X
HER	TO
Y	Z

Let’s assume that the letters chosen are PQRSTU, the odd letters furnish the first ‘key’ and the even letters the second. In our example PRT points to ‘WENT LAMB SURE’ as the first ‘key’. For the second we use QSU so it’s ‘THE WAS TO’. The indicator showing which words were used as ‘keys’ will be PRT filled with two nulls so as to form a 5-letter group (all messages were sent in 5-letter groups), so let’s say PARNT and the final step is to move all the letters forward by using the agents’ secret number. For instance if the number was 45711 then in our example PARNT will change into TFYOU, as each letter moves forward as many positions as indicated by the secret number P+4=T, A+5=F, R+7=Y, N+1=O, T+1=U.

Note that Marks doesn’t say anything about six consecutive letters. On the contrary in his book page 324 he says ‘every poem code message began with a five letter indicator group to show which five words of the poem had been used’.

This system was preferred by agents because they did not have to carry a book around. However if the agent was captured and tortured he might reveal his poem to the Germans with the result that they would be able to decode all his messages.

The problem of indecipherables

As can be seen in the aforementioned examples the slightest mistake in numbering the key or enciphering the plaintext will result in an indecipherable message. This was the biggest problem with the double transposition system and as a result a large percentage of the messages received at SOE HQ were unreadable. This forced HQ to request another transmission of the same message, with the following problems for the agents:

1). Forcing an agent to resend the message led to loss of time. If the information was time sensitive then obviously there was danger of it becoming useless.

2). The Germans monitored radio traffic in the occupied areas  and used direction finding equipment in order to locate the sites of illegal transmissions. The longer an agent stayed ‘on the air’ the easier it was for the Germans to triangulate his position.

3). Sending the same message enciphered with different keys was dangerous from a security point because it could provide enemy cryptanalysts with a way to solve it.

4). SOE agents were taught a series of secret signs that could be inserted in their messages in order to warn HQ that they had been captured and were under German control. Usually these were spelling mistakes at a prearranged point. However the huge number of indecipherables completely negated the value of this security system since messages had so many mistakes that it was not possible to know If they were a result of operator error or a deliberate attempt to warn HQ!

In order to deal with indecipherables a codebreaking department was created in the SOE cipher section and was tasked with solving the incoming messages.

WOK’s (Worked-Out Keys)

The use of a poem as a source of keys was found to be cumbersome and prone to errors and was replaced with a new system called the ‘A-Z system’ by Lorain and ‘WOK’ by Leo Marks.

Instead of choosing the transposition keys from a poem the agent was given a silk handkerchief with prepared keys. Each key had its own discriminant. Once the key was used then it was cut off and destroyed.

This system guaranteed that even if the agent was captured he would not be able to reveal the key to his captors since he did not have to memorize it. It also minimized operator errors.

3). Delastelle system

The cipher of Felix Marie Delastelle (1840–1902) is mentioned by Pierre Lorain but not by Leo Marks. According to Lorain it was a transitional system used in 1942-43.

4). LOP’s - (Letter One time Pads)

The epitome of the spy field cipher was the letter one time pad. This was adopted thanks to the efforts of Leo Marks and was gradually introduced in late 1943. The system used a substitution table together with a set of prepared ‘keys’. Each letter of the ‘key’ was ‘coupled’ with the opposite letter of the plaintext and they were substituted using the conversion table.

‘Between Silk and Cyanide: The Story of SOE's Code War’, p248 has an example of a conversion table:

For example if we want to encode the message ‘Jacques has arrived safely’ using the ‘key’  aqgtfdpxwmvxtdndixvhydk then the cipher text will be ooleifdvmqwckwxfuewygtb as aj=o, qa=o, gc=l etc

The OTP system is mathematically unbreakable provided the key is as long as the message and each key is only used once. The security of the system was such that messages could be as small as 10 letters.

However the OTP has the problem of distribution of keys, as both the sending and the receiving party need to have the same keys.

German exploitation of SOE codes

The German agencies responsible for monitoring illicit radio transmissions were the Radio Defence Corps of the Armed Forces High Command – OKW Funkabwehr and the similar department of the regular police – Ordnungspolizei. Both agencies operated in Western Europe but they were assigned different areas. 

These agencies not only monitored the agents’ traffic but in many cases they were able to locate the site of transmissions through D/F (direction finding). In such cases the radio center was raided and often the operator and his cipher material were captured.

This cipher material was then used by Dr Vaucks agents section to identify the crypto-systems, solve them and decode the traffic. This section, headed by Dr Wilhelm Vauck, was originally part of the Army’s signal intelligence agency OKH/In 7/VI but worked closely with the Radio Defense Corps. It was established in 1942 and by the end of the year two-man teams were detached to regional Aussenstellen in Paris, Marseilles, Lyons, Prague, Oslo, Vienna, Brussels. In late 1943 the entire department was moved to the OKW Funkabwehr.

According to postwar reports they usually had success with a system if it had been physically compromised. However in some cases it was possible to solve enemy systems cryptanalytically. Mettig, head of the Army’s signal intelligence agency in 1941-43 says in TICOM I-115 that

‘a special weakness of Allied agents’ ciphers was the use of books for enciphering. Usually only a minor inroad or other clue was required to reproduce a piece of the cipher text and conclusions could thence be drawn as to which book was used. In the case of one Allied transmission in the summer of ’42, five or six French words of a text were ascertained, leading to the conclusion that the cipher book dealt with the Spanish civil war. In view of this assumption, all French books about the Spanish civil war in the State libraries of Paris, Madrid and Lisbon were read with the object of trying in these 5-6 words. The book was found. PW always looked on a great research effort as worthwhile. The greatest weakness in using books for enciphering lay in the fact that, once a book had been compromised, an entire transmission could be broken automatically. The weakness existed even if the book in question could not be secured in the same edition or impression. It was still possible for Referat Vauck (though again only after considerable research) to find the right place in the book and to secure a fluent deciphering system by means of conversion tables.

Another weakness of Allied agent ciphers was the use of poetry. Here the verse metre was an additional help in solving the cipher text, as was done in the case of a Czech transmission in the autumn of 42/43.’

Notice that Mettig mentions in his report the use of poems and books as key generators. As we have seen these were indeed the main SOE systems (and probably SIS too).

How successful was the German effort vs SOE codes?

Unfortunately it is impossible to answer this question conclusively since I have not seen any TICOM reports giving details on the work of the Vauck section. Nor does it seem that Dr Vauck was interrogated by TICOM authorities after the war.

The Germans certainly decoded some messages as can be seen in file HW 40/76 ‘Enemy exploitation of SIS and SOE codes and cyphers: miscellaneous reports and correspondence’:

Fenner, head of the cryptanalysis department of OKW/Chi, said in DF-187F, p20 about the Vauck section that ‘there may have been some 50 messages decrypted weekly, among them some to be sure which were almost a year old and hence had only historical significance’. Fenner however was not the best source since he makes many ‘mistakes’ in his reports. TICOM report DF-9 ‘Captured Wehrmacht Sigint Document: Translation of Activity Report of OKW/Chi for the Period 1st January, 1944 to 25th June, 1944’, p4 gives the messages decoded by month and says in the end ‘The 6.000 agents messages handed to Fu III are not included in these figures.’

Hans Kurfess, a member of the Agents section detailed to the Paris Aussenstelle says in report CSDIC/CMF/SD 80, p24 ‘KURFESS, whose attachment was more "normal" than that of LENTZ and who consequently has a clearer idea of the sort of traffic that came through the Aussenstelle, states that most of the deciphered messages were short (40-50 groups) and used a double transposition cipher with a key phrase consisting of a line of poetry. They nearly all concerned the resistance movement in FRANCE, giving times of rendezvous, parachute dropping of supplies and WT sets. He remembers the code names "LYSANDER" and "EIFFEL" but cannot state in exactly what connection, and also one message of about 250 groups giving military information. He has forgotten for whom it was intended.’

There is also this information from ‘The German Penetration of SOE: France, 1941-44’ by Jean Overton Fuller:

One day `Archambaud' was all on edge, and to my question, 'What is the matter?' he replied. "Mr Goetz has given me, in clear, the text of a radiophonic message I received from London several weeks before my arrest. He had received the deciphered text of the message from Berlin. Now that was a message I had never been able to decipher myself, as London had committed a fault in the ciphering. Well, in Berlin they had deciphered it, and so it is from the Germans that I learn what it contained."I know that the central department in Berlin recorded almost all the enemy radiophonic messages from France and elsewhere, and that every time we arrested a radio operator Kieffer immediately asked Berlin to send, still ciphered or deciphered, the texts of the messages which he had sent to and received from London. For a long time after that `Archambaud' racked his brains as to how Berlin had been able to decipher his messages.

This passage seems to support the Abbe Guillaume's belief that the arrival of the two Canadians by parachute in the Sologne was known to the Germans through their having broken Archambaud's code, while he was still at liberty. Germaine Tambour, two days before her arrest, had told Laure Lebras the Germans seemed to know of parachutings at the same time as the Resistance and she believed they had the code. Professor Foot wrote that he had seen no evidence causing him to believe the Germans ever broke the code of an operator still at liberty, but Professor Foot had not the benefit of having seen Vogt's letter to me about this. That they asked the agents to give their codes may seem evidence against their ability to break them, but I suspect it may have been a question of time. From Vogt's letter, it appears to me that sometimes they could and sometimes they could not break the code.

Perhaps the British know more about what really happened since the first page in HW 40/76 says:

So I guess we’ll have to wait…

Sources:

‘Secret Warfare: The Arms and Techniques of the Resistance’ by Pierre Lorain, ‘Between Silk and Cyanide: The Story of SOE's Code War’ by Leo Marks, ‘Secret War: The Story of SOE, Britain's Wartime Sabotage Organization’, ‘MI6: The History of the Secret Intelligence Service 1909-1949’, ‘European Axis Signals Intelligence’ vol3 and 4, TICOM reports I-115, I-200, DF-187B, DF-187F, DF-9 , HW 40/76 ‘Enemy exploitation of SIS and SOE codes and cyphers’, ‘The German Penetration of SOE: France, 1941-44’, CSDIC/CMF/SD 80 - 'First Detailed Interrogation Report on LENTZ, Waldemar, and KURFESS, Hans', CSDIC (UK) SIR 1106 ‘Report on information obtained from PW CS/495 Uffz MIERSEMANN’, S.O.E. FIELD CIPHERS
Acknowledgements: Once again I have to thank Ralph Erskine for helping me identify the SOE cryptosystems.

Updates:

1). More information on SOE indicator procedures is available in SOE cryptosystems – The German view.

2). An overview of the work of Referat 12 is available from Allied agents codes and Referat 12.