Thursday, May 1, 2014

The Japanese FUJI diplomatic cipher 1941-43

In order to protect its diplomatic communications Japan’s Foreign Ministry used several cryptologic systems during WWII. In 1939 the PURPLE cipher machine was introduced for the most important embassies, however not all stations had this equipment so hand systems continued to play an important role in the prewar period and during the war.

One of the main hand systems was the J-19 code, enciphered either with bigram substitution tables or with transposition using a stencil.
Historical overview

The fist Japanese diplomatic system identified by US codebreakers was introduced during WWI and it was a simple bigram code called ‘JA’. There were two code tables, one of vowel-consonant combinations and the other of consonant vowel. Similar systems, some with 4-letter code tables were introduced in the 1920’s.
These unenciphered codes were easy to solve simply by taking advantage of the repetitions of the codegroups of the most commonly used words and phrases. US codebreakers solved these codes and thus learned details of Japan’s foreign policy. During the Washington Naval Conference the codebreakers of Herbert Yardley’s Black Chamber  were able to solve the Japanese code and their success allowed the US diplomats to pressure the Japanese representatives to agree to a battleship ratio of 5-5-3 for USA-UK-Japan. However this success became public knowledge when in 1931 Yardley published ‘The American Black Chamber’, a summary of the codebreaking achievements of his group. The book became an international best seller and especially in Japan it led to the introduction of new, more secure cryptosystems.

In the 1930’s the Japanese Foreign Ministry upgraded the security of its communications by introducing the RED and PURPLE cipher machines and by enciphering their codes mainly with transposition systems.

J-19 FUJI code
The J-19 code (also listed as system JAE in US reports) had bigram and 4- letter code tables similar to the ones used previously by the Japanese Foreign Ministry. According to the NSA study ‘West Wind Clear: Cryptology and the Winds Message Controversy A Documentary History’ it was used from 21 June 1941 till 15 August 1943.

In terms of security the J-19 FUJI and the similar codes J-16 MATSU to J-18 SAKURA, that preceded it in the period 1940-41, were much more sophisticated than the older Japanese diplomatic systems. They had roughly double the number of code groups at ~1.600, these included 676 bigram entries and in addition there was a 4-letter table with 900 entries for ‘common foreign words, usually of a technical nature, proper names, geographic locations, months of the year, etc’. 
Example of the code table from West Wind Clear’:

Once the messages were encoded using the J-19 code table then they had to be enciphered. There were two cipher procedures used with the J-19, substitution and transposition.

CIFOL VEVAZ substitution
J-19 messages with the indicators CIFOL or VEVAZ were enciphered using bigram substitution tables. A random letter sequence was coupled with the coded text and each pair of letters was substituted using the substitution table. According to the NSA study this cipher procedure was rarely used.

Columnar transposition using a stencil
The basic cipher system used with the J-19 code was columnar transposition based on a numerical key, with a stencil being used for additional security. The presence of ‘blank’ cages in the box created irregular lengths for each column of the text. Three different stencils were used each month with each being valid for 10 days. The numerical key changed daily. There were four different settings for the J-19 system: General, Europe, America and Asia.

Examples of stencils and transposition keys from ‘West Wind Clear’:

The use of a stencil and daily changing transposition key was a big improvement over past systems and offered considerable security in that time period. However both Allied and Axis codebreakers would eventually solve this complicated system during WWII.

Importance of the J-19 FUJI code
The J-19 code was important enough for both the Allies and the Axis to devote significant resources in solving it, even going so far as to build special cryptanalytic equipment. The reason they went through all this trouble is that during the war only a small number of Japanese embassies had the PURPLE cipher machine so the rest had to rely on hand systems and one of the main diplomatic codes was the J-19 FUJI.  Intercepted diplomatic traffic from around the world on this system carried economic, political, military and secret intelligence information. 

A special case was the Moscow embassy (moved to Kuibyshev during the war) and their use of the J-19 for communications with Tokyo. It seems that this embassy was either not given a PURPLE machine or perhaps they had to dismantle it in 1941, so in 1941-43 they relied on the J-19 for their most important messages. During WWII Japan fought on the side of the Axis but was careful to avoid a confrontation with the Soviet Union. War between the SU and Japan finally broke out in August 1945 but during the period 1941-45 Japanese diplomats were free to collect and transmit important information from the SU on military and political developments as well as their discussions and negotiations with Soviet officials. These messages were a prime target for the Allied and German codebreakers.
Allied exploitation of the improved J series codes

When the new code J-16 MATSU was introduced in August 1940 it proved much harder to solve that the previous Japanese systems. A team of cryptanalysts of the Army’s Signal Intelligence Service, led by Frank Rowlett analyzed this traffic and came to the conclusion that it was a bigram code enciphered with a transposition key. Rowlett realized that this system was similar to the German ADFGVX cipher of WWI that they had already researched extensively as part of their training program. Their effort to solve this Japanese code had not progressed much when they were given copies of the code, the stencils and some of the numerical keys. These had been copied by US Naval Intelligence from a Japanese embassy or consulate in the US. Thanks to these specimens (J-16 codebooks, ten-day forms and auxiliary transposition keys) Rowlett’s work became much easier and messages could be read. Changes in the code could be followed by taking advantage of operator mistakes.
From the 1974 interview of Frank Rowlett, pages 25-28:

It is interesting to note that before they solved this code the head of the codebreaking department, William Friedman was very pessimistic about the prospects of success.
When the new J-19 system was introduced the US codebreakers were already familiar with the basic characteristics of the cipher and Rowlett quickly made important discoveries regarding the underlying code. However solution of the daily key settings was a difficult problem, especially since more resources were put into the solution of the traffic sent on the PURPLE cipher machine.

During the war traffic on this system increased significantly and the solution of the daily changing settings became a problem for the small group working on Japanese transposed codes, so there was an effort to automate the process. The device built was an attachment for standard IBM punch card equipment called the ‘Electromechanagrammer’ or ‘Gee-Whizzer’.

According to the NSA study ‘It Wasn’t All Magic: The Early Struggle to Automate Cryptanalysis, 1930s – 1960s’, p50-51:
‘The Gee Whizzer had been the first to arrive. In its initial version it did not look impressive; it was just a box containing relays and telephone system type rotary switches. But when it was wired to one of the tabulating machines, it caused amazement and pride. Although primitive and ugly, it worked and saved hundreds of hours of dreadful labor needed to penetrate an important diplomatic target. It proved so useful that a series of larger and more sophisticated "Whizzers" was constructed during the war……………….When the Japanese made one of their diplomatic "transposition" systems much more difficult to solve through hand anagramming (reshuffling columns of code until they made "sense"), the American army did not have the manpower needed to apply the traditional hand tests.

Friedman's response was to try to find a way to further automate what had become a standard approach to mechanically testing for meaningful decipherments……………………………………..Rosen and the IBM consultants realized that not much could be done about the cards; there was no other viable memory medium. But it was thought that it might be possible to eliminate all but significant results from being printed. Rosen and his men, with the permission and help of IBM, turned the idea into the first and very simple Gee Whizzer. The Whizzer's two six-point, twenty-five-position rotary switches signalled the tabulator when the old log values that were not approaching a criterion value should be dropped from its counters. Then they instructed the tabulator to start building up a new plain-language indicator value.
Simple, inexpensive, and quickly implemented, the Gee Whizzer reinforced the belief among the cryptoengineers in Washington that practical and evolutionary changes were the ones that should be given support.’

Australian effort
The American codebreakers were not the only ones who were regularly reading this system. Their British allies were also exploiting the J-19 and in Australia a small group called Diplomatic Special Section solved several Japanese diplomatic systems. 

In Australia the Diplomatic Special Section (D Special Section) of the Australian Military Forces HQ in Melbourne decrypted Japanese diplomatic ciphers. This unit was headed in the period 1942-44 by A.D. Trendall, Professor of Greek at Sydney University. Despite the small size of the unit considerable success was achieved in the solution of Japanese communications.

According to the book ‘Breaking Japanese Diplomatic Codes David Sissons and D Special Section during the Second World War’, p62 the J-19 was the main Japanese diplomatic code used in the period 1941-43.

Progress in 1941 was slow and up to February 1942 the only keys solved were those for messages whose content was known (for example messages reporting the departure of ships). However in 1942 things progressed rapidly.

In March ‘42 a member of the British Foreign Office from Singapore who possessed an excellent knowledge of Japanese joined the section. At the same time personnel of the unit developed elaborate cryptanalytic methods for recovering the daily settings and by May ‘42 the section was able to read virtually all FUJI traffic and ‘all bigrams, except those of very rare occurrence, and most tetragrams had been recovered’.

The solution of the traffic on the Kuibyshev–Tokyo link was one of their main commitments and in page 38 it says:

The report alludes, very briefly, to the high intelligence value of the intercepts of the telegrams exchanged between the Japanese Foreign Ministry and its Ambassador in Russia, Sato Naotake.

It seems that in its coverage of the Kuibyshev–Tokyo–Kuibyshev circuit, the Section was able to provide strategic intelligence of value.
German exploitation of J-19 system

Foreign diplomatic codes were worked on by three different German agencies, the German High Command’s deciphering department – OKW/Chi, the Foreign Ministry’s deciphering deparment Pers Z and the Air Ministry’s Research Department - Reichsluftfahrtministerium Forschungsamt.
Forschungsamt effort

At the Air Ministry’s Research Department Japanese systems were worked on by Abteilung 7 (USA, UK, Ireland, South America, Spain, Portugal, Turkey, Egypt, Far East). The department had about 60-70 workers. 

At this time there is very limited information available on the work of the Forschungsamt. EASI vol7, p82 says that they worked on ‘a transposition with nulls over two and four letter code’. This was clearly either the J-19 or one of the similar J series systems.
Pers Z effort

At the Foreign Ministry’s deciphering department Pers Z Japanese systems were worked on by a group headed by Senior Specialist dr Rudolf Schauffler. This section successfully solved the Japanese diplomatic transposed codes. 

According to paragraph 176 of TICOM report I-22 ‘Interrogation of German Cryptographers of Pers Z S Department of the Auswaertiges Amt’, FUJI was designated as Japanese Code 57 and it was read for about two years.

JB 57. Another Japanese two letter book with a recypherment consisting of stencil transposition with nulls, which was read for about two years. There was also a variant with substitution recypherment using a table of about 30 alphabets.

More information is available from the TICOM report DF-31B ‘How J.B. 57 Japanese Letter System Was Solved’, written by the cryptanalysts Annalise Huenke and Hans Rohrbach

The first break into system JB 57 came through two messages that had the same indicator (meaning they used the same transposition key). Once these were solved the system was identified as a transposed code, using a stencil.

Solution of this indicator led to the decipherment of more messages and dr Kunze (head of the ‘Mathematical Cryptanalytic Subsection’ of Pers Z) was able to use the information recovered in order to solve more message indicators. The inroads made by the solution of indicator groups led to the eventual recovery of the underlying code by the linguistic group and the current exploitation of this traffic.

The Germans were very interested in the messages from the embassy in the Soviet Union and according to David Kahn’s ‘The codebreakers’, p444:

the subsequent solutions provided the Germans with information about Russian war production and army activities
OKW/Chi effort

At the High Command’s deciphering department - OKW/Chi, Japanese diplomatic systems were worked on by a subsection of Referat 13, headed by 1st Lieutenant dr Adler. About 15 people were employed by the unit.

The OKW/Chi designation for FUJI was system J-13/J2B4BCüRuW (Japanese 2-letter and 4-letter code with stencil and transposition – Raster und Würfel). FUJI messages were first solved thanks to a repeat message sent from Paris to Tokyo. The first message and the repeat had the same plaintext (with small variations) and they had both been enciphered with the same key. This mistake facilitated their solution and the basic characteristics of the system were identified.

The solution of the daily transposition settings and the different stencils was taken over by personnel of the mathematical research department, specifically by the mathematician dr Werner Weber.

According to Part 3 of the report I-181 ‘Homework by Dr Werner Weber of OKW/Chi’, Weber started working on Japanese diplomatic messages in July ’41 and he identified the system as a transposed code. The underlying code for some of the messages was the previously solved LA code, thus they could be read. The rest of the messages had a new code.

Solution of the new system and recovery of the code proceeded slowly in 1941. In September ’41 Weber was allocated a small staff to help him with the Japanese traffic and by February ’42 some material could be read. During the year the new system was solved and most of the circular and European/Middle East traffic could be read. In the period summer ’42 to summer ’43 the previous year’s indicators were reused and the old transposition keys and stencils were either repeated or were modified in a predictable manner (with some exceptions).

At OKW/Chi they not only solved this code but also built a specialized cryptanalytic device called the ‘Bigram search device’ (bigramm suchgerät) for recovering the daily settings. EASI vol3, p65 says:

FUJI, a transposition by means of a transposition square with nulls applied to a two and four letter code. This system was read until it ended in August, 1943. It was broken in a very short time by the use of special apparatus designed by the research section and operated by Weber. New traffic could be read in less than two hours with the aid of this machine.
The ‘Bigram search device’ is called ‘digraph weight recorder’ in the US report ‘European Axis Signal Intelligence in World War II’ volume 2. In pages 51-53 details are given on the operation of this device:

The digraph "weight" recorder consisted of: two teleprinter tape reading heads, a relay-bank interpreter circuit, a plugboard ‘’weight’’ assignor and a recording pen and drum.
Each head read its tape photoelectrically, at a speed of 75 positions per second.

The machine could find a solution in less than two hours and did the work of 20 people, thus saving manpower.

Bigram apparatus versus Gee-Whizzer

How did the German machine compare with the American ‘Gee-Whizzer’? European Axis Signal Intelligence in World War II’ volume 2 points out the differences in their operation and the pros and cons of each device:
If the sections of text chosen from any given message were well chosen from a cryptanalytic viewpoint, the American machine proved much faster than the German machine, because the print of totals was easier for the cryptanalyst to analyze than just the individual values listed by the German machine; but if the sections of text were not well chosen and no true columns were included in the choice, then the German machine ,had the advantage since it recorded all possible juxtapositions quickly, and all true matches were included in the data.

Sources: ‘European Axis Signal Intelligence in World War II’ volumes 2,3,6,7 , TICOM reports I-22, I-31, I-25, I-37, I-90, I-118, I-124, I-150, I-181, DF-31B, DF-187B , ‘The Codebreakers’, ‘Breaking Japanese Diplomatic Codes David Sissons and D Special Section during the Second World War’, United States Cryptologic History Series IV: World War II Volume X: ‘West Wind Clear: Cryptology and the Winds Message Controversy A Documentary History’, United States Cryptologic History, Special Series, Volume 6, ‘It Wasn’t All Magic: The Early Struggle to Automate Cryptanalysis, 1930s – 1960s’, NSA interviews of Frank Rowlett 1974

Additional information: In summer 1943 J-19 FUJI was replaced by three new systems. The transposed codes TOKI and GEAM and the enciphered code ‘Cypher Book No1’.
TOKI and GEAM were used in the period 1943-45 and they were similar to J-19 in that they were codes transposed on a stencil.

Information on the TOKI cipher is available from The Japanese TOKI diplomatic cipher 1943-45.

1 comment: