New Worlds: Ciphers and Steganography

(This post is part of my Patreon-supported New Worlds series.)

Sometimes the things you most need to protect aren’t objects, people, or places: they’re pieces of information.

There are three primary ways to do this. The first is to hide the physical medium of the message itself, the paper or whatever it’s written on. Remember the hidden compartments we mentioned before? These come into play again here, especially because the hiding spot doesn’t have to be large. Curling a slip of paper inside a pen is a common notion, but my favorite method is something I didn’t believe was possible until I saw it done: hiding a message inside a raw egg. You soak the egg in vinegar until its shell becomes pliable, then cut a small slit and push in your paper (using iron gall ink to write with, so it won’t run), then soak the egg in clean water until it hardens again. Hey presto, a basket full of ordinary-looking eggs, and who would think to crack them open to find a note inside?

But if you want to keep something secure, your real go-to techniques will usually fall into the other two categories: steganography and cryptography.

Steganography, aka “hidden writing,” isn’t about keeping someone’s eyes from falling on the message; it’s about keeping them from realizing that what they’re looking at is a message. These days we have highly technological ways of doing that with digital media, but it goes back a good deal further than that. The classic here is invisible ink, which can’t be seen until you apply the right effect. If you ever wrote a hidden message with lemon juice, then used heat to reveal it, you’ve done steganography! Or you can use the separate ingredients of the aforementioned iron gall ink: write with an extract of oak gall, then apply vitriol of iron (iron sulfate) to reveal it. Nor is this purely an old-fashioned thing or a childish trick; “security markers” can be used to label your belongings, with the writing only becoming visible under UV light.

But that’s only one way to hide a message. Morse code lends itself very nicely to steganography, though often in forms that aren’t usually considered writing per se; a pattern of blinking or a design embroidered into a garment can actually be text. Or you can distribute the elements of your message throughout another text, either with a null cipher or by marking the key bits in some other fashion, e.g. by pricking the paper, using a subtly different color or typeface, and so forth. Anything that attempts to conceal the existence of the message even when someone looks right at it falls under this header.

Which makes it different from cryptography, aka (and here we’re about to fall down a technical rabbit hole) codes and ciphers — which are not the same thing, though we often use the words interchangeably. Passphrases, e.g. to get into a speakeasy or cue rebels to set off the bombs, are extremely simple one-time codes; others amount almost to made-up languages, with a vocabulary of words, phrases, or numerical sequences that indicate particular places, people, items, or actions.

One of the strengths of such a thing is that it can be very difficult to break without getting hold of a codebook, the dictionary that translates all the terms. One of the weaknesses of such a thing is that if your enemy gets hold of a codebook, you’re screwed. You can also break a code by building up a large enough corpus of examples and looking at their context; from that you may be able to suss out the meaning of particular elements. In this sense the decipherment of codes may not be much different from the decipherment of dead languages; the cracking of both Egyptian hieroglyphics and Linear B used many of the same techniques to identify the proper names of people and places, thereby opening up the relevant phonetic systems.

The decipherment of Linear B also used a great deal of math — and that brings us around to ciphers, whose making and breaking can both involve way more math than the layperson might expect. (Unless they’ve paid attention to WWII history and the work done at Bletchley Park.)

Historically speaking, most ciphers were of one of two basic types. A substitution cipher replaces letters or groups of letters with other letters or groups of letters. Rot 13, or “Rotate 13,” is an example most commonly used nowadays for hiding spoilers. By contrast — or sometimes in addition — a transposition cipher changes the order of elements within the plaintext, using a variety of methods.

Simple forms of these are fairly easy to crack. A cipher like Rot13 can be solved by the brute-force method of trying out different substitutions until you get a readable message. Even more complex forms are vulnerable to frequency analysis: if you know what letters or letter combinations are common in the source language, you can count the frequencies of letters and letter combinations in the ciphertext and try to match the most common ones. Transposition ciphers tend to require the use of a key: either a physical one like a grille or a one-time pad, or instructions for how to rearrange the letters to return the plaintext. The former can be stolen and the possibilities for the latter are often finite, which means that analysis can again break the cipher.

It’s possible to get more complex . . . but the problem with that is, the more complex the cipher, the more likely it is that somebody’s going to screw it up, rendering the message unreadable on the far end. Nowadays we rely on computers to do our ciphering for us, which allows for systems that no human analysis can hope to break — but another computer might be able to. There’s a reason cryptography is a major field of study in the modern world.

The Patreon logo and the text "This post is brought to you by my imaginative backers at Patreon. To join their ranks, click here!"



About Marie Brennan

Marie Brennan is a former anthropologist and folklorist who shamelessly pillages her academic fields for inspiration. She recently misapplied her professors' hard work to the short novel Driftwood and Turning Darkness Into Light, a sequel to the Hugo Award-nominated Victorian adventure series The Memoirs of Lady Trent. She is the author of several other series, over sixty short stories, and the New Worlds series of worldbuilding guides; as half of M.A. Carrick, she has written The Mask of Mirrors, first in the Rook and Rose trilogy. For more information, visit, Twitter @swan_tower, or her Patreon.


New Worlds: Ciphers and Steganography — 16 Comments

  1. Pingback: New Worlds: Ciphers and Steganography - Swan Tower

  2. I took a cryptology [1] course in grad school out of the math department and let me tell you, one of my least favorite assignments was decrypting a bunch of texts encrypted in Vigenère *by hand*. It’s doable (the short version is that it’s a frequency analysis attack) but time-consuming and really, really annoying.

    [1] cryptography = writing things in code, decryption = getting the plaintext back out, cryptology = the whole shebang

    The other thing I have been trying to find more information on is how non-alphabetic languages handed ciphers. The only abstract I’ve ever found on the web (in English) on Chinese crypto, for instance, suggests that they relied on codes because ciphers for logograms was…not going to be a thing. But I’d love to find out more.

    • My first thought is to assign numeric codes to the characters, then encrypt the numbers. This can work particularly well when the numbers end up getting dispersed. E.g. if a character is 6328, after transposition those digits can end up nowhere near each other in the ciphertext, making a lot of simple cryptanalysis difficult.

      (E.g. in English, if you turn ‘e’ into ’32’, the 3 and 2 can end up separated, and letter frequency analysis becomes much sadder.)

      • That’s probably possible . . . but how feasible is it, when doing everything by hand? It strikes me as the kind of complexity that’s extremely prone to both encryption and decryption error.

        • I dunno. But one stackexchange answer says that Chinese telegraph operators used a codebook with 4 digit codes per character, so that mapping seems practically feasible. Then you can use a standard hand-based keyed transposition cipher on the digit stream, that should be as feasible as doing it in Europe.

          A decent pre-modern cipher uses substitution and transposition, and mapping the characters to numbers already gives you substitution. Or more specifically, fractionation and thus diffusion.

          Another answer claims grille ciphers work well for Chinese.

          There’s using a phonetic script like Nushu, though that’s ducking the character issue. But could give you a setting where secure communication requires women, because they’re the ones who came up with phonetic scripts like Nushu or hiragana. (Though hiragana is pretty easy to learn, unlike the 700 characters of Nushu…)

          • These are called “straddling” ciphers, and they’re at least eight hundred years old (and hinted at in alchemical works older than that). The nastiest admitted-to-in-public variant — used by, among others, Richard Sorge, “Gordon Lonsdale,” and “Rudolf Abel” — uses differing-length encoding for letters (some one digit, some two) combined with a relatively simple transposition, buried inside a one-time-key (or, in Sorge’s case, a fairly secure book-code superencipherment).

  3. I saw a report in the Guardian that chinese researchers had developed a new ‘invisible ink’/specially coated paper combination that can be read under UV and then erased with heat and the coated paper reused for another invisible message.
    No more writing with lemon juice!

  4. The gold standard for civilians learning the context of secret writing is

    David Kahn, The Codebreakers. 1968 (but 1994 updated edition preferred). Almost certainly available through your public library.

    Warning: It’s a doorstop that will suck your attention away from anything else you may be doing. Forever. (And you will no longer have any respect for DVD encryption.)

    * * *

    The key problem with all ciphers (pun intended) is not the ciphering system itself, but key distribution and security. That’s what all of the “public key” systems (like PGP… and the DES variants that encrypt all financial transactions) try to evade by making the “key” a very-hard-to-solve function rather than a string of digits. Else, in order to securely send a message, both the sender and recipient also have to exchange (and protect and verify) a key that is as long as the message itself… else the message is subject to the Kasiski* analysis Yoon Ha Lee describes above for breaking the supposedly-unbreakable Vigenère cipher.

    * Reputedly a distant relative of mine on my mother’s side of the family.

    • Oh, yeah, my degree’s in math so the way mathematicians talk about crypto is so ivory tower compared to practical concerns like key distribution issues! I remember being so excited when we learned that the one-time pad was “mathematically impossible to break” and then the deflating news that…but you have to distribute the key and that’s the catch.

      And then I remember reading Ross Anderson’s Security Engineering and being horrified to learn that you can make decryption attacks by examining the activity on computer chips. To say nothing of that one paper co-authored by Bruce Schneier–where was it, here it is, “Cryptanalytic Attacks on Pseudorandom Number Generators” by John Kelsey et al. That was both terrifying and fun to read. XD

      I did take an abstract/applied algebra class that led up to public-key encryption and ended with Prof. Lou Billera handing out photocopies of the RSA paper. If it hadn’t been for a flood, I’d still have my copy, for nostalgia. (I mean, I redownloaded it.) That was fun. 🙂

    • “Else, in order to securely send a message, both the sender and recipient also have to exchange (and protect and verify) a key that is as long as the message itself”

      That’s for provably perfect security via one-time pad. AFAIK other symmetric ciphers like AES are considered practically secure, with much shorter keys.

      • History demonstrates that every “practically secure” cipher… isn’t. Both the Japanese Purple and German Enigma (variants on the same fundamental system) were considered “practically secure.” And things have not actually gotten better in that sense with more-powerful mathematical tools (both theoretically and computationally). The less said about the “practical security” associated with financial-transaction software, the better.

  5. Another alternative that can be quite hard to break is the Codetalkers used in WWII. As I understand it, the Codetalkers were native speakers of languages unrelated (in grammar, vocabulary and phonetics) to the languages of any of the major participants in the war. And the speakers were discussing things that had no standard vocabulary terms in the languages in use, so the Codetalkers were using private dialects/slangs that even ‘outsider’ native speakers of their languages would not be able to follow. Something similar could be done in writing. (I wonder if Navaho can be transcribed in cuneiform…)

    Also, ‘The Family Vault’ by Charlotte MacLeod includes a confession embroidered in Braille in white French knots on white linens…

    • My understanding of how Navajo was used in WWII was that they essentially spelled out military terms using the Navajo translations of words that in English began with the relevant letters — so that “bomb,” for example, would be conveyed with the Navajo words for bear, owl, mouse, bear. But that’s me working of a recollection of what my Navajo professor said about fifteen years ago, so . . . not positive I have it right.

      Regardless, though, they had to keep it simple, because anything complex risked too much misunderstanding. What kept it secure was what keeps any code (not cipher) secure: the unavailability of the codebook to the enemy. In this case, the “codebook” would be a Navajo dictionary, which was highly unlikely to exist anywhere in Japan at the time.

      Also, ‘The Family Vault’ by Charlotte MacLeod includes a confession embroidered in Braille in white French knots on white linens…

      Steganography at its best! And I suppose a cipher, too, insofar as a writing system like Braille can be seen as a symbol replacement for more widely-known letterforms.

      • At least according to the archival material I’ve reviewed, that summary of what your Navajo professor said a few years ago isn’t that closely connected to what was actually going on with the “technical terms” vocabulary. Of course, they were Official Files, so (a) I can’t disclose exactly what they do say (except that it wasn’t related to English-language spelling) and (b) the “story” that was in there was… let us say not recorded by individuals with either respect for the culture or experience in reporting on differing cultures.

        But at least nobody is going into Sherlock Holmes. (The most prominent “cipher” in the Holmes canon, in “The Dancing Men,” isn’t a soluble cipher, and Holmes’s “solution” is wrong.)