Character Encoding Explained

Text is stored as bytes, and an encoding is the rule that turns one into the other. This page walks through ASCII, Unicode, UTF-8 and the bugs a wrong guess causes.

Bytes are not characters

Files, network packets and strings in memory are all sequences of bytes, and a byte is just a number from 0 to 255. Nothing inside the byte says whether it stands for a letter, part of a picture, or a price. A character encoding is the agreed rule that maps characters to byte values and back again.

When the program that writes text and the program that reads it use the same encoding, everything looks fine. When they disagree, you get garbled text, question marks, or little boxes.

Keep two ideas apart. A character is an abstract thing, such as "the letter e with an acute accent". An encoded form is the concrete list of bytes used to store it. One character can take one byte in one encoding and several bytes in another.

ASCII: the 7-bit starting point

ASCII uses 7 bits, so it defines 128 codes, numbered 0 to 127. Codes 0 to 31 and code 127 are control codes that were meant to drive printers and terminals. Codes 32 to 126 are printable: the space, punctuation, digits, and the English letters.

GroupDecimalHex
Space320x20
Digits48 to 570x30 to 0x39
Uppercase letters65 to 900x41 to 0x5A
Lowercase letters97 to 1220x61 to 0x7A
Tilde (last printable)1260x7E

A few control codes still matter every day. Line endings are the main source of confusion: Unix-style files end each line with LF alone, while Windows-style files use CR followed by LF.

NameShort formDecimalHexEscape in code
Horizontal tabTAB90x09'\t'
Line feedLF100x0A'\n'
Carriage returnCR130x0D'\r'
DeleteDEL1270x7F'\x7f'

ASCII covers English and nothing else. For decades, many systems filled the unused upper half of the byte (128 to 255) with their own extra letters. Latin-1 (ISO-8859-1) is one such 8-bit set.

Unicode code points

Unicode gives every character a number called a code point. Code points are written as U+ followed by at least four hex digits, such as U+0041 for "A". If hex digits are new to you, the page on number bases explains them. The first 128 code points match ASCII exactly, so plain English text has the same numbers in both.

Unicode itself is a catalogue, not a byte format. To store code points you still need an encoding form. The common ones are UTF-8, UTF-16 and UTF-32.

UTF-8: variable length

UTF-8 stores each code point in one to four bytes. ASCII characters use a single byte with the same value as in ASCII, so a pure ASCII file is already valid UTF-8. Characters further up the range use two, three or four bytes. The leading bits of each byte tell a decoder how long the sequence is, so a reader can always find where one character starts.

The table below was produced by encoding each character in Python and printing the bytes in hex.

CharacterNameCode pointUTF-8 bytesByte countUTF-16 (big-endian)
ALatin capital AU+004141100 41
ée with acute accentU+00E9C3 A9200 E9
€Euro signU+20ACE2 82 AC320 AC
😀Grinning face emojiU+1F600F0 9F 98 804D8 3D DE 00

Notice that string length and byte length are different numbers. The word café has 4 characters but takes 5 bytes in UTF-8, because the accented letter needs two. Code that slices strings by byte count can cut a character in half and produce invalid data.

UTF-16 in brief

UTF-16 uses 16-bit units. Most common characters fit in one unit (two bytes). Code points above U+FFFF, such as most emoji, need two units called a surrogate pair. The emoji in the table becomes D8 3D DE 00. Some languages and platforms store strings internally as UTF-16, so a "length" function there may report 2 for that single emoji.

Mojibake: what a wrong guess looks like

Mojibake is the name for text decoded with the wrong encoding. Here is a real example run in Python: the string is encoded as UTF-8, then decoded as Latin-1.

>>> "café".encode("utf-8")
b'caf\xc3\xa9'
>>> "café".encode("utf-8").decode("latin-1")
'café'

The two bytes C3 A9 that form "é" in UTF-8 are read as two separate Latin-1 characters, "Ã" and "©". When you see pairs like é or è in text, UTF-8 data was almost certainly shown as Latin-1 or a similar single-byte set. Fix it by decoding correctly where the bytes enter, not by patching the odd characters later.

The byte order mark

The byte order mark (BOM) is the code point U+FEFF placed at the very start of a file. In UTF-16 it tells the reader the byte order: FE FF means big-endian and FF FE means little-endian. In UTF-8 it encodes as EF BB BF. UTF-8 has no byte order, so there the BOM only acts as a signature.

A UTF-8 BOM often causes trouble. A shell script with a BOM may fail because the first line no longer starts with #!. A CSV header may gain an invisible character on its first column name. Unless a program you depend on requires it, save UTF-8 files without a BOM.

Habits that prevent encoding bugs

  • Use UTF-8 for files, databases, APIs and source code unless something forces a different choice.
  • Decode bytes to text once, at the edge of your program, and encode once on the way out. Work with text in the middle.
  • Always name the encoding when you open a file, for example open(path, encoding="utf-8") in Python, instead of trusting the system default.
  • Declare it in HTTP headers: Content-Type: text/html; charset=utf-8.
  • When text arrives in a URL, remember it is first encoded to UTF-8 bytes and then percent-escaped; see URL encoding.
  • Test with accented letters, a currency sign and an emoji to expose most mistakes quickly.