Text that survives: encodings, and why the accents turned into question marks

Two different things called text

A str in Python 3 is a sequence of characters. A bytes object is a sequence of numbers from 0 to 255. Files, networks and disks hold bytes. Your program holds characters. An encoding is the rule for converting between them.

"नमस्ते".encode("utf-8")      # b'\xe0\xa4\xa8\xe0\xa4\xae...'
b"hello".decode("utf-8")     # 'hello'

Every file you open involves that conversion, whether or not you name it. Naming it is the whole lesson.

Always pass encoding="utf-8"

open(path, encoding="utf-8")
path.read_text(encoding="utf-8")

Without it, Python uses the platform's default. On Linux and modern macOS that is UTF-8 and everything works. On Windows it has historically been the system code page — cp1252 in Western Europe, cp1251 in Russia, and so on — which cannot represent most of the world's writing systems.

The consequence is a script that works on the developer's Mac, is emailed to a colleague on Windows, and fails there with:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1247

or, worse, does not fail and writes नमसà¥ into the output. Passing encoding="utf-8" explicitly removes the entire class of problem, and it is eight words.

What mojibake actually is

नमस्ते displayed as नमसà¥à¤¤à¥‡ is text that was encoded as UTF-8 and then decoded as if it were Latin-1. The bytes are intact; the interpretation is wrong. That is good news — the data is usually recoverable:

broken.encode("latin-1").decode("utf-8")

If that produces sense, you have found the mistake. The free ftfy package automates this and handles the double-encoded cases.

The reverse case, where the bytes were genuinely replaced by ? on the way out, is not recoverable. That happens when text is encoded to something that cannot represent it with errors="replace", and it is why you should let an encode fail loudly rather than papering over it.

The spreadsheet problem

Excel on Windows opens a UTF-8 CSV as if it were the local code page unless the file begins with a byte order mark. So for a file a colleague will open in a spreadsheet:

path.write_text(csv_text, encoding="utf-8-sig")

utf-8-sig writes three extra bytes at the start. Reading a file that has them with plain utf-8 leaves a stray \ufeff glued to the first column name, which produces the mystifying KeyError: 'id' when the key is visibly id. Reading with encoding="utf-8-sig" strips the mark if present and does nothing if not, so it is the safer choice for reading anything that came from a spreadsheet.

len does not count what you see

len("café")          # 4
len("👍")            # 1
len("👨‍👩‍👧")        # 5
len("नमस्ते")         # 6

len counts code points, not what a reader would call characters. The family emoji is three people joined by two invisible joiner characters. नमस्ते is six code points forming four visible clusters, because vowel signs combine with the consonant before them.

This matters whenever you truncate. Slicing a string at 100 characters can cut a Devanagari syllable in half or split an emoji into its components, producing something that renders as nonsense. For display truncation on non-Latin text, count grapheme clusters with the free regex package (regex.findall(r"\X", s)) rather than slicing raw.

It also matters for token counting with language models, which is a third unit again — not characters, not code points, not words. Module seven takes that up.

The same text, two byte sequences

é can be one code point, or e followed by a combining accent. They look identical and compare as unequal:

import unicodedata
a = "café"
b = unicodedata.normalize("NFD", a)
a == b                                    # False
len(a), len(b)                            # 4, 5
unicodedata.normalize("NFC", b) == a      # True

macOS stores filenames in the decomposed form and Linux in the composed one, so a filename copied between them can fail to match. Normalise to NFC on the way in, once, whenever you compare or deduplicate user-supplied text.

When you genuinely do not know the encoding

raw = path.read_bytes()
raw[:200]                    # look at it

Then guess in order: UTF-8, then utf-8-sig, then cp1252, then the likely regional encoding. The free charset-normalizer package guesses statistically and is right most of the time, but it is a guess — encodings are not recorded in the file, so no tool can be certain. Ask the sender if you can. Record the answer next to the data, because the next person will need it too.