To turn bytes into a string in Python: data.decode("utf-8"). To go the other way: text.encode("utf-8"). The confusion is not the syntax but the concept — bytes are raw numbers, strings are characters, and an encoding is the rulebook that maps between them. Get the rulebook wrong and you either get mojibake or a UnicodeDecodeError.
The two types#
text = "café" # str - characters
data = b"cafe" # bytes - raw numbers
print(type(text)) # <class 'str'>
print(type(data)) # <class 'bytes'>
print(len("café")) # 4 characters
print(len("café".encode("utf-8"))) # 5 bytes
That last pair is the whole idea in two lines. The word has four characters, but the accented e needs two bytes in UTF-8, so the byte count differs from the character count.
Converting#
text = "café"
data = text.encode("utf-8") # str -> bytes
print(data) # b'caf\xc3\xa9'
back = data.decode("utf-8") # bytes -> str
print(back) # café
A mnemonic that sticks: you encode text into bytes for storing or sending, and decode bytes back into text for reading.
UTF-8 is the default for both, so these are equivalent:
text.encode()
text.encode("utf-8")
Being explicit is still worth it. It documents the assumption, and the assumption is the thing that breaks.
UnicodeDecodeError#
data = "café".encode("utf-8")
data.decode("ascii")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3:
ordinal not in range(128)
The message tells you three useful things: the encoding it tried, the byte that failed, and where. Position 3 is the accented character; byte 0xc3 is the first half of its UTF-8 representation, which ASCII has no meaning for.
Almost always the fix is to use the right encoding:
data.decode("utf-8") # correct
When you do not know the encoding#
Try the likely candidates in order:
def try_decode(data, encodings=("utf-8", "utf-8-sig", "cp1252", "latin-1")):
for encoding in encodings:
try:
return data.decode(encoding), encoding
except UnicodeDecodeError:
continue
return data.decode("utf-8", errors="replace"), "utf-8 (with replacements)"
text, used = try_decode(raw_bytes)
print("decoded as", used)
Note that latin-1 never fails — every byte value is a valid character in it. That makes it useful as a last resort and dangerous as a default, because it turns a clear error into silently wrong text. Keep it last in the list.
For a real guess, use a detection library:
python -m pip install charset-normalizer
from charset_normalizer import from_bytes
result = from_bytes(raw_bytes).best()
print(result.encoding)
print(str(result)[:200])
The errors argument#
data = b"caf\xe9" # latin-1 bytes, decoded as utf-8
data.decode("utf-8", errors="strict") # raises - the default
data.decode("utf-8", errors="replace") # 'caf\ufffd' - a placeholder
data.decode("utf-8", errors="ignore") # 'caf' - the byte is dropped
data.decode("utf-8", errors="backslashreplace") # 'caf\\xe9' - visible
Files#
# Text mode - decoding happens for you
with open("data.txt", encoding="utf-8") as f:
text = f.read()
# Binary mode - you get bytes and decode yourself
with open("data.txt", "rb") as f:
data = f.read()
text = data.decode("utf-8")
The byte order mark#
data = b"\xef\xbb\xbfHello"
print(data.decode("utf-8")) # '\ufeffHello' - invisible junk at the front
print(data.decode("utf-8-sig")) # 'Hello' - BOM removed
Files saved by Excel and some Windows editors start with a three-byte marker. It is invisible when printed but breaks comparisons and CSV headers. Use utf-8-sig when reading files that may have one — it handles files with and without it.
Encodings worth knowing#
| Encoding | Covers | Where you meet it |
|---|---|---|
utf-8 |
every character | the web, modern files, JSON |
utf-8-sig |
as above, strips the BOM | Excel exports |
ascii |
128 characters | old protocols, identifiers |
latin-1 |
256 characters | older European files |
cp1252 |
Windows Western | legacy Windows exports |
utf-16 |
every character | some Windows APIs |
Network and subprocess output#
import requests
response = requests.get("https://example.com")
print(type(response.content)) # bytes - raw
print(type(response.text)) # str - decoded for you
import subprocess
result = subprocess.run(["ls"], capture_output=True)
print(result.stdout.decode("utf-8", errors="replace"))
# Or let subprocess decode
result = subprocess.run(["ls"], capture_output=True, text=True, encoding="utf-8")
print(result.stdout)
Bytes behave like strings, mostly#
data = b"hello world"
print(data.upper()) # b'HELLO WORLD'
print(data.split(b" ")) # [b'hello', b'world']
print(data[0]) # 104 - an int, not b'h'
print(data[0:1]) # b'h' - slicing gives bytes
b"hello" + "world" # TypeError - cannot mix
Indexing a bytes object gives you an integer, while slicing gives you bytes. That asymmetry surprises people the first time.
Questions people ask#
What is the difference between Unicode and UTF-8?
Unicode assigns a number to every character. UTF-8 is one way of storing those numbers as bytes. A Python str holds Unicode characters; bytes holds an encoded form of them.
Why did Python 2 not have this problem?
It did — it just hid it, by allowing bytes and text to mix and converting implicitly. That produced errors far from their cause. Python 3 separates them deliberately, which is why the errors are noisier and much easier to fix.
How do I write a string to a file as bytes?
Open in binary mode and encode: open(path, "wb").write(text.encode("utf-8")). Or just use text mode with encoding="utf-8", which does it for you.
What does the b prefix mean?
It creates a bytes literal. b"abc" is three bytes; "abc" is three characters. Only ASCII characters can appear directly in a bytes literal.
Where to go next#
- Python file handling — where encoding matters most.
- Cleaning Python strings — Unicode categories in practice.
- Python string methods — once the bytes are text.
