The essence of mojibake: bytes read with the wrong rule
A computer only has bytes. A character encoding is the byte sequence ↔ character mapping. Mojibake is rarely lost data — it is the same bytes interpreted by a different rule: text written as UTF-8 and read as GBK shows entirely different characters.
Three concepts you must separate
| Concept | What it is | Common confusion |
|---|---|---|
| Character set (Unicode) | Assigns a number (code point) to each character | It is not itself an encoding |
| Encoding (UTF-8 / UTF-16) | The rule turning code points into bytes | "Save as Unicode" is imprecise |
| Byte order and BOM | The order of multi-byte units | UTF-8 does not need a BOM |
Why UTF-8 became the default
- ASCII compatible: English stays 1 byte, so old systems still read it;
- Variable length saves space: common CJK characters take 3 bytes, rare ones 4;
- No byte-order problem: no BOM required, consistent across platforms.
Common symptoms and their causes
| Symptom | Typical cause |
|---|---|
| Text turns into unrelated characters | UTF-8 bytes read as GBK |
| The classic "replacement chain" garbage | UTF-8 decoded as GBK then re-encoded as UTF-8 |
| Question marks or the diamond character | Target encoding cannot represent the character |
| Chinese in a URL shown as %E4%B8%AD | Normal percent-encoding, not an error |
A four-step investigation
- Confirm the source encoding: what the file or API declares;
- Confirm the reader's encoding: editor, database and terminal settings;
- Check the transport layer: database connection charset, HTTP Content-Type charset;
- Standardise on UTF-8 end to end and declare it explicitly at every boundary — never rely on defaults.
Three things not to do
- Converting repeatedly "just to try": every round trip can drop characters — determine the original encoding and convert once;
- Shipping BOM-prefixed files to scripts: the BOM becomes part of the content and breaks parsing;
- Using utf8 instead of utf8mb4 in MySQL: its utf8 supports only 3 bytes, so emoji and some CJK characters fail to store.
Common questions
Is UTF-8 always better than GBK? For new systems, essentially yes — but do not bulk-convert legacy GBK data: back it up and validate database by database. Does Base64 fix mojibake? No. It only turns bytes into printable characters; the encoding problem remains. Its purpose is moving binary safely through text-only channels.
A quick self-check
Encode the suspect text as UTF-8 and look at the hex bytes: common CJK characters are three-byte sequences starting with E (for example "中" is E4 B8 AD). A run of bytes starting with C0 or 80 usually means the text has been mis-converted. Once you know the source encoding, convert once from source → UTF-8; never convert back and forth.
Try it: Base64 encoder/decoder — see how text maps to bytes
A fixed routine for locating mojibake
- Read the shape first: runs of question marks mean characters were replaced, so something does not support the target charset; fixed sequences are the signature of a wrong conversion applied twice; single characters becoming several odd symbols usually means mismatched encode and decode.
- Confirm the declared encoding at every hop: database connection, application read, template rendering and response headers should each state the same encoding rather than relying on defaults.
- Check for double conversion: bytes interpreted under two different charsets produce results that look unrecoverable. When you see that, go back to the original bytes.
- Compare hex dumps: printing the byte sequence of broken and expected text shows immediately whether multibyte characters were split or re-encoded.
- Fix every entry point: repairing the display leaves imports, exports and APIs producing the same corruption, so normalise at the system boundary.
Normalise and validate encoding on input rather than papering over it in the view — the latter hides the problem and lets dirty data accumulate.