Why Two Identical-Looking Strings Differ: Unicode Normalization

Where the problem comes from

An accented character has two encodings: a precomposed code point (a single code point for é), or a base letter plus a combining mark (e followed by a combining accent). They render identically but have different bytes and length, so a naive equality check returns false. That is what Unicode normalization fixes.

Four normalization forms

Use compatibility forms with care: NFKC replaces full-width characters, ligatures (fi) and circled numbers (①) with compatible forms. Great for search, but storing it loses the original text.

Code samples

// JavaScript: normalize before comparing
const equal = 'a'.normalize('NFC') + 'é'.normalize('NFC') === 'é'.normalize('NFC')

// Length differs (emoji and combining marks span multiple code points)
[...'é'].length   // 1 under NFC, 2 under NFD

// Python
import unicodedata
unicodedata.normalize('NFC', s)

Four practical tips

1. Normalize to NFC before storing so one text has one shape in the database;
2. Normalize before comparing, especially usernames, emails and tags;
3. Fix the form before hashing, or the same text yields different digests and dedup breaks;
4. Don't assume case-folding rules: Turkish casing differs from English, so pass a locale when sorting.

Two more traps

• Truncation and length limits: emoji, combining marks and flags span several code points; slicing by code point can cut a glyph in half, so slice by grapheme clusters;
• Order of sanitization: normalize before a blocklist check, or an attacker can bypass keyword filters with an equivalent spelling.

Try it

Hash after normalization: Hashing, JSON formatter.

What NFC and NFD actually break

A character can have several equivalent representations, and different normalisation makes "looks identical" compare unequal:

1. Duplicate usernames: the same name typed through different input methods may arrive as NFC or NFD, so without normalising on write a "unique" username registers twice;
2. Filenames and paths: macOS historically normalises filenames to NFD, so syncing across platforms can yield two copies of one file;
3. Search and sort: skipping normalisation misses visually identical characters and produces ordering that ignores language intuition;
4. Length validation: strings with combining marks have fewer characters than code points, so code-point limits reject valid input.

Three rules to apply

• Normalise at the boundary: convert all external input to NFC as it enters the system and keep it consistent in the database;
• Normalise before comparing: any field used for comparison, lookup or deduplication should be normalised first, with case folding where relevant;
• Store both: keep the user's original for display and a normalised copy for indexes and unique constraints.

Applying it in databases and search

1. Put the unique constraint on the normalised column: for usernames and emails, add a column holding the normalised value with the unique constraint, or correct normalisation still cannot stop duplicates.
2. Normalise both sides before matching: normalise the query string and the stored value, or one-sided handling misses matches.
3. Pair with a collation: collations differ on punctuation and case, so pick one deliberately for international business and fix it in the database. Normalisation handles equivalence; collation handles order, neither replaces the other.
4. Watch index size: normalised strings can change length, so size indexes on the maximum normalised length to avoid truncation and false matches.
5. Migrate existing data: introducing normalisation needs a one-off backfill, and duplicates must be resolved during it or the unique constraint cannot be created.

Normalisation is cheap early and expensive late, setting the rules while the system is small costs far less than migrating and de-duplicating later.