Why Two Identical-Looking Strings Differ: Unicode Normalization

Where the problem comes from

An accented character has two encodings: a precomposed code point (a single code point for é), or a base letter plus a combining mark (e followed by a combining accent). They render identically but have different bytes and length, so a naive equality check returns false. That is what Unicode normalization fixes.

Four normalization forms

Use compatibility forms with care: NFKC replaces full-width characters, ligatures (fi) and circled numbers (①) with compatible forms. Great for search, but storing it loses the original text.

Code samples

// JavaScript: normalize before comparing
const equal = 'a'.normalize('NFC') + 'é'.normalize('NFC') === 'é'.normalize('NFC')

// Length differs (emoji and combining marks span multiple code points)
[...'é'].length   // 1 under NFC, 2 under NFD

// Python
import unicodedata
unicodedata.normalize('NFC', s)

Four practical tips

1. Normalize to NFC before storing so one text has one shape in the database;
2. Normalize before comparing, especially usernames, emails and tags;
3. Fix the form before hashing, or the same text yields different digests and dedup breaks;
4. Don't assume case-folding rules: Turkish casing differs from English, so pass a locale when sorting.

Two more traps

• Truncation and length limits: emoji, combining marks and flags span several code points; slicing by code point can cut a glyph in half, so slice by grapheme clusters;
• Order of sanitization: normalize before a blocklist check, or an attacker can bypass keyword filters with an equivalent spelling.

Try it

Hash after normalization: Hashing, JSON formatter.

What NFC and NFD actually break

A character can have several equivalent representations, and different normalisation makes "looks identical" compare unequal:

1. Duplicate usernames: the same name typed through different input methods may arrive as NFC or NFD, so without normalising on write a "unique" username registers twice;
2. Filenames and paths: macOS historically normalises filenames to NFD, so syncing across platforms can yield two copies of one file;
3. Search and sort: skipping normalisation misses visually identical characters and produces ordering that ignores language intuition;
4. Length validation: strings with combining marks have fewer characters than code points, so code-point limits reject valid input.

Three rules to apply

• Normalise at the boundary: convert all external input to NFC as it enters the system and keep it consistent in the database;
• Normalise before comparing: any field used for comparison, lookup or deduplication should be normalised first, with case folding where relevant;
• Store both: keep the user's original for display and a normalised copy for indexes and unique constraints.

Applying it in databases and search

1. Put the unique constraint on the normalised column: for usernames and emails, add a column holding the normalised value with the unique constraint, or correct normalisation still cannot stop duplicates.
2. Normalise both sides before matching: normalise the query string and the stored value, or one-sided handling misses matches.
3. Pair with a collation: collations differ on punctuation and case, so pick one deliberately for international business and fix it in the database. Normalisation handles equivalence; collation handles order — neither replaces the other.
4. Watch index size: normalised strings can change length, so size indexes on the maximum normalised length to avoid truncation and false matches.
5. Migrate existing data: introducing normalisation needs a one-off backfill, and duplicates must be resolved during it or the unique constraint cannot be created.

Normalisation is cheap early and expensive late — setting the rules while the system is small costs far less than migrating and de-duplicating later.