SimpleWebUtils
ENKO

Document Workflow

Normalize Unicode before comparing text across systems

Choose NFC, NFD, NFKC, or NFKD deliberately, inspect code-point and byte changes, separate encoding loss from normalization, and validate the destination.

Written and tested by Published: Reviewed:

How this workflow was checked

The review loaded the unmodified “Normalize a multilingual lookup key without compatibility folding” sample into Unicode Normalizer. We exercised the path from “Preserve the original representation” to “Investigate every warning”, checked the displayed or downloaded result against the example, and separately examined “Using NFKC as universal cleanup” and “Assuming hidden characters are removed”.

NFC produced “Café / 각” as canonically equivalent text, reported no compatibility fold, and left unrelated full-width or mathematical characters distinct.

Open Unicode Normalizer | Compare NFC, NFD, NFKC and NFKD

Problem

Two strings can render alike while using different code-point sequences, so binary equality, filenames, cache keys, imports, routes, or deduplication may fail. Blind normalization can create a second problem: NFKC and NFKD may erase meaningful width, mathematical, presentation, superscript, or circled-character distinctions. Normalization also cannot recover bytes decoded with the wrong character set, remove every zero-width character, prevent script-confusable identifiers, or guarantee that separately normalized fragments remain normalized after concatenation. The safe workflow identifies the actual layer of failure, chooses a form from the destination contract, records exact changes, and tests the completed result.

Sources and standards

These authoritative references define the formats or security boundaries used in this workflow. Tool-specific verification is documented separately above.

When to use this

  • Visually identical values fail equality, lookup, route, cache, or duplicate checks.
  • Filenames move between macOS, Windows, Linux, an archive, or version control.
  • Multilingual labels with accents, combining marks, or Hangul jamo enter a system with a documented normalization rule.
  • A restricted search key may intentionally fold fullwidth forms, ligatures, circled values, or presentation variants.
  • A migration needs evidence of code-point and UTF-8 changes before stored values are rewritten.

Steps

  1. Step 1

    Preserve the original representation

    Keep the source text and, for imported files, the original bytes, filename, declared encoding, and decode path. A U+FFFD replacement character or mojibake pattern indicates earlier decoding loss that normalization cannot reverse. Diagnose the character set before rewriting text.

  2. Step 2

    Identify the destination contract

    Check whether the database, filesystem, protocol, programming language, signature, identifier policy, or search index requires a specific form. Do not change a signed value, checksum input, security token, or externally specified identifier just because NFC is common on the Web.

  3. Step 3

    Choose canonical or compatibility behavior

    Use NFC when canonically equivalent sequences should be composed, or NFD when they should remain decomposed. Choose NFKC or NFKD only when the application intentionally treats width variants, ligatures, circled characters, superscripts, presentation forms, and related compatibility values as equivalent.

  4. Step 4

    Run a bounded sample first

    Test representative positive and negative examples in Unicode Normalizer. Include accents, Hangul, emoji, combining marks, expected width variants, line endings, and any identifiers from the real workflow. Review input and output code-point counts, UTF-8 bytes, and the bounded previews.

  5. Step 5

    Investigate every warning

    A compatibility-folding warning means the result is not merely another canonically equivalent encoding. Remaining format characters need separate inspection, and U+FFFD means the original bytes should be decoded again if possible. None of these warnings should be hidden by a successful run.

  6. Step 6

    Compare exact output and normalize completed values

    Use Diff Checker and code-point evidence to account for every changed sequence. If the value is assembled from fragments, concatenate first and normalize the completed value because normalized strings are not guaranteed to stay normalized across a new boundary.

  7. Step 7

    Validate storage, lookup, and round trip

    Test the reviewed result in a disposable copy of the real database, import pipeline, filename operation, route matcher, or search index. Confirm expected equality and distinctness, re-read stored bytes, and retain the chosen form and test evidence without retaining sensitive text unnecessarily.

Example

Normalize a multilingual lookup key without compatibility folding

Input

Source code points: C a f e U+0301 / U+1100 U+1161 U+11A8
Target contract: NFC
Hidden-character count: 0
Keep the original import bytes and compare the completed key.

Output

Result: Café / 각
Canonical equivalent: yes
Compatibility folded: no
Then test equality against the destination index and verify that unrelated fullwidth or mathematical forms remain distinct.

Common mistakes

Using NFKC as universal cleanup

Compatibility normalization can remove distinctions that matter in mathematics, identifiers, typography, legacy round trips, and user-visible content. Apply it only to a restricted field with an explicit equivalence policy.

Confusing decoding with normalization

Wrong character-set decoding changes or loses information before normalization starts. Preserve and re-decode the bytes instead of repeatedly normalizing mojibake or U+FFFD.

Assuming hidden characters are removed

Unicode normalization normally preserves zero-width and other format characters. Inspect their count and use a dedicated removal workflow only after deciding which characters are invalid.

Normalizing each fragment only once

A new concatenation boundary can form a composeable or reorderable sequence. Normalize the complete value that will actually be compared or stored.

Treating canonical equivalence as application equality

An application may use locale collation, case folding, security profiles, or exact binary identity in addition to normalization. Validate the target behavior rather than inferring it from one normalization result.

FAQ

Why can identical-looking strings have different bytes?

Unicode can encode some abstract characters as a precomposed code point or as a base character plus combining marks. Correct rendering may look the same even though binary comparison differs.

Which form is safest for ordinary Web text?

NFC is widely recommended for interoperable Web content, but the destination contract still controls. Preserve source evidence and do not apply a default to signatures, identifiers, or protocol fields without checking their rules.

When is NFKC appropriate?

It can be appropriate for a restricted search or identifier profile that explicitly wants compatibility variants to compare alike. It is not appropriate as an automatic rewrite of arbitrary prose or mathematically meaningful content.

Will normalization remove spoofing risks?

No. Script-confusable characters can remain after normalization. Security-sensitive identifiers need an appropriate Unicode security profile, script policy, and application-specific review.

Why normalize after concatenation?

Unicode normalization forms are not closed under string concatenation. Characters at the new boundary may compose or combining marks may reorder, so the final value needs normalization.

Can this workflow prove a migration is safe?

It provides local transformation and exact evidence, but safety also depends on the destination's comparison, indexing, uniqueness, signing, and round-trip behavior. Test those systems with a reversible migration plan.