Verify text limits across Unicode counting units

Problem

A label such as “maximum 30 characters” is incomplete unless the system defines character. A joined emoji may be one grapheme, several code points, more UTF-16 code units, and many UTF-8 bytes. Canonically equivalent text can also have the same appearance but different byte and code-point totals. If a client validates one unit while an API, database, or publisher enforces another, users see late rejection, damaged truncation, uniqueness surprises, or inconsistent counts. The workflow must establish the actual unit and test the system that owns the limit.

When to use this

• An API, database column, form, message, filename, or import has a documented length limit.
• A multilingual field contains accents, Hangul, CJK text, emoji, combining marks, or right-to-left content.
• Browser and server validation disagree about whether a value is too long.
• A migration, normalization rule, or encoding change can alter code-point or byte totals.
• A publisher specifies words, characters, lines, or reading time and the submission must be auditable.

Steps

1. Step 1

   Find the component that owns the limit

   Read the protocol, schema, database type, form contract, or publisher instructions. Record the unit, whether the bound is inclusive, the character encoding, normalization behavior, newline handling, and whether truncation or rejection occurs. A UI placeholder is not an authoritative contract.

2. Step 2

   Preserve exact source evidence

   Keep the original text and, for a file or network payload, the original bytes and declared encoding. Do not normalize, trim, replace line endings, or remove invisible characters before recording what the source actually contains.

3. Step 3

   Build representative boundary fixtures

   Include plain ASCII, precomposed and decomposed accents, Hangul, no-space CJK text, a joined emoji, combining marks, CRLF and LF, a trailing newline, and any allowed format characters. Add an intentionally damaged U+FFFD case so decoding loss cannot pass unnoticed.

4. Step 4

   Measure every plausible unit

   Run each fixture through Word Counter and record grapheme clusters, code points, UTF-16 code units, UTF-8 bytes, word-like segments, and logical lines. Use Unicode Normalizer only to compare forms required by the destination; do not silently replace the source evidence.

5. Step 5

   Investigate hidden and line-ending differences

   Review format-character and U+FFFD warnings. Use the line-ending evidence to explain CRLF, LF, CR, trailing empty lines, and paragraph boundaries. Remove a zero-width character only when the field policy says it is invalid, because joiners can be meaningful in emoji or scripts.

6. Step 6

   Exercise below, at, and above the boundary

   Submit fixtures whose authoritative unit is limit minus one, exactly the limit, and limit plus one. Test both direct API or database access and the normal client. Confirm that rejection is explicit and that no implementation slices through a surrogate pair, combining sequence, or joined emoji.

7. Step 7

   Re-read and document the result

   Retrieve the stored or transmitted value and compare exact text, bytes, normalization form, and line endings with the accepted fixture. Record the rule, fixtures, observed errors, and software versions in an automated regression test so a runtime or schema change cannot alter the contract silently.

Example

Contract: maximum 31 UTF-8 bytes, NFC on write, reject rather than truncate.
Fixture A: Café 👨‍👩‍👧‍👦 (NFC)
Fixture B: Cafe + U+0301 + space + family emoji (NFD representation)

Fixture A: 6 graphemes, 12 code points, 16 UTF-16 units, 31 UTF-8 bytes; accept.
Fixture B before normalization: 6 graphemes, 13 code points, 17 UTF-16 units, 32 UTF-8 bytes. Verify that documented NFC normalization occurs before the 31-byte check, then re-read the stored NFC value.

Common mistakes

FAQ