Inspect zero-width characters before removing them from text

Problem

A zero-width code point can make an email address, account key, source identifier, URL label, or copied paragraph differ from what a reviewer sees. Some occurrences are accidental residues from editors, exports, or copy-and-paste, while others are intentional: ZWJ can form one emoji or affect complex-script shaping, ZWNJ can prevent joining, ZERO WIDTH SPACE can add a break opportunity, and WORD JOINER can suppress one. Deleting every invisible character can therefore fix a comparison while corrupting language, rendering, or line layout. A defensible cleanup keeps the original, identifies exact code points and positions, removes only a confirmed set, preserves unrelated bytes in the decoded string, and validates the destination behavior separately from normalization or encoding repair.

When to use this

• Two visually identical usernames, email addresses, search terms, or database keys compare as different values.
• Pasted source code, CSV data, environment values, or identifiers fail parsing or validation without a visible cause.
• Copied text wraps at an unexpected position or refuses to wrap where the visible spacing suggests it should.
• An emoji or Arabic, Persian, Indic, or Mongolian string contains a joining control that must be reviewed rather than deleted blindly.
• A suspicious string needs code-point evidence for human triage without treating a hidden control as proof of malicious intent.

Steps

1. Step 1

   Preserve the exact source and provenance

   Keep the original file, clipboard value, database field, or message unchanged. Record where it came from, how it was decoded, and the comparison or rendering symptom. A screenshot alone is not a source because it cannot preserve invisible code points.

2. Step 2

   Separate text inspection from byte encoding

   Decide whether the evidence is a decoded string or an original file. A leading U+FEFF in browser text can be reported, but only the original bytes can prove a UTF-8, UTF-16, or UTF-32 signature. Keep byte-level BOM work in a separate step.

3. Step 3

   Run an inspection without removal

   Choose Inspect only first. Review total detections, affected lines, code-point types, one-based positions, line and column values, and escaped context. A zero result means none of the six supported controls were found, not that the text has no invisible Unicode characters.

4. Step 4

   Classify each control by its possible effect

   Treat U+200B and U+FEFF as common cleanup candidates, then examine the source before changing them. Protect ZWNJ, ZWJ, WORD JOINER, and MVS unless the destination contract proves they are unwanted; they can affect shaping, emoji sequences, line breaking, or Mongolian text.

5. Step 5

   Remove the narrowest confirmed set

   Use Common cleanup for U+200B and U+FEFF, or Choose types for a smaller or explicitly reviewed set. Use All supported only when the source owner and destination rules allow the resulting changes to joining and wrapping.

6. Step 6

   Compare exact output and change counts

   Confirm detected equals removed plus preserved, inspect the first changed contexts, and compare the output with the untouched source. Verify LF, CRLF, CR, tabs, spacing, terminal line endings, emoji, and multilingual shaping before running normalization, trimming, or other cleanup.

7. Step 7

   Validate the destination and record the decision

   Test a non-sensitive copy in the database, identity provider, parser, search index, editor, URL handler, or publishing system that will consume it. Record which code points were removed and why, then retain the original when auditability or rollback matters.

Example

account​@example.com
Emoji: 👩‍💻
legacymark

account@example.com
Emoji: 👩‍💻
legacymark

Common mistakes

FAQ