Recover garbled text from the original file bytes

Problem

Mojibake appears when one encoding's bytes are interpreted through a different character map. UTF-8 bytes for ü can be displayed as ü when read as Windows-1252, and Korean text can become several Latin-looking fragments after the same mistake. Once that broken display is saved, the application may encode the wrong characters again, creating a second transformation and sometimes losing information through replacement characters. BOM handling is a separate issue: a hidden U+FEFF in the first CSV field can break a header even when the rest of the text decodes correctly. Unicode normalization is separate again; NFC and NFD can compare differently after correct decoding, but normalization cannot repair wrong byte interpretation. Recovery must begin with the earliest untouched bytes, use byte evidence and source metadata to choose decoder candidates, verify known names or headers, and distinguish reversible mojibake from data that has already been replaced or discarded.

Sources and standards

These authoritative references define the formats or security boundaries used in this workflow. Tool-specific verification is documented separately above.

• Encoding Standard

  WHATWG

When to use this

• Text looked correct in the source system but became ü, ’, mojibake fragments, boxes, or replacement characters after export or import.
• A multilingual CSV has correct delimiters but broken names, cities, comments, or headers.
• The first column behaves differently from later columns because a BOM may have become part of its name.
• A copied broken string exists, but you need to determine whether the untouched file still contains recoverable source bytes.
• A repair script must be documented and tested before it changes a full customer or production dataset.

Steps

1. Step 1

   Stop resaving the broken preview

   Close the editor or spreadsheet without saving. Preserve the original export and calculate a hash if the incident matters operationally. Work on copies so each decoder trial starts from identical bytes.

2. Step 2

   Locate the first wrong boundary

   Identify where bytes became text: export, HTTP response, file reader, spreadsheet import, database client, or terminal. Record the charset declared or assumed at that boundary.

3. Step 3

   Inspect the original byte evidence

   Run the original file through the character set detector. Record the BOM, strict UTF-8 status, NUL pattern, control-byte warning, leading hex, and whether the result is confirmed, strong, heuristic, compatible, or undetermined.

4. Step 4

   Choose candidates from evidence and provenance

   Prefer an exact BOM or valid non-ASCII UTF-8. For undetermined legacy text, use the producing application, locale, operating system, protocol header, and vendor documentation to choose a short candidate list.

5. Step 5

   Compare known-text previews

   Decode a copy with each candidate and compare expected headers, names, punctuation, CJK, emoji, and delimiters. Reject any preview containing unexpected replacement characters or implausible control characters.

6. Step 6

   Handle BOM and normalization separately

   Remove a BOM only when the destination rejects it or treats it as data. Apply NFC or another normalization form only after decoding is correct and the destination contract requires it.

7. Step 7

   Convert once and verify the destination

   Write a new file with an explicit target encoding, normally UTF-8 when the destination supports it. Reinspect its bytes, run a small import, compare counts and key values, and retain the original for rollback.

Example

Visible broken value: Jürgen
Original bytes around the name: 4A C3 BC 72 67 65 6E
Detector: strict UTF-8, no BOM, strong evidence

Decode the original bytes as UTF-8 -> Jürgen
Do not encode the visible Jürgen text as a new source
Verify the corrected name and surrounding CSV delimiters in a sample import

Common mistakes

FAQ