Detect a text file's encoding before import

Problem

An importer must turn bytes into characters before it can parse columns, delimiters, or records. If its decoder does not match the source, a visible header such as name can acquire a hidden U+FEFF, a city such as Zürich can become Zürich, and CJK text can turn into unrelated symbols. The extension .csv or .txt does not carry a reliable charset, and opening the file in an editor may silently decode and resave it before you inspect the evidence. Detection also has hard limits: a BOM is a strong signature but does not prove that every following code unit is valid; printable ASCII bytes are shared by UTF-8 and many legacy encodings; and similar single-byte charsets often cannot be distinguished without expected text or source metadata. A safe import workflow therefore preserves the original, records exact byte evidence, separates confirmation from heuristic classification, and tests a small destination import before modifying the full dataset.

Sources and standards

These authoritative references define the formats or security boundaries used in this workflow. Tool-specific verification is documented separately above.

• Encoding Standard

  WHATWG

When to use this

• A CSV header, customer name, accent, symbol, or CJK field changes after spreadsheet, database, or ETL import.
• A receiving system requires UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE, or a BOM-free stream.
• A source or translation repository must be audited before standardizing files on UTF-8.
• An attachment has an uncertain file type and must be separated into plausible text or likely binary data before decoding.
• You need a reproducible byte report for a vendor or data-pipeline incident rather than an editor's encoding label.

Steps

1. Step 1

   Preserve the untouched source and destination requirement

   Keep a byte-for-byte copy before opening or saving the file. Record the exact encoding names accepted by the importer, whether a BOM is required or forbidden, and one or more known values that a correct decode must reproduce.

2. Step 2

   Inspect exact bytes manually

   Select the original file in the character set detector and run the inspection. The file extension and MIME label are not evidence; the report starts with byte count and the leading 24-byte hex prefix.

3. Step 3

   Separate signatures from structure and heuristics

   Treat EF BB BF, FE FF, FF FE, 00 00 FE FF, or FF FE 00 00 at byte zero as confirmed BOM evidence. Read strict UTF-8 validation and UTF-16 or UTF-32 structure separately, because a complete signature can still precede malformed data.

4. Step 4

   Keep ambiguous results ambiguous

   ASCII-compatible means several encodings can produce the same bytes. A legacy or single-byte candidate means strict UTF-8 failed without strong binary evidence; it does not prove Windows-1252, ISO-8859-1, Shift_JIS, or another named charset.

5. Step 5

   Try controlled candidate decoders

   Start with a decoder supported by the byte evidence and source metadata. Compare known headers, accents, names, delimiters, and replacement characters. Do not overwrite the source while trying alternatives.

6. Step 6

   Run a small destination import

   Import a few representative rows containing ASCII, accents, CJK, emoji, empty fields, and quoted delimiters. Verify both visible text and the exact first field name before scaling the job.

7. Step 7

   Convert only after validation

   When the source decoder and destination contract are confirmed, convert once with an explicit output encoding. Reinspect the output bytes and keep the original plus the conversion record for rollback.

Example

customers.bin
Leading bytes: FF FE 00 00 69 00 00 00 64 00 00 00
Expected first field: id

Classification: UTF-32 LE
Evidence: confirmed four-byte BOM; valid UTF-32 structure
Do not classify as UTF-16LE from FF FE alone
Next step: test a UTF-32LE decoder against the id header

Common mistakes

FAQ