SimpleWebUtils
ENKO

Document Workflow

Deduplicate lines from lists and logs without losing order

Remove repeated IDs, URLs, labels, or log messages with explicit case and whitespace rules, then verify first-occurrence order, counts, and line endings before export.

Written and tested by Published: Reviewed:

How this workflow was checked

To check “Preserve first-seen issue IDs”, we used Remove Duplicate Lines with the guide's exact source data and applied “Choose the case rule”. The output had to match the documented result; evidence for “Using case-insensitive matching for opaque IDs” and “Expecting trim comparison to clean the kept row” was reviewed before recording the check.

Case-insensitive comparison reduced six issue-ID rows to four in first-seen order and reported two removals without sorting the surviving values.

Open Remove Duplicate Lines | Keep First Occurrences

Problem

Merged exports and copied logs often contain exact repeats, case variants, invisible surrounding spaces, and mixed line endings. Blind deduplication can collapse distinct identifiers, rewrite spacing unexpectedly, or destroy chronology by sorting too early.

When to use this

  • A spreadsheet column or export contains repeated IDs, email addresses, labels, or URLs.
  • Several allowlist, denylist, hostname, or configuration fragments must be combined into one unique list.
  • A log excerpt repeats identical messages and first-seen sequence still matters.
  • Human-edited rows may differ only by case or surrounding spaces and need an explicit comparison policy.
  • You need before-and-after counts and an exact text download rather than a hidden spreadsheet operation.

Steps

  1. Step 1

    Preserve the source block

    Keep an untouched copy and note whether order, capitalization, indentation, and the final line ending matter to the destination.

  2. Step 2

    Remove unrelated blank rows separately

    If blank rows are noise, clean them first with the empty-line tool. Duplicate removal treats a blank row as a real comparison value and keeps one copy.

  3. Step 3

    Choose the case rule

    Use exact case for machine identifiers when AbC1 and abc1 can differ. Leave it off for labels where capitalization is only presentation.

  4. Step 4

    Choose the whitespace rule

    Enable surrounding-whitespace comparison only when leading and trailing spaces are accidental. The first kept line is not trimmed in the output.

  5. Step 5

    Keep source order for logs

    Leave sorting off when chronology, priority, or first-seen order carries meaning. The tool retains the first occurrence of every comparison key.

  6. Step 6

    Run and inspect the diagnostics

    Compare source, unique, removed, and duplicate-group counts. Read the comparison-collision warning and any mixed-line-ending normalization notice.

  7. Step 7

    Verify and export

    Spot-check values that differ by case, spaces, or Unicode composition, then copy or download the exact result and test it in the destination system.

Example

Preserve first-seen issue IDs

Input

BUG-104
bug-104
BUG-208
BUG-315
BUG-208
BUG-401

Output

Case-insensitive, sorting off:
BUG-104
BUG-208
BUG-315
BUG-401

Source: 6 | Unique: 4 | Removed: 2

Common mistakes

Using case-insensitive matching for opaque IDs

Some databases, tokens, paths, and external systems distinguish case. Confirm the destination rule before merging variants.

Expecting trim comparison to clean the kept row

Trim changes only the key used to find duplicates. Run a separate whitespace cleanup when the output itself must be trimmed.

Sorting a chronological log

Natural sorting makes lists easier to scan but removes first-seen chronology. Keep sorting off during incident analysis.

Assuming visual Unicode equality

Composed and decomposed characters can look the same but remain distinct because duplicate comparison does not normalize Unicode.

Ignoring mixed line endings

Mixed LF, CRLF, and CR are normalized to a predominant style. Check the notice before a byte-sensitive import or patch.

FAQ

Which duplicate does the tool keep?

It keeps the first original line for each comparison key. With sorting off, those kept lines remain in their first-seen order.

Are whitespace-only lines duplicates?

Exact whitespace-only rows are duplicates. With trimmed comparison on, all surrounding-whitespace variants share an empty comparison key and one original row remains.

Should logs ever be sorted after deduplication?

Only when message sequence no longer matters. Keep the ordered result for debugging and make a separate sorted copy for inventory or frequency review.

Will case-insensitive matching handle every language identically?

It uses JavaScript lowercase conversion, not application-specific collation. Test special casing rules against the target system before import.

Does the browser upload my list?

The selected file is read locally and deduplication runs in a bounded Worker. Analytics receives counts and option flags, not line content.

How large can the input be?

The candidate accepts at most 1 MiB of UTF-8 text and 200,000 source lines, with a three-second Worker deadline. Split larger datasets in a proper data-processing environment.