README.md — Hidden Character Scanner

# Hidden Character Scanner

Hidden Character Scanner is an offline, read-only Agent Skill and command-line tool for finding invisible Unicode, bidirectional text controls, hidden tag text, unusual whitespace, and mixed-script words in code, prompts, and skill packages. It reports exact code points and locations so a person can inspect the source safely. It never modifies a file. Version 0.1.1.

## Requirements

File scanning supports macOS and Linux with POSIX directory-descriptor and no-follow APIs; Windows file scanning is unsupported. Python 3.10 or newer and only its standard library are required. A scan containing any path other than `-` checks `os.name == "posix"`, `os.O_NOFOLLOW`, `os.O_DIRECTORY`, `os.open` and `os.stat` in `os.supports_dir_fd`, `os.stat` in `os.supports_follow_symlinks`, and `os.scandir` in `os.supports_fd` before reading any input. Missing capabilities produce a one-line error and status 1. Stdin-only `scan -` and `decode-tags TEXT|-` do not access file paths and remain available on Windows and other systems with Python 3.10+. The tool uses the standard library, needs no credentials or environment variables, and makes no network request. Supply paths with read access or pass text on standard input.

## Install and run

Copy this complete package into the skills directory supported by your agent harness, following that harness's installation instructions. Run commands from the directory containing `SKILL.md`:

```sh
python3 scripts/hidden_character_scanner.py scan ./src ./prompts
python3 scripts/hidden_character_scanner.py scan ./package --json
python3 scripts/hidden_character_scanner.py scan - < ./prompt.txt
python3 scripts/hidden_character_scanner.py decode-tags - < ./prompt.txt
python3 scripts/hidden_character_scanner.py --help
```

`scan` accepts one or more paths; `-` reads standard input. Directory traversal skips symlinks and common dependency, build, and repository directories. An explicitly supplied symlink or path with a symlinked parent is skipped and makes the scan incomplete; use a resolved path when needed. Files with NUL bytes or invalid UTF-8 are skipped as binary. Set `--max-file-bytes N` to limit each file, `--include-hidden` to include hidden entries outside the fixed skip directories, or `--allow-partial` to opt into partial results. `decode-tags TEXT|-` only reveals ASCII hidden in Unicode tag characters; it does not encode text.

`decode-tags -` accepts UTF-8 standard input up to 1 MiB (1,048,576 bytes), reads at most the limit plus one byte to detect overflow, and returns a one-line error and status 1 for oversized or invalid UTF-8 input. This fixed limit does not apply to a literal TEXT argument. `scan -` uses the configurable `--max-file-bytes` limit (1 MiB by default).

Findings include category, severity, Unicode code point and name, file, line, and column. An unterminated bidirectional embedding or isolate on a line is high severity `trojan-source`. Human-readable output escapes untrusted text and paths onto single lines. `--json` emits structured findings and a summary.

Other format characters (`Cf`) and unusual whitespace are reported only in code files. Zero-width characters, bidirectional controls, tag characters, and variation selectors outside expected emoji use are reported in every text file. The emoji-base allow-list for variation selectors is a fixed heuristic, not full Unicode emoji data.

Exit status is 0 when no findings are present, 1 for findings, usage errors, unsupported file-scanning capabilities, or invalid decoder stdin, and 2 when unreadable or oversized files or explicitly supplied symlink paths make a scan incomplete unless `--allow-partial` is supplied. Binary files and symlinks discovered during directory traversal are skipped without making it incomplete. Incomplete results list skipped inputs; a partial scan must not be reported as clean even when `--allow-partial` changes the exit status.

See [SETUP.md](SETUP.md) for invocation details and [references/findings.md](references/findings.md) for how to interpret categories. The [companion agent](agents/hidden-character-scanner-agent.md) can explain results, but it never changes source files automatically.

MIT licensed. Copyright 2026 Zero Width Lab.

By Zero Width Lab (zerowidthlab.com)

License: MIT (read it).

Back to the listing