SETUP.md — Duplicate File and Photo Report

# Setup

## Install

Use Python 3.10 or newer. The CLI needs only the standard library, and it
never uses the network. Copy the complete package to the location your agent
harness uses for skills. Run commands from the directory containing
`SKILL.md`; no package installation or configuration is necessary.

```sh
python3 scripts/duplicate_file_report.py --help
python3 scripts/duplicate_file_report.py scan "/path/to/folder"
```

No environment variables, credentials, or default scan roots are required.
Select every root explicitly. The scanner needs read access to those roots.
For a report it also needs permission to create a new file at the chosen
`--report` path.

## Select scope

`scan ROOT...` accepts one or more root paths. `--min-size BYTES` defaults to
1, so zero-byte files are skipped. Set it to 0 to include them. Hidden entries and
macOS package directories are skipped unless `--include-hidden` and
`--include-packages` are supplied. Symlinks and special files are never read.

```sh
python3 scripts/duplicate_file_report.py scan "/path/to/folder" --min-size 1024
python3 scripts/duplicate_file_report.py scan "/path/to/folder" --include-hidden --json
```

Use `--include-packages` only when the user wants inspection of package
contents. A Photos library is managed by its application; reading inside it
is not a supported way to deduplicate it.

## Optional JSON report

Choose the exact destination for a new report file. Preview the write and
record its approval digest:

```sh
python3 scripts/duplicate_file_report.py scan "/path/to/folder" --report "/path/to/new-report.json" --dry-run
```

After reviewing the same roots, options, destination, and reported content,
create it with that digest:

```sh
python3 scripts/duplicate_file_report.py scan "/path/to/folder" --report "/path/to/new-report.json" --approve DIGEST
```

The dry run creates nothing. The live command rescans and rejects a missing
or mismatched approval, including if the report destination or result changed.
Report creation is exclusive: an existing file or symlink is refused. The
report never modifies scanned files. An incomplete report needs
`--allow-partial` in both commands. `--json` prints structured output to
standard output without creating a file.

## Results and troubleshooting

- Exit `0`: complete scan, no duplicates.
- Exit `1`: complete scan, duplicates found.
- Exit `2`: incomplete scan; skipped entries are listed and the result says
  `incomplete`. `--allow-partial` permits normal duplicate/no-duplicate exit
  handling but does not clear that marker.

On macOS, `SF_DATALESS` marks an iCloud file that is not downloaded. The
scanner lists it under `not_downloaded` and never opens it. Unreadable entries
also make the scan incomplete. Resolve filesystem access or download selected
files through the user's normal workflow, then rescan if full coverage is
needed. Do not interpret a partial report as a complete inventory.

The report contains local paths and modification times. Keep it in a location
appropriate for the user's files. Read
[references/report-format.md](references/report-format.md) for field meanings.

## Scope, platform and authorization

Python 3.10 or newer on macOS or Linux with POSIX directory descriptors and O_NOFOLLOW; Windows is unsupported and exits 2 before filesystem I/O.

The approval binds the fully resolved report destination, its directory and parent device/inode identities, every ancestor identity, and exact JSON bytes. Live writes walk directories with no-follow descriptor opens and create the report relative to the pinned directory. No directories are created; the report parent must already exist. All path components must be real directories, not symlinks.

Roots may be absolute or relative caller-selected paths; parent traversal (..) and NUL are rejected before scanning. Percent signs and backslashes are literal local filename characters; paths are never URL-decoded. Directory metadata is read to enumerate selected roots, including entries excluded by filters; hidden/package filters prevent descending excluded directories and opening excluded files. Candidate file bytes are read only for size groups with at least two files. Hashing streams chunks of at most 1 MiB, reads at most the observed size plus one growth-detection byte per full hash, and rejects changed files as unreadable/incomplete. No stdin is read.

SHA-256 matches describe observed bytes, not photo similarity or a permanent snapshot. Files may change after hashing. Reclaimable bytes are a logical estimate; hard links, clones, compression and allocation can make physical savings smaller.

Use the physical path when an ancestor is a symlink (for example /private/tmp on macOS). A symlinked or non-directory ancestor of a scan ROOT is an input error (exit 2, even with --allow-partial), rather than an unreadable partial result. Regular-file ancestors are reported as not directories. A ROOT that is itself a symlink is skipped, like other symlink entries.

By Zero Width Lab (zerowidthlab.com)

License: MIT (read it).

Back to the listing