README.md — Duplicate File and Photo Report

# Duplicate File and Photo Report

Duplicate File and Photo Report is a local, read-only duplicate scanner for
user-selected folders. It compares file size, a partial hash over the first
and last 64 KiB, and full SHA-256 before reporting byte-identical groups.
Each group lists paths, sizes, modification times, and the bytes that could
be reclaimed if one copy were retained. The tool never deletes, moves, or
changes the source files.

## Requirements and install

Python 3.10 or newer and read access to the selected files are required. The
script uses the Python standard library only. No account, credential,
telemetry, or network access is involved. Copy the complete package into
your agent harness's skill directory, or run the script directly from this
folder. The package includes the script, skill instructions, companion agent,
references, license, and tests. See [SETUP.md](SETUP.md).

## Quick start

Run commands from the folder containing `SKILL.md`:

```sh
python3 scripts/duplicate_file_report.py scan "/path/to/folder"
python3 scripts/duplicate_file_report.py scan "/path/to/photos" "/path/to/documents" --min-size 4096 --json
```

The roots are explicit and repeatable. `--include-hidden` includes hidden
entries; `--include-packages` traverses macOS package directories normally
skipped. A `.photoslibrary` is an application-managed database. Reading
inside a Photos library is not a supported way to deduplicate it; do not
remove internal files based on this report.

## Create a report

Choose a new report path and preview it:

```sh
python3 scripts/duplicate_file_report.py scan "/path/to/folder" --report "/path/to/new-report.json" --dry-run
python3 scripts/duplicate_file_report.py scan "/path/to/folder" --report "/path/to/new-report.json" --approve DIGEST
```

Replace `DIGEST` with the approval digest from the dry run. The dry run
writes nothing. Live mode rescans and accepts the approval only when both
the destination identities and exact JSON report content match. An existing
report path or symlink is refused; the tool never overwrites. The report
uses the same structured result as `--json`. Treat output paths as private
data when sharing a report. If the scan is incomplete, a live report also
requires `--allow-partial`.

## Reading results

Each duplicate group has one size and at least two files. Its reclaimable
estimate is the sum of those files' sizes minus one retained copy. The total
is an estimate, not a deletion plan: files may be referenced by applications,
backups, or workflows, and duplicate bytes can have different meanings.
Review candidates manually and make any deletion decision outside this tool.

On macOS, files marked `SF_DATALESS` are never opened: reading one could
download it from iCloud. They are counted as `not_downloaded` and listed.
Unreadable entries are listed too. Either makes the scan `incomplete` and
returns exit `2` unless `--allow-partial` was selected. That flag only changes
exit handling; it does not fill the gap. A complete result exits `1` when
duplicates were found, or `0` when none were found. Symlinks and special
files are skipped without reading them.

Human-readable output escapes control and formatting characters in names
onto a single line. JSON preserves paths as JSON strings. See
[references/report-format.md](references/report-format.md) for fields and
interpretation.

## Tests and license

```sh
python3 -m unittest discover -s tests -v
```

Version `0.1.0`. Free under the MIT license; see [LICENSE.txt](LICENSE.txt).
Apple Photos is a trademark of Apple Inc. This independent tool is not
affiliated with or endorsed by Apple.

## Scope, platform and authorization

Python 3.10 or newer on macOS or Linux with POSIX directory descriptors and O_NOFOLLOW; Windows is unsupported and exits 2 before filesystem I/O.

The approval binds the fully resolved report destination, its directory and parent device/inode identities, every ancestor identity, and exact JSON bytes. Live writes walk directories with no-follow descriptor opens and create the report relative to the pinned directory. No directories are created; the report parent must already exist. All path components must be real directories, not symlinks.

Roots may be absolute or relative caller-selected paths; parent traversal (..) and NUL are rejected before scanning. Percent signs and backslashes are literal local filename characters; paths are never URL-decoded. Directory metadata is read to enumerate selected roots, including entries excluded by filters; hidden/package filters prevent descending excluded directories and opening excluded files. Candidate file bytes are read only for size groups with at least two files. Hashing streams chunks of at most 1 MiB, reads at most the observed size plus one growth-detection byte per full hash, and rejects changed files as unreadable/incomplete. No stdin is read.

SHA-256 matches describe observed bytes, not photo similarity or a permanent snapshot. Files may change after hashing. Reclaimable bytes are a logical estimate; hard links, clones, compression and allocation can make physical savings smaller.

Use the physical path when an ancestor is a symlink (for example /private/tmp on macOS). A symlinked or non-directory ancestor of a scan ROOT is an input error (exit 2, even with --allow-partial), rather than an unreadable partial result. Regular-file ancestors are reported as not directories. A ROOT that is itself a symlink is skipped, like other symlink entries.

By Zero Width Lab (zerowidthlab.com)

License: MIT (read it).

Back to the listing