references/report-format.md — Duplicate File and Photo Report

# Report format and interpretation

The command `scan ROOT... --json` prints a JSON result. `--report PATH`
creates the same result at a new path, after a dry run and matching approval
digest. Plain output renders paths on one line with control and formatting
characters escaped. JSON stores paths as JSON strings.

## Result fields

| Field | Meaning |
| --- | --- |
| `roots` | Selected scan roots. |
| `min_size` | Minimum file size used for duplicate comparison, in bytes. |
| `include_hidden` | Whether hidden entries were included. |
| `include_packages` | Whether package directories were traversed. |
| `scanned_files` | Regular files considered by the scan. |
| `not_downloaded` | Count of macOS dataless files deliberately not opened. |
| `unreadable` | Count of entries that could not be read. |
| `skipped` | Paths and reasons for missing coverage. |
| `incomplete` | True if dataless or unreadable entries prevented full coverage. |
| `groups` | Byte-identical file groups after size, partial hash, and full SHA-256 checks. |
| `reclaimable_bytes` | Sum of each group's total size minus one retained copy. |

Each group records its `size`, `reclaimable_bytes`, and `files`. Each file
records its `path`, `size`, and `mtime_ns` (modification time as nanoseconds
since the Unix epoch). The hashes are comparison steps, not a permission to
delete. Two matching files may both matter to applications or workflows.

## Coverage and safety

The scanner skips symlinks and special files. It skips hidden entries and
macOS package directories by default. `--include-packages` can traverse a
Photos library, but reading inside one is not a supported way to deduplicate
it; never remove its internal files based on this output.

On macOS, the `SF_DATALESS` stat flag (`0x40000000`) means a file may be
stored in iCloud and absent locally. Opening it can trigger a download, so
the scanner never reads it. These entries appear in `skipped` and count as
`not_downloaded`. Unreadable entries also appear in `skipped`. The result is
`incomplete` in either case and exits `2` unless `--allow-partial` is set.

The estimate retains one copy per group and assumes the other bytes are
reclaimable. Filesystem sharing, backups, application references, and other
storage details can make the actual freed space different. Review manually.

## Scope, platform and authorization

Python 3.10 or newer on macOS or Linux with POSIX directory descriptors and O_NOFOLLOW; Windows is unsupported and exits 2 before filesystem I/O.

The approval binds the fully resolved report destination, its directory and parent device/inode identities, every ancestor identity, and exact JSON bytes. Live writes walk directories with no-follow descriptor opens and create the report relative to the pinned directory. No directories are created; the report parent must already exist. All path components must be real directories, not symlinks.

Roots may be absolute or relative caller-selected paths; parent traversal (..) and NUL are rejected before scanning. Percent signs and backslashes are literal local filename characters; paths are never URL-decoded. Directory metadata is read to enumerate selected roots, including entries excluded by filters; hidden/package filters prevent descending excluded directories and opening excluded files. Candidate file bytes are read only for size groups with at least two files. Hashing streams chunks of at most 1 MiB, reads at most the observed size plus one growth-detection byte per full hash, and rejects changed files as unreadable/incomplete. No stdin is read.

SHA-256 matches describe observed bytes, not photo similarity or a permanent snapshot. Files may change after hashing. Reclaimable bytes are a logical estimate; hard links, clones, compression and allocation can make physical savings smaller.

Use the physical path when an ancestor is a symlink (for example /private/tmp on macOS). A symlinked or non-directory ancestor of a scan ROOT is an input error (exit 2, even with --allow-partial), rather than an unreadable partial result. Regular-file ancestors are reported as not directories. A ROOT that is itself a symlink is skipped, like other symlink entries.

By Zero Width Lab (zerowidthlab.com)

License: MIT (read it).

Back to the listing