Limitations
These are the design weaknesses to weigh before relying on the tool. None of them loses data silently; most cost time or space.
Topology and scale
- One master. The topology is a star around one server. The replica is a pull-only mirror, not a failover: if the server is lost, clients stop until an administrator points them at a rebuilt master.
- Every run exchanges the whole manifest. The client sends one entry per file and the server answers with the whole plan as a single JSON line, so a run costs O(files) in time, memory and bandwidth even when nothing changed. A tree of a million files means a plan message of the order of a hundred megabytes. A change journal or Merkle summary would remove that, at the cost of the "everything is inspectable with cat" property.
- No line-length cap on the wire. Because of the previous point, a control message cannot be bounded without breaking large trees. The peer is authenticated by ssh and pinned by the forced command, so this is a resource risk from a misbehaving client, not from the network.
- Lock-step protocol. One request is in flight at a time and each 256 KiB chunk waits for its acknowledgement, so throughput is bound by the round-trip time on a high-latency link.
- Large trees cost memory. The plan holds every path in memory three times over — the union of the three manifests — and the server holds the whole client manifest and the plan. That is the price of a stateless server.
- Every run stats every file. There is no change notification. On a very large tree the walk, not the transfer, dominates a quiet run.
Consistency
- Clock-dependent tie-breaks. When both sides changed a file, the newer mtime wins (the master wins ties within two seconds). A client with a wrong clock picks the wrong winner; the loser is still kept as a conflict copy.
- The cache trusts size and mtime. A tool that rewrites a
file and restores its mtime — some backup restorers do — goes
unnoticed until a
verify --deep. - One scan per session. The plan is computed once, from a scan taken while the lock is held. Anything other than filesync writing into the master during a session is not seen until the next session, and a file it replaces may be overwritten.
- Pulls are verified against the plan. If the master changes during the session (only possible if something other than filesync writes to it), the pull fails with a hash mismatch and the next sync retries.
- Uploads are verified only at the end. A client that sends a wrong file of the announced size wastes the transfer before it is refused.
Durability
- No fsync. Renames are atomic between processes but not
durable across a power cut; the new name can point at a file whose data
never reached the disk. Adding
IO::Handle::syncbefore each rename would make every upload pay a disk flush; the trade-off has not been measured. - No extended metadata.
File::Copy::copydoes not copy extended attributes, ACLs or alternate data streams. - Snapshots share the disk. They live inside the replica root, on the same filesystem: they guard against bad syncs, not against losing that disk.
Deltas
- Pure Perl. The rolling loop runs at a few MB/s, which is why only 8 MiB after the matched prefix is scanned. An edit near the end of a large file whose earlier blocks also shifted is sent mostly literally.
- Block-aligned moves only. A block that moved by a whole number of blocks is found; a block that occurs twice in the receiver's file is matched to its first occurrence.
- 32-bit offsets. A delta cannot refer beyond 4 GiB into the receiver's file; larger files fall back to whole-file transfer of the remainder as literals.
What is not synced
- Symlinks, devices, sockets, extended attributes, ACLs and ownership. Permission bits are carried only between POSIX systems.
- Unicode normalisation is not reconciled, so the same name in NFC (Linux) and NFD (older macOS) is two different files. A case-insensitive client cannot hold two master paths that differ only in case.
- A rename is a deletion plus a creation, so a renamed large file is transferred again rather than moved.
- Ignore matching is literal and linear. Character
classes (
[abc]) and backslash escapes are taken literally; a trailing/is dropped, sobuild/matchesbuildat any depth, and as the matcher has no type information it matches a file of that name too. Combining patterns into one alternation would be faster but would lose "later patterns win".
Recovery and administration
- Trash is per repo and per client. Local trash written by
releases before 0.001.1 sits in one shared directory. It is still searched
by
restoreand pruned with the default 30 days, but arestorecan then offer another repo's file of the same relative path. - Same-day trash collisions are numeric. A real file whose
name already ends in
.N(backup.1) deleted the same day asbackupis listed as a version ofbackuptoo. Pruning works in whole UTC days, so retention is accurate to a day. - Trash cannot cross filesystems by rename. Files are copied and removed; directories must rename.
- Admin methods mostly take no lock. Only
reindexdoes;restoreinto the master rebuilds the index by walking the tree and does not take the repo lock, so a session running at the same moment can save an index that omits the restored file until its next sync. - The server journal is read whole.
filesync-admin logloads every monthly journal into memory before filtering and applying the limit. A repo with years of history pays for all of it on every call. - A shallow verify trusts size and mtime, so it cannot see
a change that kept both; use
--deepor--sample. The report says what differs, not which side is right. - Snapshot listing is slow.
snapshot listwalks every snapshot to count it.
Costs and implementation notes
- Encapsulation has a price. Private methods use
Sub::Private in
enforcemode, about 1.4 microseconds per call (measured on 5.40). Helpers that run per file or per JSON value — the merge comparisons, the JSON byte downgrade, the rolling checksum — are therefore lexical code references: private by scope, with no wrapper. - Validation has a price. Params::Validate::Strict costs about 11 microseconds per call, so it guards the public entry points that run a few times per command. Functions that run per path or per protocol frame check their arguments by hand.
- English only. The catalogue mechanism supports other
languages, plural forms and gender, but only the English catalogue ships.
The plural rules cover English, German, French, Chinese, Japanese and Korean;
other languages use the English rule until one is added. Operating-system
reasons (
$!) come from the C library and follow the process locale, not the catalogue language. - Option parsing is Getopt::Long's. Its diagnostics for an
unknown option are printed in English before the catalogued usage line.
--jsonoutput is not localised, because field names and values are data. filesync-adminreads--rootonly before the command name.- Two configuration mechanisms. The client's repo list is
JSON written by
filesync initand must round-trip exactly; process settings go through Object::Configure. Two mechanisms is one too many, but the layered merge does not promise an exact round trip.