FREE · TOPIC 082

Blocklist Versioned File Metadata

460 words·Updated 2026-07-18·
#system-design#file-sync#metadata#storage
# Blocklist Versioned File Metadata A blocklist is the ordered list of block hashes that defines a file version. The metadata layer can be append-only even when files look mutable to users. ## Schema Shape ```text file_journal( namespace_id, relative_path, version_id, blocklist, metadata, created_at ) ``` Key properties: - `namespace_id` scopes the file tree. - `relative_path` identifies the user-visible file path. - `version_id` is monotonically increasing within the namespace. - `blocklist` is ordered. - rows are appended, not updated in place. ## Why Append-Only Works If `video.avi` changes from: ```text [h1, h2, h3, h4] ``` to: ```text [h1, h2, h5, h4] ``` the server appends a new row. Clients with the old cursor fetch the new row, notice they only lack `h5`, and download one block. ## Sync Cursor Each client tracks: ```text namespace_id -> last_seen_version_id ``` To sync: ```text list_changes(namespace_id, after_version_id) ``` This avoids a full directory tree comparison on every sync cycle. ## Modeling Deletes And Renames Do not delete the old journal row. Append a tombstone or rename event: ```text {version_id: 41, path: "/a.txt", op: "delete"} {version_id: 42, path: "/b.txt", op: "rename", from: "/old.txt"} ``` This keeps incremental sync simple. A client that was offline can replay every namespace change after its cursor and converge without scanning the whole tree. ## Commit Validation The metadata service should check: - caller has write permission for the namespace, - parent folder still exists, - base version is compatible with current path state, - every block hash is present or reported as missing, - mutation ID has not already committed. If validation fails because blocks are missing, return the missing hashes instead of appending a partial version. If validation fails because the base version is stale, return conflict information. ## Failure Modes | Failure | Mitigation | |---|---| | two clients edit same path offline | conflict copy or merge policy | | client retries commit after timeout | idempotency key returns committed version | | block garbage collection races commit | mark referenced blocks before deleting old ones | | rename and edit cross in flight | serialize by namespace version order | | privacy deletion requests old versions | retention policy must cover journal and blocks | ## Senior-Level Checks - Is `version_id` generated transactionally? - Is the journal scoped per account, namespace, shared folder, or global system? - How are deletes represented? - How are renames represented? - Can clients detect local corruption and ask for missing blocks again? - How long are old versions retained? - How does privacy deletion interact with immutable blocks and dedupe? - Can change listing be paginated without skipping versions? - Is the cursor durable across client reinstall or device restore? ## Related Pages - [[wiki/remote-file-sync-design]] - [[wiki/fixed-block-chunking-and-content-addressing]] - [[wiki/immutable-versioned-data-files]] - [[wiki/metadata-db-for-object-storage]]
Primary References & Engineering Sources