# Byte-Range Indexed Object Storage
Byte-range reads let an application fetch only part of a large object. With a separate index, object storage can support point-lookups without behaving like a database.
## Pattern
```text
index[key] -> object_path, start_offset, byte_length
range read object_path[start_offset : start_offset + byte_length]
```
This is the core trick behind the lecture's dictionary design.
## Why It Works
Object storage is good at durable blob storage, high aggregate throughput, and simple object reads. It is not a query engine.
The index supplies the missing query plan:
```mermaid
flowchart LR
Key[key] --> Index[in-memory index]
Index --> Range[start + length]
Range --> Obj[object storage byte range]
Obj --> Value[value bytes]
```
## S3 And HTTP Range
S3 supports fetching part of an object using a byte range. HTTP also defines range requests for `GET`, with `206 Partial Content` responses when a satisfiable range is served.
Use cases:
- large dictionary/catalog lookup,
- reading media/video chunks,
- reading indexed binary file sections,
- parallel download of huge objects,
- serving static immutable data with a small hot index.
## Latency And Cost Model
A point lookup is cheap only if the index lookup avoids extra network calls.
```text
total latency ~= API work + index lookup + one object range GET
```
If the index lives in a remote database, the design becomes two network hops before any bytes return. If the index fits in process memory, the object store is the only remote read. That is why this pattern works best for static catalogs, model artifacts, media manifests, or dictionary-like datasets whose index can be loaded at startup.
| Choice | Effect |
|---|---|
| one record per range | simple reads, many small object requests |
| block of records per range | fewer requests, extra local scan/decode |
| local hot-value cache | lower p95 for popular keys, cache invalidation by version |
## Design Checks
- Is the object immutable while this index is active?
- Is the index version tied to the object version?
- Does the range read include exactly one logical value or a block containing many values?
- Are offsets byte offsets after compression or before compression?
- What happens if object storage returns partial/failing reads?
- Can API servers cache hot values locally?
## Compression Caveat
Compression changes range-read math.
If the whole file is compressed as one stream, arbitrary byte ranges may be useless because decompressing a value can require earlier bytes. Prefer:
- per-record compression,
- block compression with a block index,
- or no compression for the data section if direct ranges matter more than space.
## Security And Access
Do not expose raw object paths and byte ranges as an authorization bypass. The API should still enforce:
- caller permission to the logical key,
- allowed range boundaries,
- content type/encoding,
- maximum range length,
- request rate limits.
Object storage is the byte source; the application remains the policy layer.
## Operational Failure Modes
Track range-read errors separately from full-object errors. A bad offset, stale index, wrong compression mode, or off-by-one length can all return valid HTTP responses with semantically wrong bytes. Checksums or record delimiters are the fastest way to catch that class of bug.