sliding-time-window

Scaling with a Sliding Time Window

Use when only recent data needs fast search -- social media posts, news articles, support tickets, logs, job listings. Old data either becomes irrelevant or can tolerate slower access.

Three strategies: shard rotation (recommended), collection rotation (when per-period config differs), and filter-and-delete (simplest, for continuous cleanup).

Shard Rotation (Recommended)

Use when: data has natural time boundaries (daily, weekly, monthly). Preferred because queries span all time periods in one request without application-level fan-out. User-defined sharding

1. Create a collection with user-defined sharding enabled
2. Create one shard key per time period (e.g., 2025-01, 2025-02, ..., 2025-06)
3. Ingest data into the current period's shard key
4. When a new period starts, create a new shard key, confirm it exists and is active, run a test write and read against it, then redirect writes. Keep the previous shard key until new writes are confirmed.
5. Before deleting the oldest shard key: list the existing shard keys, compute the retention boundary explicitly, confirm the target is not the current write target, take a snapshot if the data may be needed later, and get explicit user confirmation naming the exact shard key. Then delete the oldest shard key outside the retention window.

• Deleting a shard key removes the whole shard without tombstones or compaction, which is far cheaper than deleting points; still schedule it off-peak and verify resources are freed
• Pre-create the next period's shard key before rotation to avoid write disruption
• Use shard_key_selector at query time to search only specific periods for efficiency
• Shard keys can be placed on specific nodes for hot/cold tiering

Collection Rotation (Alias Swap)

Use when: you need per-period collection configuration (e.g., different quantization or storage settings). Collection aliases

1. Create one collection per time period, point a write alias at the newest
2. Query across all active collections in parallel, merge results client-side
3. When a new period starts, create the new collection, confirm it exists and is active, and swap the write alias in a single atomic alias update Switch collection. Then run a test write and read through the alias, and keep the previous collection until new writes are confirmed.
4. Before dropping the oldest collection: list the existing collections, compute the retention boundary explicitly, confirm that no alias points to it and that it is removed from the client-side query fan-out, take a snapshot if the data may be needed later, and get explicit user confirmation naming the exact collection. Then drop the oldest collection outside the window.

Trade-off vs shard rotation: allows per-collection config differences, but requires application-level fan-out and more operational overhead.

Filter-and-Delete

Use when: data arrives continuously without clear time boundaries, or you want the simplest setup.

1. Store a timestamp payload on every point, create a payload index on it Payload index. Use one consistent timestamp type, unit, and timezone that matches the payload index.
2. Filter to the desired window at query time using range condition Range filter
3. Periodically delete expired points using delete-by-filter Delete points. Use only an upper bound cutoff computed as now minus the retention period. When setting up or changing the cleanup filter, first count points with the identical filter, check that the count is below the collection total and within expected bounds, and get user confirmation. On each later run, repeat the count preview and stop if it is far above the expected volume.

• Run cleanup during off-peak hours in batches (10k-50k points as a starting heuristic; tune against observed latency and optimizer status) to limit optimizer and indexing load and latency spikes
• Deletes are not free: tombstoned points degrade search until optimizer compacts segments
• Does not reclaim disk instantly (compaction is asynchronous)

Hot/Cold Tiers

Use when: recent data needs fast in-RAM search, older data should remain searchable at lower performance.

• Shard rotation: place current shard key on fast-storage nodes, move older shard keys to cheaper nodes via shard placement. All queries still go through a single collection. Moving an existing shard key means moving its shards between nodes, which transfers data: check disk and RAM capacity on target nodes, run moves off-peak, and monitor transfer status.
• Collection rotation: keep quantized vectors of the current collection in RAM (always_ram: true), a setting that applies to quantized vectors only; move older collections to mmap/on-disk vectors and store their index on disk too. Quantization

What NOT to Do

• Do not use filter-and-delete for high-volume time-series with millions of daily deletes (use rotation instead)
• Do not forget to index the timestamp field (range filters without an index cause full scans)
• Do not use collection rotation when shard rotation would suffice (unnecessary fan-out complexity)
• Do not drop a shard key or collection before verifying its period is fully outside the retention window
• Do not skip pre-creating the next period's shard key or collection (write failures during rotation are hard to recover)