BT9G LABS / DHT Deep Dive #17

How to Deduplicate Billions of InfoHash Observations Without Wasting Metadata Work

Design a multi-tier deduplication layer for DHT ingestion using exact keys, Bloom filters, hot caches, and durable state.

BT9G LabsPublished 4 min read

BT9G Labs / DHT Deep Dive #17. Design a multi-tier deduplication layer for DHT ingestion using exact keys, Bloom filters, hot caches, and durable state.

How to Deduplicate Billions of InfoHash Observations Without Wasting Metadata Work

Quick summary

  • Topic: How to Deduplicate Billions of InfoHash Observations Without Wasting Metadata Work
  • Core keywords: infohash deduplication, Bloom filter, DHT database, BitTorrent indexing, dedup architecture
  • Audience: backend engineers, protocol researchers, search engineers, and BitTorrent implementers
  • Goal: explain the mechanism accurately enough to support real crawler and indexing architecture decisions

System view

Deduplication is a first-class subsystem

DHT traffic repeats the same infohash many times. If every observation creates a metadata job, downstream capacity is consumed by duplicates rather than discovery. The dedup layer converts a noisy event stream into a smaller stream of meaningful state changes.

The canonical key should be the binary content identifier plus its version or hash namespace where necessary.

Probabilistic filters belong in front, not at the source of truth

A Bloom filter can answer "probably seen" with excellent memory efficiency, but false positives mean it cannot be the only record of whether a hash exists. It is best used as a fast front filter in front of an exact persistent store.

A hot cache can also suppress repeated writes for very active hashes while periodically flushing last-seen and count aggregates.

Separate existence from scheduling state

Knowing that an infohash exists is different from knowing whether metadata has been fetched, whether it failed, and when the next retry is due. A compact durable state machine avoids repeated work and supports clean recovery after process restarts.

Useful fields include first seen, last seen, observation count, metadata status, retry count, next retry time, and last failure category.

Partition for long-term growth

At large scale, partitioning by leading hash bytes distributes writes and makes maintenance predictable. The exact storage engine can vary: relational databases work well for control state; LSM-style stores can be attractive for high write volume; Redis is useful as a hot cache but should not become the only durable catalog.

Search indexes such as Elasticsearch should be treated as derived serving layers, not the canonical dedup authority.

Implementation sketch

CREATE TABLE infohash_state (
  hash VARBINARY(32) NOT NULL,
  hash_type TINYINT NOT NULL,
  first_seen DATETIME NOT NULL,
  last_seen DATETIME NOT NULL,
  seen_count BIGINT NOT NULL,
  metadata_status TINYINT NOT NULL,
  next_retry DATETIME NULL,
  PRIMARY KEY(hash_type, hash)
);

Engineering notes for BT9G Labs

For an indexing system, protocol correctness and data semantics matter more than raw event volume. Keep durable content identity separate from volatile peer observations, retain timestamps on measurements, and make every network-derived field explicit about its source. This prevents the search layer from turning a short-lived observation into a permanent factual claim.

Where possible, record stage-level metrics rather than only total success or failure. That makes it possible to see whether the bottleneck is DHT lookup quality, peer reachability, extension support, metadata transfer, validation, storage, or search indexing.

Common mistakes

  • Using a Bloom filter as the only source of truth.
  • Using Elasticsearch as the canonical dedup database.
  • Conflating "seen" with "metadata complete."
  • Updating durable storage on every duplicate hot-path event.

References

← Previous: What It Takes to Fetch 1,000+ Torrent Metadata Records per Minute Next: Why Failed InfoHashes Should Stay: Designing a Long-Term Retry Scheduler →

This article explains public BitTorrent protocol behavior for engineering and research purposes. It does not imply that every observed swarm is available, valid, or authorized for redistribution.