BT9G LABS / DHT Deep Dive #18

Why Failed InfoHashes Should Stay: Designing a Long-Term Retry Scheduler

Build a retry model that preserves historical hashes, backs off cold swarms, and quickly reactivates when fresh activity appears.

BT9G LabsPublished 4 min read

BT9G Labs / DHT Deep Dive #18. Build a retry model that preserves historical hashes, backs off cold swarms, and quickly reactivates when fresh activity appears.

Why Failed InfoHashes Should Stay: Designing a Long-Term Retry Scheduler

Quick summary

  • Topic: Why Failed InfoHashes Should Stay: Designing a Long-Term Retry Scheduler
  • Core keywords: DHT retry strategy, metadata retry, dead torrent, exponential backoff, infohash scheduler
  • Audience: backend engineers, protocol researchers, search engineers, and BitTorrent implementers
  • Goal: explain the mechanism accurately enough to support real crawler and indexing architecture decisions

System view

A failure timestamp is not a tombstone

An infohash can fail today because no reachable peer is online and succeed months later when someone re-seeds the torrent. Deleting the hash after a short failure window throws away discovery work and prevents the system from recognizing reactivation.

The cheaper strategy is to retain the identity and make retry frequency decay over time.

Backoff should reflect failure type

No peers found might retry on a schedule such as minutes, hours, days, and then weeks. A peer-specific protocol failure should penalize that endpoint more than the swarm. Invalid metadata should trigger another peer before triggering a long swarm-level delay.

Random jitter prevents millions of old hashes from waking up at the same exact boundary after a restart or scheduler sweep.

Fresh activity should override stale pessimism

A new DHT observation, tracker hit, PEX lead, or successful peer connection is evidence that conditions changed. The scheduler can move the hash forward and reduce its next-retry delay.

This event-driven reactivation is much cheaper than constantly polling every historical hash.

Make the retry queue durable

For long-term crawling, retry state must survive deploys and crashes. A database table or durable priority queue keyed by next_retry makes scheduling reproducible. Workers claim batches with leases so that one crash does not lose a job forever.

Archival does not mean deletion. Very cold hashes can remain in cheap storage while being absent from the hot working set.

Implementation sketch

attempt 1: +30 minutes
attempt 2: +2 hours
attempt 3: +8 hours
attempt 4: +1 day
attempt 5: +7 days
attempt 6+: +30 days with jitter

fresh DHT observation => move next_retry closer

Engineering notes for BT9G Labs

For an indexing system, protocol correctness and data semantics matter more than raw event volume. Keep durable content identity separate from volatile peer observations, retain timestamps on measurements, and make every network-derived field explicit about its source. This prevents the search layer from turning a short-lived observation into a permanent factual claim.

Where possible, record stage-level metrics rather than only total success or failure. That makes it possible to see whether the bottleneck is DHT lookup quality, peer reachability, extension support, metadata transfer, validation, storage, or search indexing.

Common mistakes

  • Permanently deleting hashes after short outages.
  • Using one global retry interval for every failure class.
  • Scheduling huge retry waves without jitter.
  • Keeping every historical hash in the hot queue.

References

← Previous: How to Deduplicate Billions of InfoHash Observations Without Wasting Metadata Work Next: Building a BitTorrent Search Index: DHT to Metadata to Elasticsearch →

This article explains public BitTorrent protocol behavior for engineering and research purposes. It does not imply that every observed swarm is available, valid, or authorized for redistribution.