BT9G LABS / DHT Deep Dive #09

How a Magnet Link Becomes File Metadata: The Complete Peer and Metadata Flow

Trace a magnet link from infohash to DHT peer discovery, BEP 10 negotiation, and BEP 9 metadata retrieval.

BT9G LabsPublished 4 min read

BT9G Labs / DHT Deep Dive #09. Trace a magnet link from infohash to DHT peer discovery, BEP 10 negotiation, and BEP 9 metadata retrieval.

How a Magnet Link Becomes File Metadata: The Complete Peer and Metadata Flow

Quick summary

  • Topic: How a Magnet Link Becomes File Metadata: The Complete Peer and Metadata Flow
  • Core keywords: magnet link, torrent metadata, BEP 9, ut_metadata, peer discovery
  • Audience: backend engineers, protocol researchers, search engineers, and BitTorrent implementers
  • Goal: explain the mechanism accurately enough to support real crawler and indexing architecture decisions

System view

A magnet URI can identify a torrent without embedding the complete .torrent metadata. For a v1 torrent, the BTIH identifier is enough for a client to begin locating the swarm. The client still does not know the complete file tree, piece layout, or display name from the hash alone.

That separation is a key architectural feature: discovery can begin from a compact identifier, while metadata is obtained from the swarm itself.

Find peer candidates first

The client uses DHT, trackers embedded in the magnet URI when present, cached peers, or other discovery mechanisms to collect candidate endpoints. It then opens BitTorrent peer connections and performs the standard handshake using the target infohash.

A peer that advertises support for the extension protocol can continue into the BEP 10 extended handshake.

Negotiate ut_metadata

BEP 10 provides a namespace for negotiated extensions. During the extended handshake, peers advertise mappings for supported extension names. BEP 9 uses the ut_metadata extension and can also advertise metadata_size.

Once both sides agree on the extension message ID, the requester asks for metadata pieces. These are pieces of the torrent's info dictionary, not pieces of the torrent's payload files.

Verify before indexing

After assembling the metadata, the requester hashes the raw bencoded info dictionary and checks that it matches the expected infohash. Only then should a search system parse filenames, lengths, piece size, private flags, or other fields into its index.

This verification boundary is critical. Peers are untrusted, and a crawler should never index arbitrary metadata merely because a remote endpoint returned it.

Implementation sketch

magnet(infohash)
   -> discover peers
   -> BitTorrent handshake
   -> BEP 10 extended handshake
   -> ut_metadata requests
   -> assemble info dictionary
   -> hash(info dictionary) == expected infohash ?
   -> parse and index

Engineering notes for BT9G Labs

For an indexing system, protocol correctness and data semantics matter more than raw event volume. Keep durable content identity separate from volatile peer observations, retain timestamps on measurements, and make every network-derived field explicit about its source. This prevents the search layer from turning a short-lived observation into a permanent factual claim.

Where possible, record stage-level metrics rather than only total success or failure. That makes it possible to see whether the bottleneck is DHT lookup quality, peer reachability, extension support, metadata transfer, validation, storage, or search indexing.

Common mistakes

  • Expecting filenames to be encoded directly in the magnet hash.
  • Skipping the BEP 10 extension negotiation.
  • Indexing metadata before verifying its infohash.
  • Using one peer failure as evidence that no metadata exists.

References

← Previous: How DHT Crawlers Discover InfoHashes: Active Lookups, Observations, and Responsible Design Next: BEP 9 ut_metadata Explained: Fetching Torrent Metadata from Peers →

This article explains public BitTorrent protocol behavior for engineering and research purposes. It does not imply that every observed swarm is available, valid, or authorized for redistribution.