reference-DHT.rst
Author: Arvid Norberg, arvid@libtorrent.org

| libtorrent

Author: | Arvid Norberg, arvid@libtorrent.org
Version: | 1.2.9

home

# DHT

Table of contents

- DHT
  - dht_settings
  - dht_storage_counters
  - dht_storage_interface
  - dht_state
  - dht_default_storage_constructor()
  - sign_mutable_item()
  - announce_flags_t

[report issue]

## dht_settings

Declared in "libtorrent/kademlia/dht_settings.hpp"

structure used to hold configuration options for the DHT

The `dht_settings` struct used to contain a `service_port` member to control which port the DHT would listen on and send messages from. This field is deprecated and ignored. libtorrent always tries to open the UDP socket on the same port as the TCP socket.

    struct dht_settings
    {
       int max_peers_reply  = 100;
       int search_branching  = 5;
       int max_fail_count  = 20;
       int max_torrents  = 2000;
       int max_dht_items  = 700;
       int max_peers  = 500;
       int max_torrent_search_reply  = 20;
       bool restrict_routing_ips  = true;
       bool restrict_search_ips  = true;
       bool extended_routing_table  = true;
       bool aggressive_lookups  = true;
       bool privacy_lookups  = false;
       bool enforce_node_id  = false;
       bool ignore_dark_internet  = true;
       int block_timeout  = 5 * 60;
       int block_ratelimit  = 5;
       bool read_only  = false;
       int item_lifetime  = 0;
       int upload_rate_limit  = 8000;
       int sample_infohashes_interval  = 21600;
       int max_infohashes_sample_count  = 20;
    };

[report issue]

max_peers_reply
  the maximum number of peers to send in a reply to `get_peers`

[report issue]

search_branching
  the number of concurrent search request the node will send when announcing and refreshing the routing table. This parameter is called alpha in the kademlia paper

[report issue]

max_fail_count
  the maximum number of failed tries to contact a node before it is removed from the routing table. If there are known working nodes that are ready to replace a failing node, it will be replaced immediately, this limit is only used to clear out nodes that don't have any node that can replace them.

[report issue]

max_torrents
  the total number of torrents to track from the DHT. This is simply an upper limit to make sure malicious DHT nodes cannot make us allocate an unbounded amount of memory.

[report issue]

max_dht_items
  max number of items the DHT will store

[report issue]

max_peers
  the max number of peers to store per torrent (for the DHT)

[report issue]

max_torrent_search_reply
  the max number of torrents to return in a torrent search query to the DHT

[report issue]

restrict_routing_ips

  determines if the routing table entries should restrict entries to one per IP. This defaults to true, which helps mitigate some attacks on the DHT. It prevents adding multiple nodes with IPs with a very close CIDR distance.

  when set, nodes whose IP address that's in the same /24 (or /64 for IPv6) range in the same routing table bucket. This is an attempt to mitigate node ID spoofing attacks also restrict any IP to only have a single entry in the whole routing table

[report issue]

restrict_search_ips
  determines if DHT searches should prevent adding nodes with IPs with very close CIDR distance. This also defaults to true and helps mitigate certain attacks on the DHT.

[report issue]

extended_routing_table
  makes the first buckets in the DHT routing table fit 128, 64, 32 and 16 nodes respectively, as opposed to the standard size of 8. All other buckets have size 8 still.

[report issue]

aggressive_lookups
  slightly changes the lookup behavior in terms of how many outstanding requests we keep. Instead of having branch factor be a hard limit, we always keep branch factor outstanding requests to the closest nodes. i.e. every time we get results back with closer nodes, we query them right away. It lowers the lookup times at the cost of more outstanding queries.

[report issue]

privacy_lookups
  when set, perform lookups in a way that is slightly more expensive, but which minimizes the amount of information leaked about you.

[report issue]

enforce_node_id
  when set, node's whose IDs that are not correctly generated based on its external IP are ignored. When a query arrives from such node, an error message is returned with a message saying "invalid node ID".

[report issue]

ignore_dark_internet
  ignore DHT messages from parts of the internet we wouldn't expect to see any traffic from

[report issue]

block_timeout
  the number of seconds a DHT node is banned if it exceeds the rate limit. The rate limit is averaged over 10 seconds to allow for bursts above the limit.

[report issue]

block_ratelimit
  the max number of packets per second a DHT node is allowed to send without getting banned.

[report issue]

read_only
  when set, the other nodes won't keep this node in their routing tables, it's meant for low-power and/or ephemeral devices that cannot support the DHT, it is also useful for mobile devices which are sensitive to network traffic and battery life. this node no longer responds to 'query' messages, and will place a 'ro' key (value = 1) in the top-level message dictionary of outgoing query messages.

[report issue]

item_lifetime
  the number of seconds a immutable/mutable item will be expired. default is 0, means never expires.

[report issue]

upload_rate_limit
  the number of bytes per second (on average) the DHT is allowed to send. If the incoming requests causes to many bytes to be sent in responses, incoming requests will be dropped until the quota has been replenished.

[report issue]

sample_infohashes_interval
  the info-hashes sample recomputation interval (in seconds). The node will precompute a subset of the tracked info-hashes and return that instead of calculating it upon each request. The permissible range is between 0 and 21600 seconds (inclusive).

[report issue]

max_infohashes_sample_count
  the maximum number of elements in the sampled subset of info-hashes. If this number is too big, expect the DHT storage implementations to clamp it in order to allow UDP packets go through

[report issue]

## dht_storage_counters

Declared in "libtorrent/kademlia/dht_storage.hpp"

This structure hold the relevant counters for the storage

    struct dht_storage_counters
    {
       void reset ();

       std::int32_t torrents  = 0;
       std::int32_t peers  = 0;
       std::int32_t immutable_data  = 0;
       std::int32_t mutable_data  = 0;
    };

[report issue]

### reset()

    void reset ();

This member function set the counters to zero.

[report issue]

## dht_storage_interface

Declared in "libtorrent/kademlia/dht_storage.hpp"

The DHT storage interface is a pure virtual class that can be implemented to customize how the data for the DHT is stored.

The default storage implementation uses three maps in RAM to save the peers, mutable and immutable items and it's designed to provide a fast and fully compliant behavior of the BEPs.

libtorrent comes with one built-in storage implementation: `dht_default_storage` (private non-accessible class). Its constructor function is called dht_default_storage_constructor(). You should know that if this storage becomes full of DHT items, the current implementation could degrade in performance.

    struct dht_storage_interface
    {
       virtual void update_node_ids (std::vector<node_id> const& ids) = 0;
       virtual bool get_peers (sha1_hash const& info_hash
          , bool noseed, bool scrape, address const& requester
          , entry& peers) const = 0;
       virtual void announce_peer (sha1_hash const& info_hash
          , tcp::endpoint const& endp
          , string_view name, bool seed) = 0;
       virtual bool get_immutable_item (sha1_hash const& target
          , entry& item) const = 0;
       virtual void put_immutable_item (sha1_hash const& target
          , span<char const> buf
          , address const& addr) = 0;
       virtual bool get_mutable_item_seq (sha1_hash const& target
          , sequence_number& seq) const = 0;
       virtual bool get_mutable_item (sha1_hash const& target
          , sequence_number seq, bool force_fill
          , entry& item) const = 0;
       virtual void put_mutable_item (sha1_hash const& target
          , span<char const> buf
          , signature const& sig
          , sequence_number seq
          , public_key const& pk
          , span<char const> salt
          , address const& addr) = 0;
       virtual int get_infohashes_sample (entry& item) = 0;
       virtual void tick () = 0;
       virtual dht_storage_counters counters () const = 0;
    };

[report issue]

### update_node_ids()

    virtual void update_node_ids (std::vector<node_id> const& ids) = 0;

This member function notifies the list of all node's ids of each DHT running inside libtorrent. It's advisable that the concrete implementation keeps a copy of this list for an eventual prioritization when deleting an element to make room for a new one.

[report issue]

### get_peers()

    virtual bool get_peers (sha1_hash const& info_hash
          , bool noseed, bool scrape, address const& requester
          , entry& peers) const = 0;

This function retrieve the peers tracked by the DHT corresponding to the given info_hash. You can specify if you want only seeds and/or you are scraping the data.

For future implementers: If the torrent tracked contains a name, such a name must be stored as a string in peers["n"]

If the scrape parameter is true, you should fill these keys:

> peers["BFpe"]
>   with the standard bit representation of a 256 bloom filter containing the downloaders
> peers["BFsd"]
>   with the standard bit representation of a 256 bloom filter containing the seeders

If the scrape parameter is false, you should fill the key peers["values"] with a list containing a subset of peers tracked by the given info_hash. Such a list should consider the value of dht_settings::max_peers_reply. If noseed is true only peers marked as no seed should be included.

returns true if the maximum number of peers are stored for this info_hash.

[report issue]

### announce_peer()

    virtual void announce_peer (sha1_hash const& info_hash
          , tcp::endpoint const& endp
          , string_view name, bool seed) = 0;

This function is named announce_peer for consistency with the upper layers, but has nothing to do with networking. Its only responsibility is store the peer in such a way that it's returned in the entry with the lookup_peers.

The `name` parameter is the name of the torrent if provided in the announce_peer DHT message. The length of this value should have a maximum length in the final storage. The default implementation truncate the value for a maximum of 50 characters.

[report issue]

### get_immutable_item()

    virtual bool get_immutable_item (sha1_hash const& target
          , entry& item) const = 0;

This function retrieves the immutable item given its target hash.

For future implementers: The value should be returned as an entry in the key item["v"].

returns true if the item is found and the data is returned inside the (entry) out parameter item.

[report issue]

### put_immutable_item()

    virtual void put_immutable_item (sha1_hash const& target
          , span<char const> buf
          , address const& addr) = 0;

Store the item's data. This layer is only for storage. The authentication of the item is performed by the upper layer.

For implementers: This data can be stored only if the target is not already present. The implementation should consider the value of dht_settings::max_dht_items.

[report issue]

### get_mutable_item_seq()

    virtual bool get_mutable_item_seq (sha1_hash const& target
          , sequence_number& seq) const = 0;

This function retrieves the sequence number of a mutable item.

returns true if the item is found and the data is returned inside the out parameter seq.

[report issue]

### get_mutable_item()

    virtual bool get_mutable_item (sha1_hash const& target
          , sequence_number seq, bool force_fill
          , entry& item) const = 0;

This function retrieves the mutable stored in the DHT.

For implementers: The item sequence should be stored in the key item["seq"]. if force_fill is true or (0 <= seq and seq < item["seq"]) the following keys should be filled item["v"] - with the value no encoded. item["sig"] - with a string representation of the signature. item["k"] - with a string representation of the public key.

returns true if the item is found and the data is returned inside the (entry) out parameter item.

[report issue]

### put_mutable_item()

    virtual void put_mutable_item (sha1_hash const& target
          , span<char const> buf
          , signature const& sig
          , sequence_number seq
          , public_key const& pk
          , span<char const> salt
          , address const& addr) = 0;

Store the item's data. This layer is only for storage. The authentication of the item is performed by the upper layer.

For implementers: The sequence number should be checked if the item is already present. The implementation should consider the value of dht_settings::max_dht_items.

[report issue]

### get_infohashes_sample()

    virtual int get_infohashes_sample (entry& item) = 0;

This function retrieves a sample info-hashes

For implementers: The info-hashes should be stored in ["samples"] (N x 20 bytes). the following keys should be filled item["interval"] - the subset refresh interval in seconds. item["num"] - number of info-hashes in storage.

Internally, this function is allowed to lazily evaluate, cache and modify the actual sample to put in `item`

returns the number of info-hashes in the sample.

[report issue]

### tick()

    virtual void tick () = 0;

This function is called periodically (non-constant frequency).

For implementers: Use this functions for expire peers or items or any other storage cleanup.

[report issue]

### counters()

    virtual dht_storage_counters counters () const = 0;

return stats counters for the store

[report issue]

## dht_state

Declared in "libtorrent/kademlia/dht_state.hpp"

This structure helps to store and load the state of the `dht_tracker`. At this moment the library is only a dual stack implementation of the DHT. See BEP 32

    struct dht_state
    {
       void clear ();

       node_ids_t nids;
       std::vector<udp::endpoint> nodes;
       std::vector<udp::endpoint> nodes6;
    };

[report issue]

nodes
  the bootstrap nodes saved from the buckets node

[report issue]

nodes6
  the bootstrap nodes saved from the IPv6 buckets node

[report issue]

## dht_default_storage_constructor()

Declared in "libtorrent/kademlia/dht_storage.hpp"

    std::unique_ptr<dht_storage_interface> dht_default_storage_constructor (
       dht_settings const& settings);

constructor for the default DHT storage. The DHT storage is responsible for maintaining peers and mutable and immutable items announced and stored/put to the DHT node.

[report issue]

## sign_mutable_item()

Declared in "libtorrent/kademlia/item.hpp"

    signature sign_mutable_item (
       span<char const> v
       , span<char const> salt
       , sequence_number seq
       , public_key const& pk
       , secret_key const& sk);

given a byte range `v` and an optional byte range `salt`, a sequence number, public key `pk` (must be 32 bytes) and a secret key `sk` (must be 64 bytes), this function produces a signature which is written into a 64 byte buffer pointed to by `sig`. The caller is responsible for allocating the destination buffer that's passed in as the `sig` argument. Typically it would be allocated on the stack.

[report issue]

## announce_flags_t

Declared in "libtorrent/kademlia/announce_flags.hpp"

seed
  announce to DHT as a seed

implied_port
  announce to DHT with the implied-port flag set. This tells the network to use your source UDP port as your listen port, rather than the one specified in the message. This may improve the chances of traversing NATs when using uTP.

ssl_torrent
  Specify the port number for the SSL listen socket in the DHT announce.