Network Status Documents
************************

Parsing for Tor network status documents. This supports both the v2
and v3 dir-spec. Documents can be obtained from a few sources…

* The ‘cached-consensus’ file in Tor’s data directory.

* Archived descriptors provided by CollecTor.

* Directory authorities and mirrors via their DirPort.

… and contain the following sections…

* document header

* list of "stem.descriptor.networkstatus.DirectoryAuthority"

* list of "stem.descriptor.router_status_entry.RouterStatusEntry"

* document footer

**For a great graphical overview see** Jordan Wright’s chart
describing the anatomy of the consensus.

Of these, the router status entry section can be quite large (on the
order of hundreds of kilobytes). As such we provide a couple of
methods for reading network status documents through "parse_file()".
For more information see "DocumentHandler()"…

   from stem.descriptor import parse_file, DocumentHandler

   with open('.tor/cached-consensus', 'rb') as consensus_file:
     # Processes the routers as we read them in. The routers refer to a document
     # with an unset 'routers' attribute.

     for router in parse_file(consensus_file, 'network-status-consensus-3 1.0', document_handler = DocumentHandler.ENTRIES):
       print router.nickname

**Module Overview:**

   NetworkStatusDocument - Network status document
     |- NetworkStatusDocumentV2 - Version 2 network status document
     |- NetworkStatusDocumentV3 - Version 3 network status document
     +- BridgeNetworkStatusDocument - Version 3 network status document for bridges

   KeyCertificate - Certificate used to authenticate an authority
   DocumentSignature - Signature of a document by a directory authority
   DetachedSignature - Stand alone signature used when making the consensus
   DirectoryAuthority - Directory authority as defined in a v3 network status document

class stem.descriptor.networkstatus.PackageVersion

   Bases: "stem.descriptor.networkstatus.PackageVersion"

   Latest recommended version of a package that’s available.

   Variables:
      * **name** (*str*) – name of the package

      * **version** (*str*) – latest recommended version

      * **url** (*str*) – package’s url

      * **digests** (*dict*) – mapping of digest types to their
        value

class stem.descriptor.networkstatus.SharedRandomnessCommitment

   Bases: "stem.descriptor.networkstatus.SharedRandomnessCommitment"

   Directory authority’s commitment for generating the next shared
   random value.

   Variables:
      * **version** (*int*) – shared randomness protocol version

      * **algorithm** (*str*) – hash algorithm used to make the
        commitment

      * **identity** (*str*) – authority’s sha1 identity fingerprint

      * **commit** (*str*) – base64 encoded commitment hash to the
        shared random value

      * **reveal** (*str*) – base64 encoded commitment to the shared
        random value, **None** of not provided

class stem.descriptor.networkstatus.DocumentDigest

   Bases: "stem.descriptor.networkstatus.DocumentDigest"

   Digest of a consensus document.

   New in version 1.8.0.

   Variables:
      * **flavor** (*str*) – consensus type this digest is for (for
        example, ‘microdesc’)

      * **algorithm** (*str*) – hash algorithm used to make the
        digest

      * **digest** (*str*) – digest value of the consensus

class stem.descriptor.networkstatus.NetworkStatusDocument(contents, lazy_load=False)

   Bases: "stem.descriptor.Descriptor"

   Common parent for network status documents.

   digest(hash_type='SHA1', encoding='HEX')

      Digest of this descriptor’s content. These are referenced by…

         * **DetachedSignature**

           * Referer: "DetachedSignature" **consensus_digest**
             attribute

           * Format: **SHA1/HEX**

      New in version 1.8.0.

      Parameters:
         * **hash_type** (*stem.descriptor.DigestHash*) – digest
           hashing algorithm

         * **encoding** (*stem.descriptor.DigestEncoding*) – digest
           encoding

      Returns:
         **hashlib.HASH** or **str** based on our encoding argument

class stem.descriptor.networkstatus.NetworkStatusDocumentV2(raw_content, validate=False)

   Bases: "stem.descriptor.networkstatus.NetworkStatusDocument"

   Version 2 network status document. These have been deprecated and
   are no longer generated by Tor.

   Variables:
      * **routers** (*dict*) – fingerprints to "RouterStatusEntryV2"
        contained in the document

      * **version** (*int*) – ***** document version

      * **hostname** (*str*) – ***** hostname of the authority

      * **address** (*str*) – ***** authority’s IP address

      * **dir_port** (*int*) – ***** authority’s DirPort

      * **fingerprint** (*str*) – ***** authority’s fingerprint

      * **contact** (*str*) – ***** authority’s contact information

      * **signing_key** (*str*) – ***** authority’s public signing
        key

      * **client_versions** (*list*) – list of recommended client
        tor version strings

      * **server_versions** (*list*) – list of recommended server
        tor version strings

      * **published** (*datetime*) – ***** time when the document
        was published

      * **options** (*list*) – ***** list of things that this
        authority decides

      * **signing_authority** (*str*) – ***** name of the authority
        signing the document

      * **signature** (*str*) – ***** authority’s signature for the
        document

   ***** attribute is either required when we’re parsed with
   validation or has a default value, others are left as **None** if
   undefined

   TYPE_ANNOTATION_NAME = 'network-status-2'

   classmethod content(attr=None, exclude=(), sign=False)

      Creates descriptor content with the given attributes. Mandatory
      fields are filled with dummy information unless data is
      supplied. This doesn’t yet create a valid signature.

      New in version 1.6.0.

      Parameters:
         * **attr** (*dict*) – keyword/value mappings to be included
           in the descriptor

         * **exclude** (*list*) – mandatory keywords to exclude from
           the descriptor, this results in an invalid descriptor

         * **sign** (*bool*) – includes cryptographic signatures and
           digests if True

      Returns:
         **str** with the content of a descriptor

      Raises:
         * **ImportError** if cryptography is unavailable and sign
           is True

         * **NotImplementedError** if not implemented for this
           descriptor type

class stem.descriptor.networkstatus.NetworkStatusDocumentV3(raw_content, validate=False, default_params=True)

   Bases: "stem.descriptor.networkstatus.NetworkStatusDocument"

   Version 3 network status document. This could be either a vote or
   consensus.

   Variables:
      * **routers** (*dict*) – fingerprint to "RouterStatusEntryV3"
        mapping for relays contained in the document

      * **version** (*int*) – ***** document version

      * **version_flavor** (*str*) – ***** flavor associated with
        the document (such as ‘ns’ or ‘microdesc’)

      * **is_consensus** (*bool*) – ***** **True** if the document
        is a consensus

      * **is_vote** (*bool*) – ***** **True** if the document is a
        vote

      * **is_microdescriptor** (*bool*) – ***** **True** if this is
        a microdescriptor flavored document, **False** otherwise

      * **valid_after** (*datetime*) – ***** time when the consensus
        became valid

      * **fresh_until** (*datetime*) – ***** time when the next
        consensus should be produced

      * **valid_until** (*datetime*) – ***** time when this
        consensus becomes obsolete

      * **vote_delay** (*int*) – ***** number of seconds allowed for
        collecting votes from all authorities

      * **dist_delay** (*int*) – ***** number of seconds allowed for
        collecting signatures from all authorities

      * **client_versions** (*list*) – list of recommended client
        tor versions

      * **server_versions** (*list*) – list of recommended server
        tor versions

      * **packages** (*list*) – ***** list of "PackageVersion"
        entries

      * **known_flags** (*list*) – ***** list of "Flag" for the
        router’s flags

      * **params** (*dict*) – ***** dict of parameter(**str**) =>
        value(**int**) mappings

      * **directory_authorities** (*list*) – ***** list of
        "DirectoryAuthority" objects that have generated this document

      * **signatures** (*list*) – ***** "DocumentSignature" of the
        authorities that have signed the document

   **Consensus Attributes:**

   Variables:
      * **consensus_method** (*int*) – method version used to
        generate this consensus

      * **bandwidth_weights** (*dict*) – dict of weight(str) =>
        value(int) mappings

      * **shared_randomness_current_reveal_count** (*int*) – number
        of commitments used to generate the current shared random
        value

      * **shared_randomness_current_value** (*str*) – base64 encoded
        current shared random value

      * **shared_randomness_previous_reveal_count** (*int*) – number
        of commitments used to generate the last shared random value

      * **shared_randomness_previous_value** (*str*) – base64
        encoded last shared random value

   **Vote Attributes:**

   Variables:
      * **consensus_methods** (*list*) – list of ints for the
        supported method versions

      * **published** (*datetime*) – time when the document was
        published

      * **flag_thresholds** (*dict*) – ***** mapping of internal
        performance thresholds used while making the vote, values are
        **ints** or **floats**

      * **recommended_client_protocols** (*dict*) – recommended
        protocols for clients

      * **recommended_relay_protocols** (*dict*) – recommended
        protocols for relays

      * **required_client_protocols** (*dict*) – required protocols
        for clients

      * **required_relay_protocols** (*dict*) – required protocols
        for relays

      * **bandwidth_file_headers** (*dict*) – headers from the
        bandwidth authority that generated this vote

      * **bandwidth_file_digest** (*dict*) – hashes of the bandwidth
        authority file used to generate this vote, this is a mapping
        of hash functions to their resulting digest value

   ***** attribute is either required when we’re parsed with
   validation or has a default value, others are left as None if
   undefined

   Changed in version 1.4.0: Added the packages attribute.

   Changed in version 1.5.0: Added the
   is_shared_randomness_participate, shared_randomness_commitments,
   shared_randomness_previous_reveal_count,
   shared_randomness_previous_value,
   shared_randomness_current_reveal_count, and
   shared_randomness_current_value attributes.

   Changed in version 1.6.0: Added the recommended_client_protocols,
   recommended_relay_protocols, required_client_protocols, and
   required_relay_protocols attributes.

   Changed in version 1.6.0: The is_shared_randomness_participate and
   shared_randomness_commitments were misdocumented in the tor spec
   and as such never set. They’re now an attribute of votes in the
   **directory_authorities**.

   Changed in version 1.7.0: The
   shared_randomness_current_reveal_count and
   shared_randomness_previous_reveal_count attributes were
   undocumented and not provided properly if retrieved before their
   shred_randomness_*_value counterpart.

   Changed in version 1.7.0: Added the bandwidth_file_headers
   attributbute.

   Changed in version 1.8.0: Added the bandwidth_file_digest
   attributbute.

   classmethod content(attr=None, exclude=(), sign=False, authorities=None, routers=None)

      Creates descriptor content with the given attributes. Mandatory
      fields are filled with dummy information unless data is
      supplied. This doesn’t yet create a valid signature.

      New in version 1.6.0.

      Parameters:
         * **attr** (*dict*) – keyword/value mappings to be included
           in the descriptor

         * **exclude** (*list*) – mandatory keywords to exclude from
           the descriptor, this results in an invalid descriptor

         * **sign** (*bool*) – includes cryptographic signatures and
           digests if True

      Returns:
         **str** with the content of a descriptor

      Raises:
         * **ImportError** if cryptography is unavailable and sign
           is True

         * **NotImplementedError** if not implemented for this
           descriptor type

   classmethod create(attr=None, exclude=(), validate=True, sign=False, authorities=None, routers=None)

      Creates a descriptor with the given attributes. Mandatory fields
      are filled with dummy information unless data is supplied. This
      doesn’t yet create a valid signature.

      New in version 1.6.0.

      Parameters:
         * **attr** (*dict*) – keyword/value mappings to be included
           in the descriptor

         * **exclude** (*list*) – mandatory keywords to exclude from
           the descriptor, this results in an invalid descriptor

         * **validate** (*bool*) – checks the validity of the
           descriptor’s content if **True**, skips these checks
           otherwise

         * **sign** (*bool*) – includes cryptographic signatures and
           digests if True

      Returns:
         "Descriptor" subclass

      Raises:
         * **ValueError** if the contents is malformed and validate
           is True

         * **ImportError** if cryptography is unavailable and sign
           is True

         * **NotImplementedError** if not implemented for this
           descriptor type

   type_annotation()

      Provides the Tor metrics annotation of this descriptor type. For
      example, “@type server-descriptor 1.0” for server descriptors.

      Please note that the version number component is specific to
      CollecTor, and for the moment hardcode as 1.0. This may change
      in the future.

      New in version 1.8.0.

      Returns:
         "TypeAnnotation" with our type information

   is_valid()

      Checks if the current time is between this document’s
      **valid_after** and **valid_until** timestamps. To be valid
      means the information within this document reflects the current
      network state.

      New in version 1.8.0.

      Returns:
         **True** if this consensus is presently valid and **False**
         otherwise

   is_fresh()

      Checks if the current time is between this document’s
      **valid_after** and **fresh_until** timestamps. To be fresh
      means this should be the latest consensus.

      New in version 1.8.0.

      Returns:
         **True** if this consensus is presently fresh and **False**
         otherwise

   validate_signatures(key_certs)

      Validates we’re properly signed by the signing certificates.

      New in version 1.6.0.

      Parameters:
         **key_certs** (*list*) – "KeyCertificates" to validate the
         consensus against

      Raises:
         **ValueError** if an insufficient number of valid signatures
         are present.

   get_unrecognized_lines()

      Provides a list of lines that were either ignored or had data
      that we did not know how to process. This is most common due to
      new descriptor fields that this library does not yet know how to
      process. Patches welcome!

      Returns:
         **list** of lines of unrecognized content

   meets_consensus_method(method)

      Checks if we meet the given consensus-method. This works for
      both votes and consensuses, checking our ‘consensus-method’ and
      ‘consensus-methods’ entries.

      Parameters:
         **method** (*int*) – consensus-method to check for

      Returns:
         **True** if we meet the given consensus-method, and **False**
         otherwise

class stem.descriptor.networkstatus.DirectoryAuthority(raw_content, validate=False, is_vote=False)

   Bases: "stem.descriptor.Descriptor"

   Directory authority information obtained from a v3 network status
   document.

   Authorities can optionally use a legacy format. These are no longer
   found in practice, but have the following differences…

   * The authority’s nickname ends with ‘-legacy’.

   * There’s no **contact** or **vote_digest** attribute.

   Variables:
      * **nickname** (*str*) – ***** authority’s nickname

      * **v3ident** (*str*) – ***** identity key fingerprint used to
        sign votes and consensus

      * **hostname** (*str*) – ***** hostname of the authority

      * **address** (*str*) – ***** authority’s IP address

      * **dir_port** (*int*) – ***** authority’s DirPort

      * **or_port** (*int*) – ***** authority’s ORPort

      * **is_legacy** (*bool*) – ***** if the authority’s using the
        legacy format

      * **contact** (*str*) – contact information, this is included
        if is_legacy is **False**

   **Consensus Attributes:**

   Variables:
      **vote_digest** (*str*) – digest of the authority that
      contributed to the consensus, this is included if is_legacy is
      **False**

   **Vote Attributes:**

   Variables:
      * **legacy_dir_key** (*str*) – fingerprint of and obsolete
        identity key

      * **key_certificate**
        (*stem.descriptor.networkstatus.KeyCertificate*) – *****
        authority’s key certificate

      * **is_shared_randomness_participate** (*bool*) – *****
        **True** if this authority participates in establishing a
        shared random value, **False** otherwise

      * **shared_randomness_commitments** (*list*) – ***** list of
        "SharedRandomnessCommitment" entries

      * **shared_randomness_previous_reveal_count** (*int*) – number
        of commitments used to generate the last shared random value

      * **shared_randomness_previous_value** (*str*) – base64
        encoded last shared random value

      * **shared_randomness_current_reveal_count** (*int*) – number
        of commitments used to generate the current shared random
        value

      * **shared_randomness_current_value** (*str*) – base64 encoded
        current shared random value

   ***** mandatory attribute

   Changed in version 1.4.0: Renamed our ‘fingerprint’ attribute to
   ‘v3ident’ (prior attribute exists for backward compatability, but
   is deprecated).

   Changed in version 1.6.0: Added the
   is_shared_randomness_participate, shared_randomness_commitments,
   shared_randomness_previous_reveal_count,
   shared_randomness_previous_value,
   shared_randomness_current_reveal_count, and
   shared_randomness_current_value attributes.

   classmethod content(attr=None, exclude=(), sign=False, is_vote=False)

      Creates descriptor content with the given attributes. Mandatory
      fields are filled with dummy information unless data is
      supplied. This doesn’t yet create a valid signature.

      New in version 1.6.0.

      Parameters:
         * **attr** (*dict*) – keyword/value mappings to be included
           in the descriptor

         * **exclude** (*list*) – mandatory keywords to exclude from
           the descriptor, this results in an invalid descriptor

         * **sign** (*bool*) – includes cryptographic signatures and
           digests if True

      Returns:
         **str** with the content of a descriptor

      Raises:
         * **ImportError** if cryptography is unavailable and sign
           is True

         * **NotImplementedError** if not implemented for this
           descriptor type

   classmethod create(attr=None, exclude=(), validate=True, sign=False, is_vote=False)

      Creates a descriptor with the given attributes. Mandatory fields
      are filled with dummy information unless data is supplied. This
      doesn’t yet create a valid signature.

      New in version 1.6.0.

      Parameters:
         * **attr** (*dict*) – keyword/value mappings to be included
           in the descriptor

         * **exclude** (*list*) – mandatory keywords to exclude from
           the descriptor, this results in an invalid descriptor

         * **validate** (*bool*) – checks the validity of the
           descriptor’s content if **True**, skips these checks
           otherwise

         * **sign** (*bool*) – includes cryptographic signatures and
           digests if True

      Returns:
         "Descriptor" subclass

      Raises:
         * **ValueError** if the contents is malformed and validate
           is True

         * **ImportError** if cryptography is unavailable and sign
           is True

         * **NotImplementedError** if not implemented for this
           descriptor type

class stem.descriptor.networkstatus.KeyCertificate(raw_content, validate=False)

   Bases: "stem.descriptor.Descriptor"

   Directory key certificate for a v3 network status document.

   Variables:
      * **version** (*int*) – ***** version of the key certificate

      * **address** (*str*) – authority’s IP address

      * **dir_port** (*int*) – authority’s DirPort

      * **fingerprint** (*str*) – ***** authority’s fingerprint

      * **identity_key** (*str*) – ***** long term authority
        identity key

      * **published** (*datetime*) – ***** time when this key was
        generated

      * **expires** (*datetime*) – ***** time after which this key
        becomes invalid

      * **signing_key** (*str*) – ***** directory server’s public
        signing key

      * **crosscert** (*str*) – signature made using certificate’s
        signing key

      * **certification** (*str*) – ***** signature of this key
        certificate signed with the identity key

   ***** mandatory attribute

   TYPE_ANNOTATION_NAME = 'dir-key-certificate-3'

   classmethod content(attr=None, exclude=(), sign=False)

      Creates descriptor content with the given attributes. Mandatory
      fields are filled with dummy information unless data is
      supplied. This doesn’t yet create a valid signature.

      New in version 1.6.0.

      Parameters:
         * **attr** (*dict*) – keyword/value mappings to be included
           in the descriptor

         * **exclude** (*list*) – mandatory keywords to exclude from
           the descriptor, this results in an invalid descriptor

         * **sign** (*bool*) – includes cryptographic signatures and
           digests if True

      Returns:
         **str** with the content of a descriptor

      Raises:
         * **ImportError** if cryptography is unavailable and sign
           is True

         * **NotImplementedError** if not implemented for this
           descriptor type

class stem.descriptor.networkstatus.DocumentSignature(method, identity, key_digest, signature, flavor=None, validate=False)

   Bases: "object"

   Directory signature of a v3 network status document.

   Variables:
      * **method** (*str*) – algorithm used to make the signature

      * **identity** (*str*) – fingerprint of the authority that
        made the signature

      * **key_digest** (*str*) – digest of the signing key

      * **signature** (*str*) – document signature

      * **flavor** (*str*) – consensus type this signature is for
        (such as ‘microdesc’), **None** if for the standard consensus

   Parameters:
      **validate** (*bool*) – checks validity if **True**

   Raises:
      **ValueError** if a validity check fails

class stem.descriptor.networkstatus.DetachedSignature(raw_content, validate=False)

   Bases: "stem.descriptor.Descriptor"

   Stand alone signature of the consensus. These are exchanged between
   directory authorities when determining the next hour’s consensus.

   Detached signatures are defined in section 3.10 of the dir-spec,
   and only available to be downloaded for five minutes between minute
   55 until the end of the hour.

   New in version 1.8.0.

   Variables:
      * **consensus_digest** (*str*) – ***** digest of the consensus
        being signed

      * **valid_after** (*datetime*) – ***** time when the consensus
        became valid

      * **fresh_until** (*datetime*) – ***** time when the next
        consensus should be produced

      * **valid_until** (*datetime*) – ***** time when this
        consensus becomes obsolete

      * **additional_digests** (*list*) – ***** "DocumentDigest" for
        additional consensus flavors

      * **additional_signatures** (*list*) – *****
        "DocumentSignature" for additional consensus flavors

      * **signatures** (*list*) – ***** "DocumentSignature" of the
        authorities that have signed the document

   ***** mandatory attribute

   TYPE_ANNOTATION_NAME = 'detached-signature-3'

   classmethod content(attr=None, exclude=(), sign=False)

      Creates descriptor content with the given attributes. Mandatory
      fields are filled with dummy information unless data is
      supplied. This doesn’t yet create a valid signature.

      New in version 1.6.0.

      Parameters:
         * **attr** (*dict*) – keyword/value mappings to be included
           in the descriptor

         * **exclude** (*list*) – mandatory keywords to exclude from
           the descriptor, this results in an invalid descriptor

         * **sign** (*bool*) – includes cryptographic signatures and
           digests if True

      Returns:
         **str** with the content of a descriptor

      Raises:
         * **ImportError** if cryptography is unavailable and sign
           is True

         * **NotImplementedError** if not implemented for this
           descriptor type

class stem.descriptor.networkstatus.BridgeNetworkStatusDocument(raw_content, validate=False)

   Bases: "stem.descriptor.networkstatus.NetworkStatusDocument"

   Network status document containing bridges. This is only available
   through the metrics site.

   Variables:
      * **routers** (*dict*) – fingerprint to "RouterStatusEntryV3"
        mapping for relays contained in the document

      * **published** (*datetime*) – time when the document was
        published

   TYPE_ANNOTATION_NAME = 'bridge-network-status'
