Summary

As of July 13, 2023, one of the four "public good" IPFS bootstrap nodes operated by Protocol Labs has been running rust-libp2p-server (opens new window) instead of Kubo (opens new window), which uses go-libp2p (opens new window). rust-libp2p-server is a thin wrapper around rust-libp2p (opens new window). We run both Kubo and rust-libp2p-server on IPFS bootstrap nodes to increase resilience. A bug or vulnerability is less likely to be in both Kubo and rust-libp2p-server than Kubo alone. In addition to increasing resilience, we gain experience running large rust-libp2p based deployments on the IPFS network.

IPFS Public DHT Bootstrap Nodes

What is an IPFS bootstrap node?

A Bootstrap Node is a trusted peer on the IPFS network through which an IPFS node learns about other peers on the network. [...]

See IPFS Glossary (opens new window).

A new node trying to join the "public IPFS DHT (opens new window)", i.e. trying to bootstrap, will:

1. Connect to its (pre-) configured bootstrap nodes.
2. Run some variation of the Kademlia bootstrap process (opens new window) which boils down to iteratively:
   1. Generating random IDs.
   2. Asking already discovered nodes whether they know anyone closer to those IDs.

Thus the only thing that an IPFS bootstrap node needs to do is:

• Allow incoming connections.
• Maintain a healthy Kademlia routing table.
• Reply to Kademlia FIND_NODE requests based on nodes in its routing table.

Let's dive a bit deeper. In the case of Kubo the DNSAddr (opens new window) addresses of the IPFS bootstrap nodes are shipped within the release binary.

// DefaultBootstrapAddresses are the hardcoded bootstrap addresses
// for IPFS. they are nodes run by the IPFS team. docs on these later.
// As with all p2p networks, bootstrap is an important security concern.
var DefaultBootstrapAddresses = []string{
	"/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
	"/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
	"/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
	"/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    // ...
}

See bootstrap_peers.go on github.com/ipfs/kubo (opens new window).

One can resolve those /dnsaddr/... through iterative DNS queries. Below is an example for the node with the peer ID QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb. This IPFS bootstrap node is running Kubo.

dig +short -t txt _dnsaddr.bootstrap.libp2p.io
"dnsaddr=/dnsaddr/am6.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb"
[...]

dig +short -t txt _dnsaddr.am6.bootstrap.libp2p.io
"dnsaddr=/ip6/2604:1380:4602:5c00::3/udp/4001/quic-v1/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb"
[...]

Finally connecting to the bootrap node shows us the protocols it supports.
Below example uses libp2p-lookup (opens new window) but ipfs swarm connect followed by ipfs id can be used instead.

libp2p-lookup direct --address /ip6/2604:1380:4602:5c00::3/udp/4001/quic-v1/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb

Lookup for peer with id PeerId("QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb") succeeded.

Protocol version: "ipfs/0.1.0"
Agent version: "kubo/0.20.0/b8c4725"
Listen addresses:
        - "/ip6/2604:1380:4602:5c00::3/udp/4001/quic-v1"
        - [...]
Protocols:
        - /ipfs/kad/1.0.0
        - [...]

Note the Agent version: "kubo/0.20.0/b8c4725" and the supported protocols Protocols: - /ipfs/kad/1.0.0.

Motivation

Why run both Kubo and rust-libp2p2-server bootstrap nodes?

This choice is influenced by three main areas: the benefit of diverse implementations, the opportunity to test rust-libp2p at large scale, and the presence of Rust in the IPFS network.

Implementation Diversity: Operating both Kubo and rust-libp2p-server bootstrap nodes contributes to the network's overall resilience and security. It's like having a second line of defense; if one system encounters an issue, the other is there to continue functioning. For instance, a recent bug impacted Kubo IPFS bootstrap nodes, closing incoming connections, right after their successful establishment, due to a QUIC version mismatch. By using both Kubo and rust-libp2p-server, we ensure that nodes can still join the network, even if one set of bootstrap nodes is unavailable.

Testing Rust-Libp2p at Large Scale: Our use of rust-libp2p-server also provides a valuable opportunity to examine how it behaves at a larger scale. Software performance can vary depending on scale, and these differences are hard to predict without actual real-world deployments. Now we can gain insights similar to those we acquired from other large deployments such as Polkadot and Ethereum (opens new window).

Encouraging Rust in the IPFS Network: Lastly, by operating a rust-libp2p bootstrap node, we hope to motivate other developers to build IPFS-based applications using rust-libp2p. This could lead to an increase in the use of Rust, fostering a more diverse and vibrant ecosystem.

rust-libp2p(-server) in Action

What is rust-libp2p(-server) and how does it operate as an IPFS bootstrap node?

rust-libp2p (opens new window) is an implementation of the libp2p specification in Rust, a popular systems programming language. The rust-libp2p project was initiated around 2018 (opens new window) and since then, it has powered network like Ethereum through its Rust implementation Lighthouse (opens new window) and Polkadot along with the Substrate (opens new window) ecosystem. You can find more rust-libp2p users here (opens new window).

rust-libp2p-server (opens new window) is just thin wrapper around rust-libp2p. It combines rust-libp2p's TCP, QUIC and Kademlia-DHT implementation into a single binary. Looking up the new rust-libp2p-server IPFS bootstrap node ny5 via libp2p-lookup (opens new window) confirms just that. Note the Agent version: "rust-libp2p-server/0.12.0". and Protocols: - /ipfs/kad/1.0.0.

libp2p-lookup direct --address /dnsaddr/ny5.bootstrap.libp2p.io

Lookup for peer with id PeerId("QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa") succeeded.

Protocol version: "ipfs/0.1.0"
Agent version: "rust-libp2p-server/0.12.0"
Listen addresses:
        - [...]
Protocols:
        - /ipfs/kad/1.0.0
        - [...]

Some Numbers

On the new bootstrap node we see around 15 new inbound connections per second. The majority of these connections are established via QUIC (see ip4/udp/quic).

The node is handling > 30k connections concurrently, thus being connected to roughly 15% of nodes of the public IPFS DHT (opens new window).

Across these connections the node handles around 40 Kademlia requests per second, most of which are Kademlia FIND_NODE requests.

The number of connections does not have a significant impact on CPU usage of the underlying machine.

The node uses < 300 kbyte of memory per connection.

A small tangent: in case you are interested in more IPFS public DHT metrics, take a look at the probelab DHT metrics and reports (opens new window).

Closing

If you want to learn more:

• Read up on the libp2p project (opens new window)
• Explore the rust-libp2p implementation (opens new window)
• See the thin rust-libp2p wrapper at mxinden/rust-libp2p-server (opens new window)
• And lastly, the public IPFS DHT measurements (opens new window) are always a good read

A lot of this work was done by @mcamou (opens new window) from the Protocol Labs EngRes Bifrost team (opens new window). Mario has handled the deployment and the team is operating the bootstrap nodes as a whole. Thanks, @mcamou (opens new window) and team!

FAQ:

• Do I have to use the default bootstrap nodes?

  No, you don't have to use /dnsaddr/bootstrap.libp2p.io. You can remove Protocol Labs' default nodes and add your own or use both for better reliability.

• Do we plan to run rust-libp2p-server on all IPFS bootstrap nodes?

  No.