All the Measurement Tools

Dennis Trautwein
All the Measurement Tools
#tools #probelab #measurements
Share this item:

At ProbeLab, we have made it our mission to measure networks and network protocols. However, measurements are not an end in itself but rather a means to an end. Measurements allow us to identify bottlenecks, quantify the available space for improvement, and eventually design and suggest protocol optimizations. Then the circle repeats.

Our work focuses on internal network protocol logic, cross-protocol interoperation and network architecture. Due to the decentralized nature of peer-to-peer networks, recording activities across all participants is challenging. Particularly, since independent node operators dominate, no complete record of network activities exists. To address this challenge, a significant component of our work consists of developing suitable measurement methodologies and corresponding tools.

However, many of the tools come from a vibrant community of collaborators. In this blog post, we want to

  • improve the discoverability of those tools to
  • limit duplicate work,
  • highlight each tool's impact, and
  • give credit where credit is due.

Below, you can find a Registry of all the tools we have used in the past or are even running continuously today. Each entry comprises a brief paragraph describing the features, functionality and purpose of the tool. Where applicable each entry also has subsection highlighting the impact by point to reports, publications or issues where the tool was instrumental.

# Table of Contents

# Highlights

Next to the registry we want to highlight three tools that have proven immensly valuable in the past.

# CID-Hoarder

# Nebula

Nebula Logo

# Thunderdome

Thunderdome (opens new window) is a system to compare the performance of different versions of IPFS gateways by using real HTTP traffic streamed from the public gateways operated by Protocol Labs.

A user can define an experiment that details what software they want to put under test, with any special configuration and test conditions. Each combination of software and configuration is called a target. Thunderdome deploys each target, connects them to the IPFS network and begins sending real-world requests at the chosen rate. Each target in the experiment receives exactly the same request at the same time.

Thunderdome collects metrics from each target and sends them to Grafana where they can be graphed and analysed. Once the experiment is done the deployed target are shut down to save resources, but they can be started once again if the experiment, or a variant, needs to be repeated. We use it to compare different implementations of the gateway spec, the impact of changing configuration settings or look for performance changes between releases.

Thunderdome differs from other tooling such as testground (opens new window) because it aims to simulate realistic load conditions using close to real time requests.

# Registry

# Thunderdome

Author: @ProbeLab (opens new window) | GitHub: dennis-tra/nebula (opens new window) | Languages: Go (opens new window)

Thunderdome (opens new window) is a system to compare the performance of different versions of IPFS gateways by using real HTTP traffic streamed from the public gateways operated by Protocol Labs.

A user can define an experiment that details what software they want to put under test, with any special configuration and test conditions. Each combination of software and configuration is called a target. Thunderdome deploys each target, connects them to the IPFS network and begins sending real-world requests at the chosen rate. Each target in the experiment receives exactly the same request at the same time.

Thunderdome differs from other tooling such as testground (opens new window) because it aims to simulate realistic load conditions using close to real time requests.

Impact

  • Thunderdome is run since Kubo v0.19 prior every release cycle to detect regressions before they get released

# Nebula

Author: @ProbeLab (opens new window) | GitHub: dennis-tra/nebula (opens new window) | Languages: Go (opens new window)

Nebula is a libp2p DHT crawler and monitor that is designed to track the liveliness and availability of peers. ProbeLab is running Nebula every 30m for various libp2p-based DHT networks. We gather information about peer uptime which translates to peer churn. These measurements guide informed decisions about certain network-wide DHT parameters like the replication factor of records. Nebula supports the Amino DHT (opens new window), Filecoin, Polkadot, Kusama, and more (opens new window).

Impact

# Parsec

Author: @ProbeLab (opens new window) | GitHub: plprobelab/parsec (opens new window) | Languages: Go (opens new window)

parsec is a DHT and IPNI (opens new window) performance measurement tool that is used to gather accurate data on the performance of DHT and IPNI lookups and publications. parsec-based experiments are aimed at improving the efficiency and speed of distributed systems by developing better algorithms for routing and data retrieval.

We at ProbeLab use it to continuously monitor the lookup and publication performance to spot degradations early as has happened earlier in 2023 (opens new window).

Impact

# Tiros

Author: @ProbeLab (opens new window) | GitHub: plprobelab/tiros (opens new window) | Languages: Go (opens new window)

ProbeLab has built a website performance measurement tool, called tiros for websites hosted on IPFS. Tiros is designed to help developers monitor and optimize the performance of their IPFS-hosted websites. Tiros-based experiments measure retrieval and rendering metrics of websites loaded over IPFS. It also measures and compares the IPFS metrics with their HTTPS counterparts.

Impact

# Boomo

Author: @ProbeLab (opens new window) | GitHub: plprobelab/boomo (opens new window) | Languages: Go (opens new window)

boomo is a bootrapper monitor and will probe a list of preconfigured
peers at a constant frequency with different transports. It was developed to alert on issues
with network bootstrappers.

Impact

  • Uncovered issue with websocket address resolution (TODO: link issue)

# Antares

Author: @dennis-tra (opens new window) | GitHub: dennis-tra/antares (opens new window) | Languages: Go (opens new window)

A gateway and pinning service probing tool. It allows to track information about the peers that are powering those services. This includes but is not limited to PeerIDs, agent versions, supported protocols, and geo-locations. Antares will generate random CIDs, write provide records to Amino (opens new window), and then request them through gateways or pin them with pinning services. That way we can identify the IP addresses and peer IDs of these services.

This methodology is inspired by "Mapping the interplanetary Filesystem". // TODO

# Kademlia Exporter

Author: @mxinden (opens new window) | GitHub: mxinden/ipfs-cid-hoarder (opens new window) | Languages: Rust (opens new window)

# CID-Hoarder

Author: @cortze (opens new window) | GitHub: cortze/ipfs-cid-hoarder (opens new window) | Languages: Go (opens new window) (Python/Jupyter)

The CID-Hoarder is a client designed for monitoring Content IDs (CIDs) within the Amino (opens new window) network. It accomplishes this by randomly generating Content+CID pairs and storing them in the network. It then periodically checks the accessibility of these CIDs by pinging the provider record holders, requesting the CIDs they should store, and conducting DHT lookups to determine if the content can be retrieved from the network. Additionally, the CID-Hoarder identifies the closest peers to the CIDs in question, providing comprehensive tracking and monitoring capabilities within the IPFS ecosystem.

Impact

# IPFS Telemetry

Author: @diogo464 (opens new window) | GitHub: diogo464/ipfs_telemetry (opens new window) | Languages: Go (opens new window)

This tool is a modified instance of Kubo and has been developed to provide telemetry information through a newly implemented protocol. This instance collects various metrics from Bitswap, Kademlia, the host system, and Go. Instead of transmitting this data to a central source, it exposes telemetry information using the newly established protocol. Additionally, it includes a network crawler that can retrieve telemetry data from nodes that communicate using this protocol.

Impact

# IPFS Metrics Exporter

Author: @trudi-group (opens new window) | GitHub: trudi-group/ipfs-metric-exporter (opens new window) | Languages: Go (opens new window)

This tool is a Kubo plugin (opens new window) that starts a TCP server over which it exposes protocol information like exchanged Bitswap messages. The plugin also expses an HTTP server that extends the existing Prometheus metrics and has an endpoint to manually trigger certain Bitswap operations.

Impact

# IPFS Tools

Author: @trudi-group (opens new window) | GitHub: trudi-group/ipfs-tools (opens new window) | Languages: Rust (opens new window)

The same group that developed the "IPFS Metrics Exporter" provide a treasure chest of analysis related tools. Most importantly, it contains a client for the above TCP server to consume and analyse Bitswap messages in real-time. Further packages include cid-decode, ipfs-gateway-finder, ipfs-json-to-csv, bitswap-monitoring-client, unify-bitswap-traces. Antares was inspired by the methodology pioneered in ipfs-gateway-finder.

Impact

# Geolocation of IPFS Users/Content

Author: @pedroAkos (opens new window) | GitHub: pedroAkos/IPFS-location-requested-content (opens new window) | Languages: Go (opens new window) (Python)

This repository contains analysis code for Bitswap and gateway request logs like they were published in Design and Evaluation of IPFS to correlate requestor to content provider location. The contained tool takes the gateway request logs, looks up their provider record in the DHT, and maps the provider peer IDs to their network addresses by also looking up their peer records in the DHT. That way the provider geo location can be determined based on their IP address. This information in combination with the client's IP from the gateway request paints a comprehensive view of the request patterns in the IPFS network.

Impact

  • RFM-3 (opens new window) - Location of IPFS end users and requested content
  • Paper submitted to DADS 2023 - SAC 2023 // TODO: ask pedro

# Testground

Author: @protocol (opens new window) | GitHub: testground/testground (opens new window) | Languages: Go (opens new window)

Testground is a platform for testing, benchmarking, and simulating distributed and peer-to-peer systems at scale. It's designed to be multi-lingual and runtime-agnostic, scaling gracefully from 2 to 10k instances, only when needed.

# Binary Trie Library

Author: @guillaumemichel (opens new window) | GitHub: guillaumemichel/py-binary-trie (opens new window) | Languages: Python | Pypi: binary-trie (opens new window)

Pythin implementation of a binary trie to experiment with binary trie structures used in the XOR keyspace of the Kademlia DHT. It builds binary tries for arbitrary sized keys and can efficiently compute the `N`` closest keys to a specific key in the trie. It supports the standard trie operations: contain, find, prefix, etc. This is a handy tool when simulating DHT interactions.

Impact