Performance

ADAvault uses a combination of Prometheus and Grafana to to collect data from node instances, and monitor the active and passive clusters that we operate. Delegators have complete transparency on pool operation and performance, with access to the same real time monitoring dashboard that our Ops team uses.

Grafana Dashboard

Stake pool availability is at 100% (allowing for cluster switch time), nodes are on the latest release. Performance tuning is performed periodically for releases with a number of improvements in place for time sync and file systems (xfs from ext4).

You will notice that the ‘Node Uptime’ and ‘Blocks Forged’ metrics will periodically reset when we move from the active to the passive cluster. We perform this switch when we need to patch or upgrade the nodes to a new cardano release. By running multiple separate clusters we can minimise downtime to seconds while the switch occurs.

Resource utilisation for memory, network and CPU is low (even with the spike at Epoch boundaries) and there is significant capacity on the existing hardware for growth in transactions.

The cardano node software has proven to be very stable to date however we always allow for a comprehensive soak test on the passive nodes when compiling new releases.

Historic Performance

Checking historical performance using the pool algorithm allows us to confirm that the ADAvault ADV stake pool has minted all blocks allocated since it started operation*. Expected blocks are based on stake per Epoch, with the recent history of minted blocks (actual/statistical) below.

EpochLeadIdealLuckConfirmMissGhostStolenInvalid
262586194%
2616960115%
2606461104%6121
2598561140%832
258606198%591
2576056107%5811
256596097%563
2555857101%58
254535694%4823
253555698%532
2525654104%56
251505297%4712
2505449108%484
2495951116%563
2485148.1106%501
2475147.9106%465
* There were stolen blocks in some Epochs where a predicted block was minted by another pool. This is known as a “slot battle” and is standard part of the Ouroboros Praos protocol decided based on a random function using the pool VRF key. These times ADAvault lost. See discussion on implementation here https://github.com/input-output-hk/ouroboros-network/issues/2014

As you can see in the table above, there is a naturally occurring random variation in blocks allocated per Epoch (some are higher, some lower). This will average over time for pools producing allocated blocks to around 5-6% annual Return on ADA (ROA).

Historic performance for ADV2 is shown in the table below.

EpochLeadIdealLuckConfirmMissGhostStolenInvalid
2625452103%
261385076%
2605750114%5511
2596252119%593
2586053113%5712
257596197%59
256545598%5113
2555852112%544
254515299%51
253505198%4721
2523838100%362
2514130135%401
2502119111%21
249109107%10
24895.4165%9
24784.8166%8
* ADV2 has lost approximately 15 slot battles to date. There is no intrinsic technical difference between the pools, clashes are simply less probable when pools are less saturated. From this date we would expect to see equivalence between ADV and ADV2 given roughly the same stake.

We will add detailed information for ADV3 once it has produced blocks for 10 Epochs.

Latency measurements

ADAvault has very low latency measurements for connectivity to peers. This is important as it ensures that the stake pools are able to mint all blocks allocated. Real time details are available from PoolTool.io who measure propagation delays across all registered pools.

ADAvault propagation delay for slot leader to broadcast to all network nodes
ADAvault propagation delay receiving from connected network nodes

The important number here is not the average (or median) latency at ~300ms, but the mode (or most frequently occurring latency) which is ~100ms. The reason the average is less relevant is that it is skewed by the very infrequent long tail which could be removed by connecting to close geographical peers, but has the side effect of making the network weaker and reducing global block propagation.

For comparison it takes approximately 200ms to send a ping request half way around the world. Therefore we can see that 100ms for the mode is towards the lower theoretical bound for latency.

We will be regularly reviewing global peer connectivity in the coming months, and are confident the pools have excellent performance characteristics for reliable block production.