Performance

ADV
%
Assigned
ADV2
%
Assigned
ADV3
%
Assigned

Assigned performance is the percentage of blocks minted versus leader slots assigned to the pool (not including stolen or ghosted blocks). Assigned performance is reduced by missing and invalid blocks.

We track a large number of metrics to ensure that the ADAvault stake pools are reliably minting blocks including:

  • assigned leader slots,
  • the ideal number of slots per epoch, based on pool stake,
  • luck, which is the ratio of assigned leader slots to ideal;
  • and lastly, the actual minted blocks confirmed on chain.

ADV

The graph shows blocks as leader, ideal block allocation based on stake, luck expressed as the percentage leader allocation against ideal; and lastly, the actual confirmed blocks which were created on the Cardano blockchain. We present data for the last 20 epochs.

EpochLeadIdealLuckConfirmMiss/Ghost/
Stolen/
Invalid*

* There were stolen and ghosted blocks in some Epochs where a predicted block was minted by another pool. This is a standard part of the Ouroboros Praos protocol decided based on a random function using the pool VRF key. See discussion on implementation here https://github.com/input-output-hk/ouroboros-network/issues/2014

As you can see in the data, there is considerable variation in leader slots. This is due to the Verifiably Random Function (VRF) used by the Cardano Node to assign blocks. The VRF will always average over time to 100% luck assuming that the pool is able to mint all blocks allocated.

ADV2

ADV2 shows a similar overall performance profile as ADV which is expected given the pool has similar stake and runs on an equivalent hardware and software stack. Again, please note there are a number of ghosted and stolen blocks which are more likely given the pool is close to 100% saturation.

EpochLeadIdealLuckConfirmMiss/Ghost/
Stolen/
Invalid

ADV3

Slot allocation as leader for ADV3 is more variable as the pool is smaller, average luck is now consistent with the larger pools but with greater variance across epochs. There are less stolen and ghosted blocks as the potential for VRF collisions with other pools is proportionately smaller.

EpochLeadIdealLuckConfirmMiss/Ghost/
Stolen/
Invalid

Key to headings
Leader: Scheduled to make block at this slot
Ideal: Expected/Ideal number of blocks assigned based on active stake (sigma)
Luck: The percentage of Leader slots assigned vs Ideal slots
Adopted: Block created successfully
Confirmed: Block created validated to be on-chain
Invalid: Node failed to create block
Missed: Scheduled at slot but no record of it in cncli DB and no other pool has made a block for this slot
Ghosted: Block created but marked as orphaned and no other pool has made a valid block for this slot, height battle or block propagation issue
Stolen: Another pool has a valid block registered on-chain for the same slot

Historic performance data is refreshed every hour, there will be some lag against the real time statistics for block production in the current epoch.

Monitoring

ADAvault uses Prometheus and Grafana to monitor the infrastructure that we operate, and collect data from node instances. Delegators have complete transparency on pool operation and performance, with access to the same real time monitoring dashboard as our Ops team.

Grafana- click to see the live dashboard. You may notice that metrics periodically reset when we move between clusters to patch or upgrade to a new Cardano release

Resource utilisation for memory network and CPU is low (even including the spike at epoch boundaries), and there is significant capacity on the existing hardware for transactions as smart contracts are rolled out. Performance tuning is performed periodically for releases with a number of improvements for time sync and file systems (xfs from ext4).

Stake pool operational availability remains at 100% and the Cardano node software has proven to be very stable, but we always allow for a comprehensive soak test on the passive nodes when compiling new releases.

Latency measurements

ADAvault has very low latency measurements for connectivity to peers. This is important as it increases the likelihood that the stake pools are able to mint all blocks allocated. Real time details are available from PoolTool.io who measure propagation delays across all registered pools.

ADAvault propagation delay for slot leader to broadcast to all network nodes
ADAvault propagation delay receiving from connected network nodes

The important number here is not the average (or median) latency at ~300ms, but the mode (or most frequently occurring latency) which is ~100ms. The reason the average is less relevant is that it is skewed by the very infrequent long tail which could be removed by connecting to close geographical peers, but has the side effect of making the network weaker and reducing global block propagation.

For comparison it takes approximately 200ms to send a ping request half way around the world. Therefore we can see that 100ms for the mode is towards the lower theoretical bound for latency.

We will be regularly reviewing global peer connectivity in the coming months, and are confident the pools have excellent performance characteristics for reliable block production.