Introduction

The proliferation of multiple Layer-1 chains in a bid to improve on the shortcomings of Bitcoin and Ethereum resulted in the Multi-chain era. This has given rise to one of the most debated issues in the industry— comparing blockchain performance. Where L1 chains market themselves as better performers. The rhetoric is typically centered around "chain A is faster than chain B,” with each claiming to outperform Ethereum or other chains. In times past, new entrant L1s have been touted as the “Ethereum killer,” churning out self-validated charts and vanity metrics. Unfortunately, these numbers are often self-reported in inconsistent or incomplete ways.

The question is, how can we objectively measure blockchain performance?

At first glance, this may seem like a simple question, but there’s more to it than meets the eye. In reality, there is no universally agreed-upon method to evaluate blockchain performance, as there are multiple dimensions to benchmarking it. In this piece, I attempt to highlight the challenges with performance evaluation of blockchain networks and also discuss key metrics in performance evaluation.

The Challenge With Benchmarking Performance

Firstly, the rigid decentralization requirements of blockchain reduce performance compared to classical transaction systems. To achieve decentralization, there are various architectural options for blockchain implementation, which involve trade-offs among various performance metrics. As such, benchmarking or performance testing of blockchain is not a straightforward exercise. By implication, this makes the comparison of performance metrics across various consensus protocols an imbalanced and biased approach; it is akin to comparing apples and oranges.

This highlights the need to adopt a more nuanced approach to assess and compare the performance of blockchains, one that evaluates consensus protocol variants and compares trade-offs across multiple dynamics.

Performance Metrics

Classically, the performance of a network or computer system relies heavily on two key metrics: latency and throughput. This holds when measuring the performance of blockchain systems.

Latency

Latency measures the time it takes for a transaction on the blockchain to be confirmed and included in a block. Typically, when a transaction is initiated, it requires time for the network to achieve consensus on its validity, and this time contributes to the overall latency. In traditional payment systems, transactions are finalized once the central system confirms them. However, in blockchains (such as PoS Ethereum), transactions go through multiple confirmation processes, resulting in a significantly higher latency compared to a single confirmation.

Measuring Latency in Blockchains

At first glance, latency appears uncomplicated: how long does it take for a transaction to be confirmed? But first, let’s examine some factors that impact latency in blockchains:

Block time: The block time, which represents the time interval between the creation of consecutive blocks, directly affects latency. A shorter block time allows for faster confirmation of transactions, reducing latency. Conversely, longer block times can increase latency, as transactions have to wait for a longer period before being included in a block and confirmed. The block time is determined by the specific blockchain protocol and can vary across different blockchain networks.

Batching: In most Layer 1 systems and blockchain networks, transactions are batched into blocks, which can introduce variable latency. Transactions that are included early in a block will experience shorter latency since they are confirmed as soon as the block is added to the blockchain. On the other hand, transactions that are initiated later and have to wait for the batch to fill up before being included in a block will experience longer latency.

Network Congestion: When network congestion occurs, it means that the rate at which transactions are posted exceeds the network's processing capacity. This can lead to increased latency as transactions have to wait for their turn to be processed and included in a block. The Poisson distribution provides a theoretical framework that gives insights into the behavior of transaction arrival patterns and subsequently latency.

Now, back to the question, how long does a transaction take to get confirmed?

There are many different points in time at which we can start and stop measuring latency, and each of these choices will give us a different answer.

For instance, we can start measuring latency from the client-side, when the user clicks the “submit” button on their computer, or we could start once the transaction gets to the mempool. We could also stop measuring latency when the transaction is in a proposed block. Furthermore, it is vital to note that latency also varies as a function of transaction fees paid. Users can pay higher fees to prioritize the inclusion of their transactions.

Throughput

Throughput is measured as the number of transactions that a system (Blockchain) handles per second. In essence, it measures the aggregate rate of transactions over time, expressed in transactions-per-seconds (TPS).

Typically, throughput is a network-wide metric, as such, it is more important to system designers; individual users are not concerned about throughput. What users are more concerned about are transaction latency and transaction costs — how quickly their transactions can be confirmed and as cheap as possible. While many computer systems are evaluated on a basis of performance, transaction fees are a somewhat new measure of performance for blockchain systems that doesn't really exist in traditional computer systems.

Measuring Throughput in Blockchains

How many transactions a system (blockchain) can process per second (TPS) has been the existing standard for measuring performance. However, the challenge lies in the definition of a "transaction", which leads to a fundamental issue: all transactions are not equal. Transactions have varying levels of complexity, with more complex transactions taking longer to process. A blockchain that processes a large number of simple transactions will perform differently than a blockchain that processes a small number of complex transactions.

For example, in Ethereum, transactions can include code that executes on the blockchain. This code can be used to create smart contracts, transfer tokens, or interact with other contracts. However, In Solana, transactions are executed in a different way. Transactions are written using the BPF environment, which is designed to be efficient for execution on the Solana blockchain.

In essence, there is no simple way to compare the total amount of work done by a set of EVM transactions to a set of Solana transactions. This makes it difficult to measure the throughput of a blockchain under different workloads. A better way to compare the throughput is to measure the amount of work that the blockchain can do per second.

Furthermore, consensus mechanism variance in different blockchains affect throughput. Typically, proof-of-work blockchains are slower than proof-of-stake blockchains. Let’s examine some other factors that should be considered.

Number of Validators

Validators are those who verify transactions on the blockchain in exchange for rewards. These validators generally dedicate a computer to the blockchain to maintain its integrity. Once a transaction is verified, it is added to the ledger of the blockchain.

Whenever a transaction is initiated on the blockchain, validators add it to a block for verification. Once the block is completed, it is then stored on the blockchain and cannot be modified anymore. All this work is done by validators. So, the more the number of validators, the better the performance of the blockchain.

Block Time

Block time is the amount of time required by validators or miners to verify the transactions stored in a block, and once it is complete, create another one. The total time elapsed between these steps comes out to be the block time. Also, miners or validators receive rewards for their participation in the form of cryptocurrency.

Conclusion

The goal of every (blockchain) system is to achieve the best possible performance, in other words, to minimize latency and response time while increasing the transaction throughput. However, due to the different factors involved, evaluating performance accurately can be pretty difficult. The safest way to carry this exercise out is by collecting and evaluating as much data as possible and making comparisons with other systems.