Improving client diversity with Vero
Client Diversity
The Ethereum network stands out for having multiple production-ready client implementations in various languages, across both the consensus and execution layers. In theory, this diversity in implementations makes Ethereum more resilient to client bugs, ensuring the network continues to function even if one implementation has a bug.
In practice, client usage distribution has been far from ideal for some time, with many node operators running the same client implementation, exposing their validators to significant risk. Until recently—and possibly still1—a bug in the most widely used execution layer client could have had disastrous consequences. The potential consequences were so severe that some began advocating for a form of bailout in these situations, where penalties would need to be manually reduced. Regardless of the eventual outcome, it’s safe to say the situation would be extremely chaotic.
At Serenita, we’ve been acutely aware of these risks and have thus far addressed them by exclusively using minority clients. This solution worked well as long as we could be reasonably sure that the clients we were using weren’t being used by too many others. However, there are edge cases where using a client utilized by just over 33% of the network can still result in losses. Exclusively using minority clients was not a good enough long-term solution for us, and soon we started looking into multi-node validator clients.
Multi-node validator clients use data from multiple beacon nodes, making it possible to avoid single-client bugs through the use of multiple client implementations.
Switching to existing multi-node validator clients proved challenging, requiring key migrations and other complex steps. The technical complexity, combined with other risk factors (see below), led us to build our own validator client.
Design Goals
Our goal was to develop a validator client that would:
combine data from multiple clients
Mitigating the impact of single-client bugs, regardless of their actual usage on the network, was the most important goal. We wanted to stop relying on self-reported client usage data, and protect ourselves against single-client bugs.
be simple
Working out which validator duties to perform, performing them at the correct time, and nothing more. This results in a highly specialized piece of software with a low surface area for bugs.
be secure
By outsourcing validator private key management to a remote signer, we reduce the attack and bug surface and ensure slashing protection relies on a proven and battle-tested mechanism.
The codebase should remain small with a minimal list of external dependencies.
be highly observable
When debugging validator performance or missed block proposals, it’s crucial to determine exactly what went wrong.
Logs, metrics and tracing data allow operators to dive into as much detail as they want, working out exactly which step of the process took longer than it should have.
be compatible with all consensus and execution layer clients
When we began this project, subtle incompatibilities existed between validator clients and beacon nodes developed by different teams. Fortunately, many of these issues have since been resolved.
Developing our own validator client ensures timely compatibility with all beacon nodes. For compatibility testing we use the same software as the EF DevOps team - Kurtosis and its ethereum-package.
be compatible with the Ethereum remote signing API / web3signer
Being able to keep using their existing remote signer infrastructure makes it easier for operators to start (or stop) using the validator client.
make it easy to switch from traditional validator clients
One of our key goals was to keep switching costs low, encouraging other node operators to adopt it and making not only our own infrastructure, but the entire ecosystem more resilient.
In case an issue does occur with our validator client, it is possible to switch to a different validator client quickly without slashing risk.
Introducing Vero
Vero is a multi-node validator client, now released as a fully open-source public good.
Its codebase consists of just ~2,100 lines of source code, with only 9 direct and 35 total external dependencies.
Vero currently powers thousands of our Holesky testnet and Gnosis Chain validators2. After months of testing on local devnets and public testnets, we have reached the point where we want to encourage other operators to try it out, especially if they haven't adopted multi-node setups yet.
When compared to existing validator clients, Vero is most similar to Vouch.
Why not simply use Vouch?
Vouch is a great and reliable piece of software used by many professional node operators. We chose not to use it for the following reasons:
Cost of switching
Switching to Vouch isn’t trivial, as it requires migrating validator keys to Dirk, Attestant's remote signer, among other steps.
Risk
Since no other validator clients support using Dirk as a remote signer, any significant issue in Vouch or Dirk would be hard to recover from and would likely require a fix from its maintainers. Many other professional node operators use Vouch, meaning any downtime caused by such a bug would possibly be followed by inactivity leak penalties that are activated when a large part of the network goes offline.
In this regard, Vouch has somewhat become a victim of its own success, much like Geth on the execution layer client side, though the consequences and risks of using Vouch are significantly lower.
What about DVT?
We are fans of the benefits DVT technology brings, and in many cases, we believe it makes a lot of sense to run distributed validators — especially when multiple operators are involved, as it reduces risks and simplifies coordination. However, running validators via DVT does have its downsides, and the cost of switching to such setups is significant.
For more details on the trade-offs, refer to the feature comparison in the documentation.
How Vero works with multiple beacon nodes
As a multi-node validator client, Vero performs best when connected to multiple beacon nodes, preferably with different client implementations.
Attestations
When it’s time to attest, Vero requests attestation data from all its configured beacon nodes. An attestation is only produced if a majority of the connected beacon nodes agrees on the head of the chain.
Block proposals
Vero requests all of its configured beacon nodes to produce blocks, and publishes the block with the highest value.
Future work
No software is ever truly finished, and Vero is no exception. In the near future, we plan to add support for the upcoming Pectra fork.
On Serenita's infrastructure side, we will be working towards what we believe to be the ultimate professional node operator setup:
Are you ready to try out Vero?
Visit Vero's project page and start running Vero yourself!
1) Measuring client usage is challenging, and fallback mechanisms may influence the attestation data that validators ultimately sign.
2) Using Vero on Gnosis Chain ensures our managed validators do not attest to a buggy supermajority chain. A supermajority client bug could realistically happen since Nethermind is likely used by a supermajority of validators on Gnosis Chain. We are looking forward to other clients supporting Gnosis Chain. Until then, we believe the safest thing is to run Vero against both Erigon and Nethermind.
About Serenita
Serenita is a leading blockchain infrastructure operator specializing in secure and reliable solutions for managing and growing digital assets, with a primary focus on Ethereum staking. Our seasoned team combines years of experience, technical expertise, and industry knowledge to ensure optimal performance and best-in-class digital asset security. Serenita is committed to contribute to the broader ecosystem by continuing to support home stakers, running minority clients and taking any future measures that benefit the network at large.
We guide our customers through the technical complexities of Ethereum staking, providing non-custodial solutions with the flexibility to stop staking at any time. Reach out to discover how Serenita can help you manage and grow your digital assets, providing expert-driven solutions tailored to your needs.