Ethereum mainstream client: Geth overall architecture

This article is the first in the Geth source code series. Through this series, we will build a framework for studying Geth implementations, and developers can use this framework to delve into the parts they are interested in. This series has six articles. In the first article, we will study the design architecture of the execution layer client Geth and the startup process of the Geth node. Geth code is updated very quickly, and the code you see later may be different, but the overall design is generally consistent, and the new code can also be read with the same idea.

01\Ethereum Client

Before the Merge upgrade, Ethereum only had one client, which was responsible for the execution of transactions and the consensus of the blockchain, ensuring that the blockchain generates new blocks in a certain order. After the Merge upgrade, the Ethereum client is divided into an execution layer and a consensus layer. The execution layer is responsible for the execution of transactions, the maintenance of status and data, and the consensus layer is responsible for the implementation of consensus functions. The execution layer and the consensus layer communicate through an API. The execution layer and the consensus layer have their own specifications. Clients can be implemented in different languages, but they must comply with the corresponding specifications. Geth is an implementation of the execution layer client. The current mainstream execution layer and consensus layer clients are implemented as follows:

Execution Layer

• Geth: Maintained by a team directly funded by the Ethereum Foundation, developed in Go, it is recognized as the most stable and time-tested client
• Nethermind: Developed and maintained by the Nethermind team, developed in C#, and funded by the Ethereum Foundation and Gitcoin community in the early stages
• Besu: originally developed by the PegaSys team at ConsenSys, now a Hyperledger community project, developed in Java
• Erigon: Developed and maintained by the Erigon team, funded by the Ethereum Foundation and BNB Chain. Forked from Geth in 2017, the goal is to improve synchronization speed and disk efficiency.
• Reth: Developed by Paradigm, using Rust as the development language, emphasizing modularity and high performance. It is now mature and can be used in production environments.

Consensus Layer

• Prysm: Maintained by Prysmatic Labs, it is one of the earliest consensus layer clients of Ethereum. It is developed in Go language, focuses on usability and security, and was funded by the Ethereum Foundation in the early days.
• Lighthouse: Maintained by the Sigma Prime team, developed in Rust, focusing on high performance and enterprise-level security, suitable for high-load scenarios
• Teku: Developed by the PegaSys team of ConsenSys, and later became part of the Hyperledger Besu community, using the Java language
• Nimbus: Developed and maintained by the Status Network team, using the Nim language, optimized for resource-constrained devices (such as mobile phones and IoT devices), with the goal of achieving lightweight operation in embedded systems

02\Introduction to the Executive Level

The Ethereum execution layer can be regarded as a transaction-driven state machine. The most basic function of the execution layer is to update state data by executing transactions through EVM. In addition to transaction execution, there are also functions such as saving and verifying blocks and state data, running p2p networks and maintaining transaction pools.

Transactions are generated by users (or programs) in the format defined by the Ethereum execution layer specification. Users need to sign transactions. If the transaction is legal (continuous Nonce, correct signature, sufficient gas fee, correct business logic), then the transaction will eventually be executed by the EVM, thereby updating the state of the Ethereum network. The state here refers to a collection of data structures, data, and databases, including external account addresses, contract addresses, address balances, and code and data.

The execution layer is responsible for executing transactions and maintaining the status after transaction execution, and the consensus layer is responsible for selecting which transactions to execute. EVM is the state transition function in this state machine. The input of the function comes from multiple places, which may come from the latest block information provided by the consensus layer, or from blocks downloaded from the p2p network.

The consensus layer and the execution layer communicate through the Engine API, which is the only way for the execution layer and the consensus layer to communicate. If the consensus layer obtains the right to produce a block, it will ask the execution layer to produce a new block through the Engine API. If it does not obtain the right to produce a block, it will synchronize the latest block for the execution layer to verify and execute, thereby maintaining consensus with the entire Ethereum network.

The execution layer can be logically divided into 6 parts:

• EVM: Responsible for executing transactions. Transaction execution is also the only way to modify the state number.
• Storage: responsible for the storage of state, block and other data
• Transaction pool: used for transactions submitted by users, temporarily stored, and propagated between different nodes through the p2p network
• p2p network: used for discovering nodes, synchronizing transactions, downloading blocks, etc.
• RPC service: provides the ability to access nodes, such as users sending transactions to nodes, and interactions between the consensus layer and the execution layer
• BlockChain: responsible for managing Ethereum's blockchain data

The following diagram shows the key processes of the execution layer and the functions of each part:

For the execution layer (here we only discuss Full Node), there are three key processes:

• If it is a new node joining Ethereum, it needs to synchronize blocks and status data from other nodes through the p2p network. If it is Full Sync, it will download blocks one by one starting from the genesis block, verify the blocks and rebuild the status database through EVM. If it is Snap Sync, it will skip the entire block verification process and directly download the latest checkpoint status data and subsequent block data.
• If it is a node that has been synchronized to the latest state, it will continue to obtain the latest output block from the consensus layer through the Engine API, verify the block, and then execute all transactions in the block through the EVM to update the state database and write the block to the local chain.
• If the node has been synchronized to the latest state and the consensus layer has obtained the right to produce blocks, it will drive the execution layer to produce the latest block through the Engine API. The execution layer obtains transactions from the transaction pool and executes them, and then assembles them into blocks and passes them to the consensus layer through the Engine API. The consensus layer broadcasts the blocks to the consensus layer p2p network.

03\Source code structure

The code structure of go-ethereum is huge, but a lot of it is auxiliary code and unit testing. When studying the Geth source code, you only need to focus on the core implementation of the protocol. The functions of each module are as follows. You need to focus on core, eth, ethdb, node, p2p, rlp, trie & triedb modules:

• accounts: manage Ethereum accounts, including the generation of public and private key pairs, signature verification, address derivation, etc.
• beacon: handles the interaction logic with the Ethereum Beacon Chain and supports the post-merge function of the Proof of Stake (PoS) consensus
• build: build scripts and compilation configuration (such as Dockerfile, cross-platform compilation support)
• cmd: command line tool entry, including multiple sub-commands
• common: general tool class, such as byte processing, address format conversion, mathematical functions
• consensus: defines consensus engines, including previous proof of work (Ethash), single-machine proof of stake (Clique), and Beacon engines, etc.
• console: Provides an interactive JavaScript console that allows users to interact directly with the Ethereum node through the command line (such as calling Web3 APIs, managing accounts, and querying blockchain data)
• core: the core logic of the blockchain, handling the life cycle management of blocks/transactions, state machines, Gas calculations, etc.
• crypto: Encryption algorithm implementation, including elliptic curve (secp256k1), hash (Keccak-256), signature verification
• docs: Documentation (such as design specifications, API descriptions)
• eth: A complete implementation of the Ethereum network protocol, including node services, block synchronization (such as fast synchronization, archive mode), transaction broadcasting, etc.
• ethclient: implements the Ethereum client library, encapsulating the JSON-RPC interface for Go developers to interact with Ethereum nodes (such as querying blocks, sending transactions, and deploying contracts)
• ethdb: database abstraction layer, supports LevelDB, Pebble, memory database, etc., stores blockchain data (blocks, status, transactions)
• ethstats: collects and reports node operation status to the statistics service to monitor network health
• Event: implements event subscription and publishing mechanisms, supporting asynchronous communication between modules within the node (such as new block arrival, transaction pool update)
• graphql: Provides GraphQL interface and supports complex queries (replacing some JSON-RPC functions)
• internal: internal tools or code with limited external access
• log: log system, supports hierarchical log output and contextual log recording
• mertrics: performance metrics collection (supported by Prometheus)
• Miner: Mining-related logic, generating new blocks and packaging transactions (in PoW scenarios)
• Node: node service management, integrating the startup and configuration of p2p, RPC, database and other modules
• p2p: peer-to-peer network protocol implementation, supporting node discovery, data transmission, and encrypted communication
• params: defines Ethereum network parameters (mainnet, testnet, genesis block configuration)
• rlp: implements Ethereum's dedicated data serialization protocol RLP (Recursive Length Prefix), which is used to encode/decode data structures such as blocks and transactions
• rpc: implements JSON-RPC and IPC interfaces for external programs to interact with nodes
• signer: transaction signature management (hardware wallet integration)
• tests: integration tests and status tests to verify protocol compatibility
• trie & triedb: Merkle Patricia Trie implementation for efficient storage and management of account status and contract storage

04\Execution layer module division

There are two ways to access the Geth node from the outside, one is through RPC, and the other is through Console. RPC is suitable for external users, and Console is suitable for node managers. But whether through RPC or Console, they use the internally encapsulated capabilities, which are built in a layered manner.

The outermost layer is the API for external access to the various capabilities of the node, the Engine API is used for communication between the execution layer and the consensus layer, the Eth API is used for external users or programs to send transactions and obtain block information, and the Net API is used to obtain the status of the p2p network, etc. For example, if a user sends a transaction through the API, then the transaction will eventually be submitted to the transaction pool and managed by the transaction pool. For example, if a user needs to obtain a block of data, then the database capabilities need to be called to obtain the corresponding block.

The next layer of the API is the realization of core functions, including transaction pool, transaction packaging, block output, synchronization of blocks and status, etc. These functions need to rely on lower-level capabilities, such as transaction pool, synchronization of blocks and status need to rely on the capabilities of the p2p network, the generation of blocks and blocks synchronized from other nodes need to be verified before they can be written to the local database, which requires the capabilities of EVM and data storage.

Execution layer core data structure

Ethereum

The Ethereum structure in eth/backend.go is an abstraction of the entire Ethereum protocol, which basically includes the main components of Ethereum, but EVM is an exception. It will be instantiated every time a transaction is processed and does not need to be initialized with the entire node. Ethereum in the following text refers to this structure:

type Ethereum struct { // Ethereum configuration, including chain configurationconfig *ethconfig.Config // Transaction pool, the user's transaction goes to the transaction pool after submissiontxPool *txpool.TxPool // Used to track and manage local transactionslocalTxTracker *locals.TxTracker // Blockchain structureblockchain *core.BlockChain // It is the core component of the network layer of the Ethereum node, responsible for handling all communications with other nodes, including block synchronization, transaction broadcasting and reception, and managing peer node connectionshandler *handler // Responsible for node discovery and node source managementdiscmix *enode.FairMix // Responsible for persistent storage of blockchain datachainDb ethdb.Database // Responsible for processing the publication and subscription of various internal eventseventMux *event.TypeMux // Consensus engineengine consensus.Engine // Managing user accounts and keysaccountManager *accounts.Manager // Managing log filters and block filtersfilterMaps *filtermaps.FilterMaps // Used to safely close filterMaps channel to ensure that resources are properly cleaned up when the node is shut downcloseFilterMaps chan chan struct{} // Provide backend support for the RPC APIAPIBackend *EthAPIBackend // Under PoS, collaborate with the consensus engine to verify blocksminer *miner.Miner // Minimum gas price accepted by the nodegasPrice *big.Int // Network ID networkID uint64 // Provide network-related RPC services, allowing network status to be queried through RPCnetRPCService *ethapi.NetAPI // Manage P2P network connections, handle node discovery and connection establishment, and provide underlying network transmission functionsp2pServer *p2p.Server // Protect concurrent access to mutable fieldslock sync.RWMutex // Track whether the node is shut down normally and help recover after abnormal shutdownTracker *shutdowncheck.ShutdownTracker }

Node

Node in node/node.go is another core data structure. As a container, it is responsible for managing and coordinating the operation of various services. In the following structure, you need to pay attention to the lifecycles field. Lifecycle is used to manage the lifecycle of internal functions. For example, the Ethereum abstraction above needs to rely on Node to start and register in lifecycles. This can separate specific functions from the node abstraction and improve the scalability of the entire architecture. This Node needs to be distinguished from the Node in devp2p.

type Node struct { eventmux *event.TypeMux config *Config // Account manager, responsible for managing wallets and accounts accman *accounts.Manager log log.Logger keyDir string keyDirTemp bool dirLock *flock.Flock stop chan struct{} // p2p network instance server *p2p.Server startStopLock sync.Mutex // Tracking node life cycle status (initialization, running, closed) state int lock sync.Mutex // All registered backends, services and auxiliary services lifecycles []Lifecycle // Currently provided API list rpcAPIs []rpc.API // Different access methods provided for RPC http *httpServer ws *httpServer httpAuth *httpServer wsAuth *httpServer ipc *ipcServer inprocHandler *rpc.Server databases map[*closeTrackingDB]struct{} }

If we look at the execution layer of Ethereum from an abstract dimension, Ethereum, as a world computer, needs to include three parts: network, computing, and storage. Then the components corresponding to these three parts in the Ethereum execution layer are:

• Network: devp2p
• Calculation: EVM
• Storage: ethdb

devp2p

Ethereum is essentially a distributed system, where each node is connected to other nodes through a p2p network. The implementation of the p2p network protocol in Ethereum is devp2p.

devp2p has two core functions: one is node discovery, which enables nodes to establish connections with other nodes when accessing the network; the other is data transmission service, which allows nodes to exchange data after establishing connections with other nodes.

The Node structure in p2p/enode/node.go represents a node in the p2p network, where the enr.Record structure stores key-value pairs of node details, including identity information (signature algorithm and public key used by node identity), network information (IP address, port number), supported protocol information (such as support for eth/68 and snap protocols) and other custom information. This information is encoded in RLP. The specific specifications are defined in eip-778:

type Node struct { // Node record, containing various properties of the node enr.Record // Unique identifier of the node, 32 bytes in length id ID // hostname Tracking the DNS name of the node hostname string // IP address of the node ip netip.Addr // UDP port udp uint16 // TCP port tcp uint16 }// enr.Recordtype Record struct { // Sequence number seq uint64 // Signature signature []byte // RLP encoded record raw []byte // Sorted list of all key-value pairs pairs []pair }

The Table structure in p2p/discover/table.go is the core data structure of the devp2p node discovery protocol. It implements a distributed hash table similar to Kademlia and is used to maintain and manage node information in the network.

printf("type Table struct { mutex sync.Mutex // Index known nodes by distance buckets [nBuckets]*bucket // Boot node nursery []*enode.Node rand reseedingRandom ips netutil.DistinctNetSet revalidation tableRevalidation // Database of known nodes db *enode.DB net transport cfg Config log log.Logger // Periodically handle various events in the network refreshReq chan chan struct{} revalResponseCh chan revalidationResponse addNodeCh chan addNodeOp addNodeHandled chan bool trackRequestCh chan trackRequestOp initDone chan struct{} closeReq chan struct{} closed chan struct{} // Interface for adding and removing nodes nodeAddedHook func(*bucket, *tableNode) nodeRemovedHook func(*bucket, *tableNode)} world!");

ethdb

ethdb abstracts Ethereum data storage and provides a unified storage interface. The underlying database can be leveldb, pebble, or other databases. There can be many extensions as long as the interface remains unified.

Some data (such as block data) can be directly read and written to the underlying database through the ethdb interface. Other data storage interfaces are built on the basis of ethdb. For example, a large part of the data in the database is status data. This data will be organized into an MPT structure. The corresponding implementation in Geth is trie. During the operation of the node, trie data will generate a lot of intermediate states. These data cannot be directly called to read and write ethdb. Triedb is needed to manage these data and intermediate states, and finally persist them through ethdb.

The read and write interfaces of the underlying database are defined in ethdb/database.go, but the specific implementation is not included. The specific implementation will be implemented by different databases themselves. For example, leveldb or pebble database. Two layers of data read and write interfaces are defined in Database, among which the KeyValueStore interface is used to store active and frequently changing data, such as the latest blocks, status, etc. AncientStore is used to process historical block data, which rarely changes once written.

//Top-level interface of the database type Database interface { KeyValueStore AncientStore} //Read and write interface for KV data type KeyValueStore interface { KeyValueReader KeyValueWriter KeyValueStater KeyValueRangeDeleter Batcher Iteratee Compacter io.Closer} //Interface for reading and writing old data type AncientStore interface { AncientReader AncientWriter AncientStater io.Closer}

EVM

EVM is the state transition function of the Ethereum state machine. All state data updates can only be performed through EVM. The p2p network can receive transaction and block information, which will become part of the state database after being processed by EVM. EVM shields the differences in the underlying hardware, allowing programs to get consistent results when executed on EVMs on different platforms. This is a very mature design method, and the JVM in the Java language is also a similar design.

The implementation of EVM has three main components. The EVM structure in core/vm/evm.go defines the overall structure and dependencies of EVM, including execution context, state database dependencies, etc. The EVMInterpreter structure in core/vm/interpreter.go defines the implementation of the interpreter, which is responsible for executing EVM bytecodes. The Contract structure in core/vm/contract.go encapsulates the specific parameters of the contract call, including the caller, contract code, input, etc., and all current opcodes are defined in core/vm/opcodes.go:

// EVMtype EVM struct { // Block context, including block-related informationContext BlockContext // Transaction context, including transaction-related informationTxContext // State database, used to access and modify account statusStateDB StateDB // Current call depth depth int // Chain configuration parameterschainConfig *params.ChainConfig chainRules params.Rules // EVM configurationConfig Config // Bytecode interpreterinterpreter *EVMInterpreter // Abort flagabort atomic.Bool callGasTemp uint64 // Precompiled contract mappingprecompiles map[common.Address]PrecompiledContract jumpDests map[common.Hash]bitvec }type EVMInterpreter struct { // Points to the EVM instance evm *EVM // Opcode jump tabletable *JumpTable // Keccak256 hasher instance, shared between opcodeshasher crypto.KeccakState // Keccak256 hash result bufferhasherBuf common.Hash // Is it read-only mode? Status modification is not allowed in read-only modereadOnly bool // Return data of the last CALL, for subsequent reusereturnData []byte }type Contract struct { // Caller addresscaller common.Address // Contract addressaddress common.Address jumpdests map[common.Hash]bitvec analysis bitvec // Contract bytecodeCode []byte // Code hashCodeHash common.Hash // Call inputInput []byte // Is it a contract deploymentIsDeployment bool // Is it a system callIsSystemCall bool // Available gasGas uint64 // Amount of ETH attached to the callvalue *uint256.Int }

Other module implementations

The functions of the execution layer are implemented in a layered manner, and other modules and functions are built on the basis of these three core components. Here are some core modules.

Under eth/protocols there are implementations of the current Ethereum p2p network subprotocols. There are eth/68 and snap subprotocols, which are built on devp2p.

eth/68 is the core protocol of Ethereum. The protocol name is eth, and 68 is its version number. Then, based on this protocol, functions such as transaction pool (TxPool), block synchronization (Downloader) and transaction synchronization (Fetcher) are implemented. The snap protocol is used to quickly synchronize blocks and status data when a new node joins the network, which can greatly reduce the time it takes to start a new node.

ethdb provides the read and write capabilities of the underlying database. Since there are many complex data structures in the Ethereum protocol, it is impossible to manage these data directly through ethdb, so rawdb and statedb are implemented on ethdb to manage blocks and state data respectively.

EVM runs through all the main processes. Whether it is block construction or block verification, EVM is required to execute transactions.

05\Geth node startup process

The startup of Geth is divided into two stages. The first stage will initialize the components and resources required to start the node. The second stage will officially start the node and then provide external services.

Node initialization

When starting a geth node, the following code is involved:

The initialization of each module is as follows:

• cmd/geth/main.go: geth node startup entry
• cmd/geth/config.go (makeFullNode): load configuration and initialize node

• node/node.go: the core container for initializing the Ethereum node

1. node.rpcstack.go: Initialize the RPC module
2. accounts.manager.go: Initialize accountManager

• eth/backend.go: Initialize Ethereum instance

1. node/node.go OpenDatabaseWithFreezer: Initialize chaindb
2. eth/ethconfig/config.go: Initialize the consensus engine instance (the consensus engine here does not actually participate in the consensus, but only verifies the results of the consensus layer and processes the withdrawal request of the validator)
3. core/blockchain.go: Initialize blockchain
4. core/filterMaps.go: Initialize filtermaps
5. core/txpool/blobpool/blobpool.go: Initialize blob transaction pool
6. core/txpool/legacypool/legacypool.go: Initialize the normal transaction pool
7. cord/txpool/locals/tx_tracker.go: local transaction tracking (need to configure to enable local transaction tracking, local transactions will be processed with higher priority)
8. eth/handler.go: Initialize the Handler instance of the protocol
9. miner/miner.go: module for instantiating transaction packaging (original mining module)
10. eth/api_backend.go: Instantiate RPC service
11. eth/gasprice/gasprice.go: instantiate gas price query service
12. internal/ethapi/api.go: Instantiates the P2P network RPC API
13. node/node.go(RegisterAPIs): Register RPC API
14. node/node.go(RegisterProtocols): Register P2P Protocols
15. node/node.go(RegisterLifecycle): Register the lifecycle of each component

• cmd/utils/flags.go(RegisterFilterAPI): Register Filter RPC API
• cmd/utils/flags.go(RegisterGraphQLService): Register the GraphQL RPC API (if configured)
• cmd/utils/flags.go(RegisterEthStatsService): Register EthStats RPC API (if configured)
• eth/catalyst/api.go: Register Engine API

The node initialization will be completed in makeFullNode in cmd/geth/config.go, focusing on initializing the following three modules

In the first step, the Node structure in node/node.go will be initialized, which is the entire node container. All functions need to run in this container. The second step will initialize the Ethereum structure, which includes the implementation of various core functions of Ethereum. Ethereum also needs to be registered in Node. The third step is to register the Engine API in Node.

Node initialization creates a Node instance, and then initializes the p2p server, account management, and http protocol ports exposed to the outside.

Ethereum initialization is much more complicated, and most of the core functions are initialized here. First, ethdb is initialized and the chain configuration is loaded from the storage, and then the consensus engine is created. The consensus engine here does not perform consensus operations, but only verifies the results returned by the consensus layer. If a withdrawal request occurs in the consensus layer, the actual withdrawal operation will also be completed here. Then the Block Chain structure and transaction pool are initialized.

After all these are completed, the handler will be initialized. The handler is the processing entry for all p2p network requests, including transaction synchronization, block downloading, etc. It is a key component for Ethereum to achieve decentralized operation. After all these are completed, some sub-protocols implemented on the basis of devp2p, such as eth/68, snap, etc., will be registered in the Node container. Finally, Ethereum will be registered in the Node container as a lifecycle, and Ethereum initialization is completed.

Finally, the initialization of the Engine API is relatively simple, just registering the Engine API to the Node. At this point, the node initialization is complete.

Node startup

After completing the node initialization, you need to start the node. The process of starting the node is relatively simple. You only need to start all the registered RPC services and Lifecycles, and then the entire node can provide services to the outside world.

06\Summary

Before we deeply understand the implementation of Ethereum's execution layer, we need to have an overall understanding of Ethereum. Ethereum as a whole can be regarded as a transaction-driven state machine. The execution layer is responsible for the execution of transactions and state changes, and the consensus layer is responsible for driving the execution layer to run, including allowing the execution layer to produce blocks, determine the order of transactions, vote for blocks, and make blocks final. Since this state machine is decentralized, it needs to communicate with other nodes through the p2p network to jointly maintain the consistency of state data.

The execution layer is not responsible for determining the order of transactions, but only for executing transactions and recording the state changes after the transactions are executed. There are two forms of records here, one is to record all state changes in the form of blocks, and the other is to record the current state in the database. At the same time, the execution layer is also the entry point for transactions, and the transaction pool is used to store transactions that have not been packaged into blocks. If other nodes need to obtain block, state and transaction data, the execution layer will send this information through the p2p network.

For the execution layer, there are three core modules: computing, storage, and networking. Computing corresponds to the implementation of EVM, storage corresponds to the implementation of ethdb, and networking corresponds to the implementation of devp2p. With this overall understanding, you can deeply understand each submodule without getting lost in the specific details.

07\Ref

[1]https://ethereum.org/zh/what-is-ethereum/

[2]https://epf.wiki/#/wiki/protocol/architecture

[3]https://clientdiversity.org/#distribution

[4]https://github.com/ethereum/devp2p

[5]https://github.com/ethereum/execution-specs

[6]https://github.com/ethereum/consensus-specs

·END·

Contents | Ray

Editing & Formatting | Huanhuan

Design | Daisy