Skip to content

Onboarding 🎓

💬 Please provide feedback on our onboarding guide. Let us know if any parts are unclear, confusing, or if you have other suggestions for improvement.

The goal of this project is to develop a framework in Go that can be used by a Polkadot node.

The Polkadot node is divided into two components, the Polkadot Runtime and the Polkadot Host. The Runtime handles the state transition logic for the Polkadot protocol and is designed to be upgradeable without the need of a fork. The Polkadot Host provides necessary functionality required for the Runtime to execute its state transition logic.

Currently, Host implementations can be developed in Rust (Substrate), C++ (Kagome) and Go (Gossamer). On the other hand, Runtime can be developed only in Rust (Substrate).

In this project, we focus only on the Runtime development in Go. The output of the Runtime code is Wasm bytecode, which can be plugged into any Host (you can consider this similar as how Solidity smart contracts are executed in Ethereum). The difference is that this bytecode takes care of the core functionality of a network.

1. Tech Stack 💘

1.1. WebAssembly 🏗️

WebAssembly (abbreviated as Wasm) is a binary instruction format for a stack-based virtual machine. It is designed as a compact, portable, and fast compilation target for high-level languages like C, C++, and Rust and many others that are being adapted. It enables execution of code at near-native speeds directly in web browsers and various host environments, as it resembles low-level machine code that modern CPUs understand.

Binary Format

WebAssembly code is delivered in a low-level binary format, which is more compact than its textual representation. This binary format is designed to be fast to decode and execute. There's also a textual representation of this binary format, which is useful for debugging and testing.

Low-Level Virtual Machine

WebAssembly provides a set of low-level virtual instructions that are closer to machine code than high-level programming languages. This enhances its execution efficiency and also makes it a suitable compilation target for other languages.

Typed Instructions

WebAssembly instructions are strongly typed. It supports several numerical types like i32, i64, f32, and f64 and a few others for handling memory and tables.

Stack-Based Architecture

Its computational model is designed around a stack-based architecture. Operations are performed by pushing and popping values from an implicit stack (the stack is inaccessible and distinct from the untrusted linear memory). Though it is not a pure stack machine, as it accommodates features like unlimited virtual registers (local variables).

Modules & Sections

In the WebAssembly binary format, code and data are organized into modules. Each module consists of various sections arranged in a specific order, though some sections are optional. The module structure is defined by the WebAssembly specification and is validated before execution.

  • Type Section - declares all function signatures used within the module.
  • Import Section - specifies all module imports, such as functions, memories, globals, and tables.
  • Function Section - contains a list of function declarations, each referencing the type section for its signature.
  • Table Section - declares the table of function references used by the module.
  • Global Section - declares global variables.
  • Export Section - specifies all module exports, such as functions, memories, globals, and tables.
  • Elem Section - initializes the table with function references for indirect function calls.
  • Code Section - contains the binary code for the module's functions.
  • Data Section - contains initial values for the module's memory.
  • Custom Section - contains custom data for the host environment, possibly containing toolchain-specific information. This section can appear multiple times and is not restricted to a specific position in the list of sections.

Modules can be dynamically loaded and combined, making it possible to build and manage larger applications effectively.

Memory Model

WebAssembly uses a single contiguous byte buffer as its memory model, referred to as linear memory. This memory is resizable, byte-addressable and is accessible by all memory operations.

Imported/Exported Memory

It is isolated from the host system, thus providing a safe environment for the execution of untrusted code. It can be imported or exported, thus facilitates reading and writing operations by both WebAssembly and the Host.

Functions

Every module may contain functions which can be either exported (made accessible outside the module) or imported (indicating dependency on an external function).

Exported/Imported Functions

The WebAssembly module and its Host communicate using host-imported and exported functions.

Host-imported functions act as an additional bridge between WebAssembly and its host, enabling the module to access resources, input-output operations, or system-specific functionality. An example would be to get the time of the machine the Host is currently running on.

On the other hand, exported functions enables customization, allowing developers to expose specific functionalities to the host environment. An example would be to export a runtime function, which takes care of the execution of blockchain transactions.

WebAssembly Extensions

WebAssembly has a minimal core specification, but it's designed to be extensible. Proposals like threads, garbage collection, and SIMD (single instruction, multiple data) operations are being worked on or have been added to provide more capabilities over time.

WebAssembly & JavaScript

A significant feature of WebAssembly is its seamless interaction with JavaScript. The two can work in tandem within web applications.

WebAssembly & WASI

Beyond browser capabilities, with the introduction of the WebAssembly System Interface (WASI), it can be integrated into a wide range of environments (Host), including web applications, desktop software, and more. In our case, we are going to embed it into another Rust/Go/C++ application. This is achieved by the WebAssembly module exposing a well-defined interface, facilitating communication with the host.

WebAssembly MVP & Polkadot

Polkadot uses a version of WebAssembly (Wasm MVP) that does not support reference types or multiple return types. Therefore, non-numeric values are exchanged through shared memory using pointer-sized allocations. This mechanism allows the WebAssembly module to interact manipulate data within the host's memory space, facilitating data exchange between the WebAssembly module and its host.

In the case of Polkadot, the WebAssembly bytecode takes care of the state transition and block execution of each Polkadot Node, which is the most critical part. It is plugged into the Polkadot Node and it is called Runtime. In case of bugs, upgrades or updates, the logic can just be replaced with a new WebAssembly bytecode. This allows the Runtime to be updated on-chain, without the need of a network fork.

  1. Documentation
  2. Intro to WebAssembly
  3. Install latest Go (1.21) - https://go.dev/dl/ or brew
  4. Guides
    1. Go to Wasm with JS apis - executed it in the browser
    2. Go to WASI

WebAssembly has different platform targets and extensions. Go supports Wasm depending on JavaScript supported APIs and with the release of Go 1.21, they’ve added support for WASI.

Unfortunately, Polkadot targets an old version of WebAssembly, called WebAssembly MVP, before spec version 1. This is why we will not use the Go toolchain for building wasm blobs, but TinyGo.

1.2. Go and TinyGo 🦖 🐣

TinyGo is a subset of Go with different goals from the standard Go. It is an alternative compiler and runtime aimed to support different small embedded devices and WebAssembly with a single processor core, emphasizing size optimizations.

  1. Documentation
  2. Install
  3. Guides
    1. TinyGo to Wasm with JS apis - compile a Wasm module and execute it inside a JS environment (browser)
    2. TinyGo to WASI - compile a Wasm blob and execute it inside another Go host application (Wazero VM)
      1. https://github.com/tetratelabs/wazero/tree/main/examples/allocation - check README and tinygo folder
      2. https://github.com/tetratelabs/wazero/tree/main/examples/import-go - check README

1.3. TinyGo fork 🧪

We have forked TinyGo as we need to add a new target for the Polkadot-specific wasm blob, targeting standalone Wasm MVP, similar to Rust's wasm32-unknown-unknown, without bulk memory operations and other extensions, also incorporating custom GC that utilizes an external allocator. In the polkawasm-target-dev branch, you can see the changes for the specific TinyGo releases.

We use a local build of TinyGo and do not depend on the already-built brew dependency.

Here are the steps how to install and build it locally.

After you have built TinyGo, execute the following:

tinygo version

The output should be similar to:

tinygo version 0.31.0-dev darwin/arm64 (using go version go1.21.6 and LLVM version 16.0.6)

2. Architecture of a Polkadot Node 🏛️

Now that you have learned about WebAssembly, shared memory, runtime imported/exported functions and the TinyGo toolchain, let’s look at the Polkadot specification.

Polkadot node architecture and protocol specification is heavily influenced by the tech stack: WebAssembly MVP and Rust. Some implementation details, like the memory management, are not well abstracted and tightly coupled with the Rust implementation and even included as part of the protocol specification.

2.1. Node 💻

Polkadot protocol has been divided into two parts, the Polkadot Runtime and the Polkadot Host.

2.2. Host-Runtime Interaction 🤝

Imported Functions 📥

External functions provided by the Host environment (Substrate/Kagome/Gossamer host) that Runtime (WebAssembly module) can invoke when needed, for more details check the Host API. The Host API provides access to memory, storage, crypto, hashing, logging and misc functionality.

Example (Storage):

Exported Functions 📤

Defined within the Runtime (WebAssembly module) and can be invoked by the Host application, for more details check the Runtime API.

The Runtime API provides core and chains specific functionality.

Example (Core API):

Memory 🧠

Shared between the Host and the Runtime and is managed by the Host allocator for all heap allocations. All data passed between Host and Runtime, like arguments to exported or imported functions or returned results, is encoded using SCALE encoding. Non numeric types, like byte buffers, are shared using a pointer-size to the allocation in the heap.

2.3. Runtime Internals ⚙️

Extrinsics (Transactions) 💳

Weights & Fees ⚖️ 💸

Accounts, addresses, and keys 👤 🔑

Storage 💾

Storing and retrieving data, key/value generation and types of storage values.

Pallets (Modules) 🧱

Pallets communicate and interact with each other via events, storage, calls, hooks, etc.

3. Specifics of implementing a Runtime in Go 🔍🐛

Developing a framework for writing Polkadot runtimes in Go is not a straight forward process, accompanied with many blockers and issues that need to be resolved. Most of the issues are related to the incompatibilities between the design decisions around the Polkadot protocol and the Go language. Here are some of the major challenges that we faced while working on the project:

  • Missing support for standalone Wasm (MVP) 🕳️
  • GC that is required to work with an external allocator, provided by the Host 💣
  • Immature toolchain based on custom Tinygo 🐣
  • SCALE codec implementation with minimal reflection 🪩
  • Writing mostly low-level and unsafe Go (none of the concurrency capabilities are utilized) ⚠️
  • The spec lacks details regarding the Runtime; thus, you should be able to read Rust code, which is the actual source of truth. 🦀

Most of the things are documented here Gosemble Runtime Architecture.

4. Tasks 📝

4.1. Compile a Runtime from Gosemble and run it in a Substrate node 🛠️

  1. Install git and docker
  2. Clone the Gosemble repo - git clone https://github.com/LimeChain/gosemble.git
  3. Checkout the development branch - git checkout develop
  4. Pull all necessary git submodules - git submodule update --init --recursive
  5. Build the runtime - make build-docker-benchmarking
  6. Run the tests - make test
  7. Start a local network - start a network
  8. Connect to Polkadot.js and do a simple transfer - transfer funds tutorial

4.2. Implement simple Runtime function and add tests 🛠️

  1. Declare a runtime exported function, example.
  2. Read a byte buffer passed as an argument, example.
  3. SCALE decode the byte buffer, example.
  4. Call 2-3 host imported functions, example, example.
  5. Return a byte buffer as a result, example.
  6. Add unit & integration tests, example.