K3 Wasm Internal Specifications

If you’re new to the K3 framework and are just trying to make applications with it this document is not meant for you. This document outlines the way our Rust SDK works internally and what the output format for compiled K3 apps looks like. It can be useful if you intend to port the SDK to another language or are trying to build and compile apps for our network from scratch.

Output format & handlers

Let’s start at the beginning, when you look at a basic Rust K3 app all you see is something like this:

use k3_wasm_macros::http_handler;
use k3_wasm_sdk::http::{Request, Response};

#[http_handler]
pub fn get(_req: Request<Vec<u8>>) -> Response<Vec<u8>> {
    Response::builder()
        .status(200)
        .header("Content-Type", "text/plain")
        .body("Hello K3!".as_bytes().to_vec())
        .unwrap()
}

k3_wasm_macros::init!();

How exactly is this compiled into a final .wasm binary? All the user has to do is call the Rust compiler’s native command to output Wasm: cargo build --target wasm32-wasi and everything just works. How do K3 executors know which export’s in your module tree correspond to which routes?

There are some constraints in Wasm that lead to us even needing such a specification:

• Wasm binaries are flat (i.e. only one module, no nested module structure)

• Wasm types are only primitives (just integers, floats, and pointers)

So the K3 SDK here needs to take this nested module tree you provide and flatten it out into a list of exports, each of which can take a number as an input and return one as output. For example the export in the first code snippet would be translated to something like:

#[no_mangle]
extern "C" pub fn __k3_handler__get(req_ptr: u32) -> u32 {
	// ...
}

If you added an API directory to your source code and had a /api/users/mod.rs file with another get export like previously we would translate that to:

#[no_mangle]
extern "C" pub fn __k3_handler_api_users_get(req_ptr: u32) -> u32 {
	// ...
}

The expected naming format is __k3_handler_{PREFIX}_{METHOD} where the prefix is the expected HTTP route with _ replacing the /s and the method is the HTTP method (GET, POST, PUT, etc.). Executors will translate the HTTP URL they get in their request into this format and look for the appropriate handler, if one is not found a 404 responses is returned with more details about the route. As you can see the executor is also expected to provide the request they have as a pointer to this export and translate back another pointer into a response, this brings us nicely to serialization.

Serialization & deserialization

The base shared translation unit used across the layers of abstraction is the buffer. Specifically a length prefixed buffer. This means given a pointer into a memory to get a buffer in your target language, you read the first 4 bytes to get the length (in little endian) and then allocate and copy in memory to a buffer based on that. For example in Rust this is what we do:

unsafe fn resolve_buf_ptr(buf_ptr: u32) -> Vec<u8> {
    let buf_ptr = buf_ptr as *const u8;
    let len =
        u32::from_le_bytes(std::slice::from_raw_parts(buf_ptr, 4).try_into().unwrap()) as usize;
    let mut buf = vec![0; len];
    buf.copy_from_slice(std::slice::from_raw_parts(buf_ptr.add(4), len));
    buf
}

It does require some low level memory access to do this, so it can be difficult for higher level languages that compile to Wasm through a nested runtime.

Executors will have to allocate buffers into the Wasm module’s memory to create these buffers which is why the module should also export a couple special functions that give the outside access into the native allocator your language uses. Like in Rust we give access to the std::alloc interface:

#[no_mangle]
extern "C" unsafe fn __k3_alloc(size: u32) -> *mut u8 {
	std::alloc::alloc(std::alloc::Layout::array::<u8>(size as usize).unwrap())
}

#[no_mangle]
extern "C" unsafe fn __k3_dealloc(ptr: *mut u8, size: u32) {
	std::alloc::dealloc(ptr, std::alloc::Layout::array::<u8>(size as usize).unwrap())
}

Different parts of the SDK have different serialization conventions/formats they use, but it’s mostly just Strings which are converted into UTF-8 buffers in transit. One special case is the http module in our SDK which is used to serialize and deserialize the HTTP request/response objects. The executor running on our operators currently is written in Rust and uses the httpcodec crate to serialize the request it receives into a buffer and it is expected that the response follow the same format. Which means if you are implementing this in another language you will have to do some reverse engineering to replicate this format. We are still open to change on this and would be willing to change to a more standardized codec if the community has better suggestions!

Environment variables

Environment variables are created at deployment time, encrypted and stored in our backend. These environment variables are called during runtime by the Executor and installed into the WASM instance before execution. The environment string is in the same format as local .env files, lines of KEY=VALUE pairs. The Env attribute tells the executor to install these Env vars to the WASM runtime:

use k3_wasm_macros::http_handler;
use k3_wasm_sdk::http::{Request, Response};
use std::env;

#[http_handler(Env)] // Env attribute
pub fn get(_req: Request<Vec<u8>>) -> Response<Vec<u8>> {
    let _env_var = env:var("ENV_VAR").expect("ENV_VAR variable must be set")

    Response::builder()
        .status(200)
        .header("Content-Type", "text/plain")
        .body("Hello K3!".as_bytes().to_vec())
        .unwrap()
}

k3_wasm_macros::init!();

Inputs

The current SDK can accept inputs using the define_k3_write_inputs!() macro. This macro adds the necessary functions to the WASM during compilation to accept inputs from the outside:

use k3_wasm_macros::{define_k3_write_inputs, http_handler};
use k3_wasm_sdk::{
    get_args,
    http::{Request, Response},
};

use serde::*;
use k3_json::parse_args;

define_k3_write_inputs!(); // tells WASM to add functions for inputs

#[derive(Debug, Serialize, Deserialize)]
struct Args {
    name: String,
    age: String,
    favorite_food: String
}

#[http_handler]
pub fn get(_req: Request<Vec<u8>>) -> Response<Vec<u8>> {
    // Get inputs from ARG_PTR and ARG_LEN (from define_k3_write_inputs! macro)
    let json_str = unsafe { get_args(ARG_PTR, ARG_LEN) };

    // Try to parse input according to the expected type.
    // Here, the input should match {name, age, favorite_food}
    let args: Args = match parse_args(&json_str) {
        Ok(parsed) => parsed,
        Err(response) => return response,
    };

   // do something with the inputs ......

    let name = &args.name;
    let age = &args.age;
    let favorite_food = &args.favorite_food;
}


// different type of input for another function/route
#[derive(Debug, Serialize, Deserialize)]
struct DifferentArgs {
    width: String,
    height: String
}

// parse input to a different type
#[http_handler]
pub fn post(_req: Request<Vec<u8>>) -> Response<Vec<u8>> {
    // Get inputs from ARG_PTR and ARG_LEN (from define_k3_write_inputs! macro)
    let json_str = unsafe { get_args(ARG_PTR, ARG_LEN) };

    let args: DifferentArgs = match parse_args(&json_str) {
        Ok(parsed) => parsed,
        Err(response) => return response,
    };

   // do something with the inputs ......

    let width = &args.width;
    let height = &args.height;
 
}

Current SDK reference

Here is the current SDK functions that executors provide (types are in Rust but are primitives so can be inferred for other languages):

// HTTP
fn __k3_http_get(url_ptr: u32) -> u32;
fn __k3_http_get_auth(url_ptr: u32, auth_ptr: u32) -> u32;
fn __k3_http_post(url_ptr: u32, body_ptr: u32) -> u32;

// IPFS
fn __k3_ipfs_upload(buf_ptr: u32, res_ptr: u32);
fn __k3_ipfs_read(cid_ptr: u32) -> u32;

// Key Value Store
fn __k3_kv_open_default() -> u64;
fn __k3_kv_get(db_ptr: u64, key_ptr: u32) -> u32;
fn __k3_kv_set(db_ptr: u64, key_ptr: u32, value_ptr: u32);
fn __k3_kv_delete(db_ptr: u64, key_ptr: u32);

// Smart Contract
fn __k3_data_sc_call(sc_address_ptr: u32, fn_data_ptr: u32) -> u32;
fn __k3_data_sc_query(sc_address_ptr: u32, fn_data_ptr: u32) -> u32;

// Space & Time
fn __k3_sxt_execute_raw(sql_ptr: u32) -> u32;
fn __k3_sxt_execute_create_table(table_name_ptr: u32, table_types_ptr: u32) -> u32;

All _ptr: u32 types are the previously explained buffer types. Any u64 types are usually executor-generated handles. The k3_wasm_sdk crate wraps these functions to provide a nicer API for all of them, you can consider that as a reference if you are doing something similar in a different language.

Some specific things (since the types might not be fully explanatory):

• sc_call returns the transaction hash as a string

• sc_query returns some fetched data serialized as a buffer

• execute_raw returns a JSON string specifying changes made to the DB

• execute_create_table exists because Space & Time requires tables to have their public/private key pair and biscuits which the executors can handle for you

Flow of operations

<DIAGRAM OF THIS WHOLE FLOW>