Backend

Sia Storage Backend

The Sia backend implements manifest page construction, slab-based upload, and claim encoding for the Sia distributed storage network.

Overview

Sia is the first implemented storage backend for Urma. It uses Sia's slab-based storage primitives to upload manifest pages and track their locations. The backend provides a PageBuilder that handles the full manifest chain construction including reverse-order page linking.

infoNote

The Sia backend is a separate module: go.lumeweb.com/urma/sia. It is not part of the core protocol: it implements the manifest types and upload logic specific to Sia's storage model.

Location Data

The Sia location identifier is the string "sia". The root location data stored in an UrmaClaim's locationData field is aslabs.SlabSlice pointing to the first ManifestPage object on Sia.

Manifest Types

Sia manifest pages follow the core ManifestPage structure. The datapayload on page 0 is a Manifest (stream metadata + blobs). Continuation pages use ManifestBlobs (blobs only).

ManifestPage (Sia)

Core type: data and next are Sia-specific payloads.

Field	Type	Description
data	json.RawMessage	Sia-specific manifest payload (Manifest or ManifestBlobs).
next	json.RawMessage	Sia SlabSlice pointing to the next page. Omitted on last page.

Manifest

Page 0 payload: stream metadata + first page of blob entries.

Field	Type	Description
streamName	string	Original stream name.
streamType	string	LBRY stream type (typically "lbryfile").
suggestedFileName	string	Suggested file name for the stream.
blobs	[]ManifestBlob	Blob entries for this page.

ManifestBlobs

Continuation page payload (page 1+): blob entries only.

Field	Type	Description
blobs	[]ManifestBlob	Blob entries for this continuation page.

ManifestBlob

Per-blob entry carrying Sia retrieval and LBRY compatibility data.

Field	Type	Description
blobHash	string	SHA-384 hash of encrypted LBRY blob (hex-encoded).
iv	string	AES IV for blob content (hex-encoded).
blobLength	int	LBRY blob size in bytes (max 2 MiB).
blobNum	int	Position in stream (0-indexed).
slabKey	slabs.EncryptionKey	Sia slab encryption key.
minShards	uint	Minimum sectors for recovery.
sectors	[]slabs.PinnedSector	Pinned sectors for this blob's slab.
slabOffset	uint32	Byte offset into slab data.
slabLength	uint32	Number of data bytes in slab slice.

Building a Manifest Chain

Pages are built in reverse order: last page first, first page last. This allows each page's next pointer to reference the storage location of the previously uploaded page.

manifest.go

import (
    "go.lumeweb.com/urma"
    "go.lumeweb.com/urma/sia"
    "go.sia.tech/indexd/slabs"
    "go.sia.tech/siastorage"
)

// 1. Create a page builder for this stream
builder := sia.NewPageBuilder(
    streamName, streamType, suggestedFileName,
    sourceClaimID, dataKey,  // urma.SourceClaimID, [32]byte
)

// 2. Add blob entries: each carries LBRY blob hash/IV + Sia slab location
//    FromSlabSlice flattens a slabs.SlabSlice into ManifestBlob fields:
blob := sia.FromSlabSlice(slabSlice, blobHash, iv, blobLength, blobNum)
builder.Add(blob)

// 3. Build the manifest chain (reverse order, handled internally)
//    The upload callback receives each page's JSON and returns the SlabSlice
//    that identifies where it was stored. BuildChain wires all Next pointers.
result, err := builder.BuildChain(maxBlobsPerPage, uploadFunc)
if err != nil {
    log.Fatal(err)
}

// result.Claim      → urma.UrmaClaim (LocationData = root slab, JSON-encoded)
// result.RootSlab   → slabs.SlabSlice pointing to the first manifest page
// result.PageCount  → total pages in the chain

Upload Callback

The BuildChain method accepts an UploadFunc callback. The builder calls this function with each serialized manifest page, expecting a slab slice in return. This decouples the builder from the specific Sia renter implementation.

upload.go

// UploadFunc is called once per page, in reverse order (last page first).
// The builder handles Next pointer wiring: just upload and return the slab.
type UploadFunc func(pageJSON []byte) (slabs.SlabSlice, error)

// Example using the siastorage SDK:
// Upload each serialized ManifestPage as a Sia object, then return
// the SlabSlice that identifies where it was stored.
func uploadFunc(pageJSON []byte) (slabs.SlabSlice, error) {
    obj := siastorage.NewEmptyObject()
    err := sdk.Upload(ctx, &obj, bytes.NewReader(pageJSON))
    if err != nil {
        return slabs.SlabSlice{}, err
    }
    err = sdk.PinObject(ctx, obj)
    if err != nil {
        return slabs.SlabSlice{}, err
    }
    // Return the first (and only) slab slice: the manifest page
    // was uploaded as a single object.
    slabs := obj.Slabs()
    return slabs[0], nil
}

Decoding & Retrieval

On the consumer side, decode the root slab from the claim, fetch manifest pages from Sia, and walk the chain. Page 0 decodes as a Manifest; continuation pages decode asManifestBlobs. Each ManifestBlob converts back to aslabs.SlabSlice for retrieval.

decode.go

import (
    "encoding/json"
    "go.lumeweb.com/urma"
    "go.lumeweb.com/urma/sia"
)

// 1. Decode the root slab from the claim's locationData
rootSlab, err := sia.DecodeRootSlab(claim)
if err != nil {
    log.Fatal(err)
}

// 2. Fetch the ManifestPage JSON from Sia using the slab slice
//    (via siastorage.SDK or any Sia client that resolves a SlabSlice)
//    pageJSON := fetchFromSia(rootSlab)
//    var page urma.ManifestPage
//    json.Unmarshal(pageJSON, &page)

// 3. Decode page 0 as a Manifest (stream metadata + blobs)
manifest, err := sia.DecodeManifestPage(page)
if err != nil {
    log.Fatal(err)
}

// 4. Walk the chain: decode Next, fetch next page, decode as ManifestBlobs
nextSlab, err := sia.DecodeNextSlab(page)
for nextSlab != nil {
    // fetch nextPageJSON at nextSlab from Sia...
    var nextPage urma.ManifestPage
    json.Unmarshal(nextPageJSON, &nextPage)
    blobs, err := sia.DecodeManifestBlobsPage(nextPage)
    // accumulate blobs...
    nextSlab, err = sia.DecodeNextSlab(nextPage)
}

// 5. Each ManifestBlob converts back to a slabs.SlabSlice for retrieval
slice := blob.ToSlabSlice()