Skip to content

go-io

forge.lthn.ai/core/go-io is a storage abstraction library that provides a single Medium interface for reading and writing files across different backends. Write your code against Medium once, then swap between local disk, S3, SQLite, or in-memory storage without changing a line of business logic.

The library also includes sigil, a composable data-transformation pipeline for encoding, compression, hashing, and authenticated encryption.

Quick Start

import (
    io "forge.lthn.ai/core/go-io"
    "forge.lthn.ai/core/go-io/s3"
    "forge.lthn.ai/core/go-io/node"
)

// Use the pre-initialised local filesystem (unsandboxed, rooted at "/").
content, _ := io.Local.Read("/etc/hostname")

// Create a sandboxed medium restricted to a single directory.
sandbox, _ := io.NewSandboxed("/var/data/myapp")
_ = sandbox.Write("config.yaml", "key: value")

// In-memory filesystem with tar serialisation.
mem := node.New()
mem.AddData("hello.txt", []byte("world"))
tarball, _ := mem.ToTar()

// S3 backend (requires an *s3.Client from the AWS SDK).
bucket, _ := s3.New("my-bucket", s3.WithClient(awsClient), s3.WithPrefix("uploads/"))
_ = bucket.Write("photo.jpg", rawData)

Package Layout

Package Import Path Purpose
io (root) forge.lthn.ai/core/go-io Medium interface, helper functions, MockMedium for tests
local forge.lthn.ai/core/go-io/local Local filesystem backend with path sandboxing and symlink-escape protection
s3 forge.lthn.ai/core/go-io/s3 Amazon S3 / S3-compatible backend (Garage, MinIO, etc.)
sqlite forge.lthn.ai/core/go-io/sqlite SQLite-backed virtual filesystem (pure Go driver, no CGO)
node forge.lthn.ai/core/go-io/node In-memory filesystem implementing both Medium and fs.FS, with tar round-tripping
datanode forge.lthn.ai/core/go-io/datanode Thread-safe in-memory Medium backed by Borg's DataNode, with snapshot/restore
store forge.lthn.ai/core/go-io/store Group-namespaced key-value store (SQLite), with a Medium adapter and Go template rendering
sigil forge.lthn.ai/core/go-io/sigil Composable data transformations: encoding, compression, hashing, XChaCha20-Poly1305 encryption
workspace forge.lthn.ai/core/go-io/workspace Encrypted workspace service integrated with the Core DI container

The Medium Interface

Every storage backend implements the same 18-method interface:

type Medium interface {
    // Content operations
    Read(path string) (string, error)
    Write(path, content string) error
    FileGet(path string) (string, error)   // alias for Read
    FileSet(path, content string) error    // alias for Write

    // Streaming (for large files)
    ReadStream(path string) (io.ReadCloser, error)
    WriteStream(path string) (io.WriteCloser, error)
    Open(path string) (fs.File, error)
    Create(path string) (io.WriteCloser, error)
    Append(path string) (io.WriteCloser, error)

    // Directory operations
    EnsureDir(path string) error
    List(path string) ([]fs.DirEntry, error)

    // Metadata
    Stat(path string) (fs.FileInfo, error)
    Exists(path string) bool
    IsFile(path string) bool
    IsDir(path string) bool

    // Mutation
    Delete(path string) error
    DeleteAll(path string) error
    Rename(oldPath, newPath string) error
}

All backends implement this interface fully. Backends where a method has no natural equivalent (e.g., EnsureDir on S3) provide a safe no-op.

Cross-Medium Operations

The root package provides helper functions that accept any Medium:

// Copy a file between any two backends.
err := io.Copy(localMedium, "source.txt", s3Medium, "dest.txt")

// Read/Write wrappers that take an explicit medium.
content, err := io.Read(medium, "path")
err := io.Write(medium, "path", "content")

Dependencies

Dependency Role
forge.lthn.ai/core/go-log Structured error helper (E())
forge.lthn.ai/Snider/Borg DataNode in-memory FS (used by datanode package)
github.com/aws/aws-sdk-go-v2 S3 client (used by s3 package)
golang.org/x/crypto BLAKE2, SHA-3, RIPEMD-160, XChaCha20-Poly1305 (used by sigil)
modernc.org/sqlite Pure Go SQLite driver (used by sqlite and store)
github.com/stretchr/testify Test assertions

Go version: 1.26.0

Licence: EUPL-1.2