skip to content
techgopal
// FEATURE

Three projects, one person: why I build my own stack

Summary. I am building three pieces of software that sit on top of each other: Krishiv, a Spark-compatible query engine in Rust; Vyasa, a block-based CMS; and Setu, a control panel that runs single-binary web apps on my own Linux servers. I did not set out to build a stack. Each one started because the tool I wanted either did not exist or came with a cost I did not want to carry. This post is the map: what each project is, what they share, and what building all three alone has cost me.

Key facts

ItemValue
KrishivBatch SQL and DataFrame engine over Arrow, written in Rust, Spark-compatible. The 0.2 release is an embedded alpha: it runs inside your Python or Rust process.
VyasaContent management system in Rust. One server binary plus PostgreSQL. Content stored as structured blocks. Version in this post: 0.4.1.
SetuControl panel that deploys compiled web apps as systemd services on your own servers. Version in this post: 0.1.6.
Shared languageRust, with unsafe confined to a few named crates in Krishiv
Shared shapeOne binary per product, PostgreSQL where state is needed, typed errors, a layer rule between crates
LicencesKrishiv: Apache-2.0. Vyasa: MIT or Apache-2.0. Setu: AGPL-3.0-only
Diagram of three projects: Krishiv query engine, Vyasa CMS and Setu control panel, with Setu deploying Vyasa.
Krishiv (data), Vyasa (content) and Setu (servers). The one arrow: Setu deploys Vyasa.

Why would one person build a query engine, a CMS and a control panel?

Because I kept hitting the same wall. I wanted to run analytics on one machine without a cluster, publish writing without a plugin ecosystem I had to audit, and put both on a server I own without learning another orchestrator. For each of those, the available answer was either too big for the problem or too loose about trust.

None of the three is a reaction to a single product. They are three answers to one question: what is the smallest honest system that does this job?

What is each project, in plain words?

Krishiv reads Parquet, CSV and JSON, runs Spark SQL and PySpark-shaped DataFrame code, and hands results back as Arrow. The 0.2 release runs in your process: krishiv.session() gives you a SparkSession-shaped object, and results move to pyarrow, Polars or DuckDB without copying buffers. It has its own parser, analyzer, optimizer and vectorised executor. There is no JVM and no DataFusion underneath. Spark Connect, streaming, incremental views and distributed execution are not in 0.2, and I will say so every time it matters.

Vyasa stores a post as a JSON document of blocks, not as HTML with comments in it. Themes are data: design tokens compiled to CSS plus sandboxed templates, so a theme cannot run code on the server. Plugins are WebAssembly components with declared capabilities, checked on every call. The server is one binary and PostgreSQL.

Setu deploys a published binary. You give it a manifest (a TOML file: where the tarball is, its SHA-256, which environment variables to set, what health path to check). It creates a user, a database, a systemd unit with cgroup limits, a Caddy route with HTTPS, and a nightly backup. Updates back up first and roll back if the new version is not healthy.

What do the three have in common?

  1. One binary, few moving parts. Vyasa is a binary and Postgres. Setu is one binary acting as control plane, agent and privileged helper, chosen by subcommand. Krishiv's first form is a wheel you install with pip and import.
  2. A strict layer rule. In Krishiv, a crate may depend only on crates in the same or a lower layer, and a tool checks it. In Vyasa the direction is common <- db <- core <- {themes, plugins, search, jobs} <- ai <- api, and only the last crate may import the HTTP framework. The rule is boring and it is the reason I can still change things.
  3. Typed failure. Library code returns typed errors and is written not to panic. Krishiv carries Spark's error classes so a failure reads the way a Spark user expects.
  4. Untrusted by default. Vyasa's plugins and themes, and Setu's sites are treated as code I do not fully trust. Krishiv does not sandbox UDFs yet: a Python UDF runs inside your process, so it is not the place for untrusted code.

What does it cost to build all three alone?

Time, mostly, and the cost shows up in a specific place: the tail. The first 80% of a parser, a block editor or a systemd provisioner is fast. The tail is compatibility with Spark's exact behaviour, migrations that can only go forward, an installer that works on five Ubuntu and Debian releases. I handle it by refusing to widen scope: each project writes down what it does not do yet, and I keep to it.

The other cost is context switching. I handle that with written notes in each repository so that any session, mine or an assistant's, can pick up where the last stopped.

Where do they go next?

Krishiv 0.2 is out as an embedded alpha, and its benchmarks are published with the method at krishiv.ai/benchmarks; I do not repeat numbers here. Its next step is 0.2.1, a release of fixes. Vyasa is pre-1.0 and running real sites. Setu's roadmap has containers as a second runtime, off-box backups and drain for servers, none of which exist in 0.1.6.

The next four posts in this series go one level down: the layers of the query engine, why I built a CMS, what a control panel for my own servers looks like, and when one machine is enough.

Next: Query engine 1: anatomy of a Spark-compatible engine in Rust.