hal0

Running local AI should be accessible, not a folder of scripts piling up

Running AI on your own machine is genuinely appealing: no per-request bill, and your data never leaves the building. But the path there is discouraging enough that most people give up and go back to a vendor’s API. You end up with a pile of half-finished pieces — one script for chat, a separate setup for search, another for image generation, none of them aware the others exist. Nothing shares a way to check whether it’s working. Nothing shares a common interface. Every capability is its own island, and adding one more means learning that island’s quirks from scratch.

It doesn’t have to work like that. hal0 makes local AI something you install once and then simply use: chat, search, embeddings, ranking, transcription, speech and image generation all answer at one address on the box, speaking the same contract every existing AI tool already expects. Point an app at it and it works, because as far as that app is concerned nothing has changed except where the answer came from.

It runs best on a machine like a Ryzen AI Max+ 395 with 128 GB of shared memory — enough to hold several models at once — but it runs on anything with enough RAM. No cost per request, and no company sitting between you and your own data.

Each feature runs on its own, so one failure doesn’t take the rest down

hal0 is explicitly not another llama-server wrapper. Every inference workload — the chat model, the embedder, the reranker, the image generator — runs as its own podman container, managed by its own hal0-slot@<name>.service systemd unit. There is no shared inference daemon juggling every model in one process, and no extra process to babysit outside systemd itself.

Slot units are cheap to define and easy to reason about in isolation, which is what makes a single control plane tractable: hal0-api doesn’t run the model, it just knows which container is ready and where to send the request.

Slot units are never hand-written. hal0-api renders a podman Quadlet .container file per slot and lets systemd generate the hal0-slot@<name>.service unit from it, so the dependency graph and restart semantics come from systemd rather than from bespoke supervision code.

WantedBy=hal0.target is the line that matters: hal0.target is the boot-enable anchor every slot depends on. If it is missing or not enabled, slots that read healthy before a reboot quietly stay down after one — which is why hal0 doctor grew a dedicated check for it.

When your tests and your users are running different software

The most useful bug in the project’s history was invisible to every test that existed. A routine dependency update — a minor version of the web framework underneath the API — quietly changed how it reported its own list of routes. hal0 builds the control surface its AI agents use by reading that list at startup. The list came back empty. The startup code caught the resulting error, logged it, and carried on reporting perfectly healthy, so the entire agent control surface simply never came up. Found live only after twenty-one boots with zero successful mounts.

The reason no test caught it is the part worth keeping. The test environment installed the framework from a locked file, pinned to an older version that still behaved the old way. Real installs resolved it from the project’s own dependency list, which set a floor and no ceiling — so every fresh install pulled the newer, changed version. The tests and the users were running different software, and had been for a while.

Three fixes, and only one of them is the code change. Capping the dependency closed the gap between test and production. Rewriting the route reader to handle both shapes made it resilient to the next change. But the one that mattered most was making a failed startup loud — the bug was survivable for twenty-one boots precisely because something caught the error and let the service claim it was fine. A health check that lies is worse than one that fails.

What the platform actually does

From a working prototype to something you can trust

The first working version shipped in ten days, with fifteen releases building up the core system and its feature model. A rework in early August took the parts that worked on one machine and rebuilt the parts that wouldn’t survive someone else’s hardware — proven by the fact that the “finished” 1.0.0 release itself needed a fix the very next day. The eight days that followed were dedicated to testing against real hardware — fresh installs, in-place upgrades, and machines with different graphics setups — with sixteen of the eighteen problems that surfaced fixed before the next release went out.

Speed is the part people assume you give up by running your own models, so it gets measured rather than claimed. Twenty-six models were benchmarked end to end on the reference machine — an AMD Ryzen AI Max+ 395 with 128 GB of shared memory — recording response speed, memory use, power draw and time to first word. A 35-billion-parameter model answers at 102 tokens per second, comfortably faster than anyone reads. A small model hits 169.8. Chat and search running concurrently sustain 258 tokens per second between them, and speech generation runs at roughly twice real time — a sentence is spoken faster than you could say it. Benchmarking is a built-in subsystem, not a spreadsheet someone maintained once: results are recorded, deduplicated and journaled so a slowdown shows up as a regression instead of a rumour.

Release process kept pace with the work: every release now ships its own changelog, and the update process checks a signed package and warns about breaking changes and migration steps before it touches anything — with a one-command rollback if something still goes wrong.