StudioOpen sourceHITL AI
akaOSS
The open-source studio for human-in-the-loop AI. Five projects, one thesis, a reproducible research feed, and the HITL Kit component registry — served as one site.
Five projects, one thesis, a live research feed. MIT.
In simple terms
akaOSS is an open-source studio: five free tools, one argument, and one website that ties them together.
The argument is that the industry measures AI wrong. Benchmarks ask whether a model can finish a task on its own. A person at work needs to know whether it made them better at their job, and those are not the same question. I call the alternative Assist-Not-Complete.
The five tools are what you need in order to act on that: ready-made interface pieces for keeping a person in charge of an agent, a way to score whether the agent actually helped, structured tagging for review work, and two tools for developers working with AI. All of them are free, and any developer installs them with one command.
The site also publishes a research feed. Every finding on it is a real experiment: the question, the runs against real models, human-scored results, the raw run data checked in, and a link to reproduce it yourself.
The impact. Most positions on AI evaluation stop at the essay. This one ships the code that makes the position buildable and the evidence that lets you check it, so a reader who disagrees can re-run the experiment instead of arguing about the claim.
What makes it unusual
An argument with an implementation attached
Then a perspective piece on how AI should be evaluated ends when the essay does.
akaOSS every primitive in the kit traces back to a specific claim in the paper, and installs into a real project with one command. The library is the argument, made usable.
Findings you can re-run
Then AI research posts report numbers you have to take on faith.
akaOSS publishes the question, the runs, the human scores, the raw run JSON in the repo, and a reproduction link for each finding.
People do the scoring
Then it is now normal to have one model grade another, which is fast and quietly circular.
akaOSS puts a human on a fixed rubric. An AI pre-fill exists, is optional, and is flagged on every score it touched.
A real registry, not a repo to copy from
Then most component libraries are source you clone and adapt by hand.
akaOSS serves nineteen install endpoints with continuous integration that fails the build on drift, so anyone on the open internet gets a working install with its dependencies resolved.
One source of truth
Then the paper, the demos and the code usually live in three places that quietly disagree.
akaOSS keeps the registry, the feed and the paper on one site with file-based content, no CMS and no database behind it.
The through-line is that each piece is checkable. The claim has a paper, the paper has primitives, the primitives have a registry, and the findings have raw data and a repro link.
The thesis
Current benchmarks ask “can the model complete this task autonomously?” In deployment, real users want an assistant that respects their authority, preserves their agency, and makes them better over time. Assist-Not-Complete: evaluate AI on whether it assists humans without displacing them, not on whether it can finish the task alone. The argument is made in full in the paper, An AI Measurement Problem, and tested in public in the research feed — every finding is a reproducible experiment run with the kits below.
The projects
Five projects across two tracks. Each lives in its own repo; four of them have a write-up here.
Human-in-the-loop measurement
Developer tooling
One site, one source of truth
akaoss.dev ties them together. The projects each live in their own repos; this one holds the HITL Kit registry (the source of truth for the shadcn primitives, served at /r/*.json, with CI failing on drift), the research feed (question → runs against real models → human-scored results → checked-in run JSON → repro link), and the paper. Next.js 16, Tailwind v4, file-based content — no CMS, no database.
Where it comes from
akaOSS is where the human-side-of-applied-AI work becomes reusable: the kits are the measurement instruments, the research feed is the evidence, and the paper is the argument. It's built at Circleheads.