The walkthrough
By the end of this walkthrough, a trained robot policy will be running evaluation rollouts on your machine. Live camera feeds and telemetry in your browser, every episode recorded, judged, and filed with full provenance. Like this:
koyu is an open platform for robot learning, built on a bet: coding agents are now good enough to do the interface work, the wiring, glue, and format shuffling that made robot learning research painful. Every koyu interface, from local memory movement on the robot, to data storage APIs in the cloud, is designed to fit in an agent's context window. koyu consists of four main components:
koyu is currently in early Alpha. Data loading is about as simple as it can be, and the templates only cover ACT and Diffusion Policy so far. We expect this to improve significantly in the coming months!
You'll need
Step 1 of 3
Copy the prompt below into your coding agent on the machine that will run the sim. It installs the workspace and runtime, downloads the trained ACT policy and the LIBERO harness from koyu.dev, wires the robot loop over shared-memory IPC from the reference files in koyu-onboard-repos, and smoke-tests one episode. The part that used to take a grad student a week.
Set up a koyu robot-learning rig on this machine: local workspace + robot
runtime + a trained ACT policy evaluated in the LIBERO simulator, wired end
to end so evals can be started from the browser and every rollout lands in
the workspace with full provenance.
0. PRE-FLIGHT (before touching anything)
STOP. Do not run a single command yet. Present this warning to the user
verbatim in your own message, and do not move on until they explicitly
approve — no installs, no clones, not even a directory listing first:
- What will be installed: one Python venv (koyu-cli, koyu-runtime,
koyu-workspace, torch+torchvision+torchcodec, transformers, hf-libero
sim + MuJoCo), npm packages for the frontend, three git clones
(koyu-runtime, koyu-workspace, koyu-onboard-repos).
- Downloads: ~207 MB ACT checkpoint, ~409 MB LIBERO sim assets,
~5 GB Python wheels (torch CUDA), ~300 MB npm. Total disk ~8 GB.
- Timing: roughly 20-40 minutes on a GPU machine with a decent
connection; CPU-only works but evals run slower. Much of that is
long-running back-to-back commands (downloads, builds, training-free
but slow sim setup), so the agent may go quiet for stretches at a
time — that is normal, not a hang. If the user would rather not be
left in the dark, offer to narrate or explain each step as it runs
instead, and let them choose.
- The recommended rig directory: $HOME/koyu-rig (all state lives under
it; nothing else on the machine is modified). Ask whether to proceed
there or elsewhere. Do not start until the user confirms.
Ground rules once approved:
- export KOYU_RIG=<confirmed dir>; immediately export
KOYU_HOME=$KOYU_RIG/home and KOYU_RUNTIME=$KOYU_RIG/rt and keep all
three exported for every command. Never rely on ambient values: this
machine may host other koyu runtimes/workspaces, and koyu's
resolution ladder will happily find them.
- Read the READMEs of the two repos you clone, then the runtime's
AGENTS.md — the laws apply to every task; do not invent seams it
doesn't describe.
- If something fails, read the error fully; koyu errors name the next
move.
1. INSTALL (one venv for everything)
- uv venv $KOYU_RIG/.venv; export PATH=$KOYU_RIG/.venv/bin:$PATH for all
later commands. Never `uv sync` or create template-local venvs.
- uv pip install --python $KOYU_RIG/.venv/bin/python koyu-cli
(uv venvs have no pip binary; always `uv pip --python`)
- git clone https://github.com/koyu-lgtm/koyu-runtime and
https://github.com/koyu-lgtm/koyu-workspace into $KOYU_RIG; install
both editable into the venv.
- git clone https://github.com/koyu-lgtm/koyu-onboard-repos into
$KOYU_RIG/onboard — it carries the rig's reference files and the
dependency pins.
2. WORKSPACE (server first; it owns the store)
- KOYU_HOME set, LOCAL_TOOL_PORT=8000:
uvicorn local_tool.server.app:app --port 8000
- cd koyu-workspace/frontend && npm install && npx vite
- Confirm http://localhost:5173 shows an empty workspace.
3. TEMPLATES (public; no account, no token)
- GET https://koyu.dev/api/projects?scope=public&tags=template
- koyu ls each; koyu clone the ACT and LIBERO projects.
- Confirm the ~200 MB checkpoint landed; record the CLONED (local)
project/run ids from the clone's id_remaps output — provenance uses
these, never the cloud ids.
- Install sim + policy deps per $KOYU_RIG/onboard/README.md's pinned
line, translated to uv pip install --python $KOYU_RIG/.venv/bin/python.
4. WIRE THE RIG (apply, don't generate)
- Apply the files from $KOYU_RIG/onboard per its README:
types.py.append, data_recorder.sources.py, ControlsPage.tsx,
index.css.append.
- Substitute services.yaml.template's {{PLACEHOLDERS}} (the cloned
local ids, the venv python, the dirs) into $KOYU_RUNTIME/services.yaml.
- Arm ingest: PUT http://localhost:8000/api/ingest/config
{"outbox": "$KOYU_RUNTIME/data-recordings"}.
5. SMOKE IT END TO END
- koyu up -r $KOYU_RUNTIME (the boot typecheck runs; fix what it names)
- Start an eval from the Controls page (or ring eval/control); watch one
episode complete. koyu frame eval/obs/agentview to see the sim;
koyu tail recorder/episode for capture events.
- Ingest on the Datasets page; open the dataset and verify reward set,
provenance links to the cloned ACT run, and the run's page shows the
eval dataset.
Report when done: the URLs, what you applied vs. modified, the smoke
result (episodes captured, success/fail), and anything that fought you.
While it works: press Next. The next page explains what's landing on your machine, and everything will be ready when you get back.
Step 2 of 3
koyu stores robot learning experiments as projects, with runs arranged as a tree inside each one. Projects and runs are almost entirely file-backed, which lets agents operate on code and weights with ordinary tools and sync everything to koyu.dev. They may look vaguely like git repos, but they are backed by a file object store and are additive-only when syncing to cloud. That makes them a poor option for tracking code changes and a great one for robot learning training runs, which tend to collect assorted file types and large weight files.
The LIBERO template in your local workspace. Note that both projects and runs are backed by files.
Notice how each run has a manifests section.
One manifest: episodes, features, and the runs it feeds.
Manifest is koyu's word for a robot learning dataset, and a manifest can
hold data of several kinds: teleop, sim, eval, synthetically generated.
Episodes can be reused across manifests, so you can mix and match data from
different sources. Manifest links to a run are virtual, so the same data can
serve many runs or projects. The projects on your computer were downloaded
without --include-manifests, which is why the LIBERO runs hold no
LIBERO training data right now. In the next step you will run evals of an
Action Chunking Transformers policy in a simulated environment, and the
resulting eval manifest will link to the ACT run, so you will watch a manifest
link form in front of you.
koyu's runtime uses supervisord and iceoryx2 to keep the distance between your agent and the machine's process management and data streams as small as possible. You can check for yourself by asking your agent to read the robot telemetry through the koyu CLI. The agent also maintains the metadata for eval provenance, which is why the page shows the correct project and run id: the agent set it during the last step. In the next section you will drive this page yourself.
A glimpse of koyu's process management and IPC, materialized as the Controls page, with telemetry streaming from the simulated robot.
Coding agents have made custom dashboards, upload scripts, and similar artifacts cheap. koyu's architecture is designed to ride this wave, and its value is meant to be the overall architecture, versus the specific code shipped in this demo. Its runtime, workspace, and cloud APIs are all designed to be customized to fit the last-mile of your setup. Maybe you keep koyu-runtime to replace your existing robot runtime and stay happy with your current data pipelines. Maybe you are happy with your ROS setup and only want the minimalistic koyu cloud APIs for experiment tracking. Or some other third thing. All are cool.
Step 3 of 3
Head over to the Controls page, click the Start button, and your simulated robot will start moving! The underlying AI policy is a trained template from koyu and uses a popular robot learning technique called ACT.
After the episode is finished, head to the Datasets page and click the Ingest button. This mechanism collects the eval episodes from the runtime and copies them into the workspace, after which you can visualize them. Note that the eval dataset has a link to the run that produced it. That link was enabled by the runtime and agent provenance mechanisms highlighted on the previous page.
Where this goes
Choose your next move.
You are pulling LIBERO benchmark datasets from koyu.dev. They are public;
no account or token is needed.
1. pip install koyu-cli
2. Ground yourself before transferring anything:
curl -fsSL https://koyu.dev/api/projects/proj_625dee3303ea0e6251816736aa06a007/files/README.md
koyu ls proj_625dee3303ea0e6251816736aa06a007 shows the four suite runs,
koyu ls <run-id> shows the run's linked manifest (the dataset), and
koyu ls <manifest-id> shows its episodes and features.
3. koyu pull <manifest-id> data/<suite-name> for each suite I want.
One suite is about 110 MB: 500 demonstrations, two cameras, 128 px.
MY GOAL: pull all four suites into ./data
(edit if needed: "just libero-spatial" / "into <path>")