All chapters
01Complete

H.A.L.O

A ground controller for the station.

2026Design, machine learning, systemsSIH26174On-board procedure intelligence

An on-board AI that watches a science procedure in orbit and speaks up the moment a step is missed or out of order. It runs fully offline on one laptop GPU.

276
tests passing
8/8
verification checks
0.79
mAP50 on real ISS footage
0
false alerts in the run
H.A.L.O dashboard showing a live procedure view with detected objects and a step timeline.
The ground dashboard: live procedure state, detector evidence and the event log.
01

The problem

A science procedure on the station is a sequence of small physical actions: open a tray, pipette a sample, seal it, place it back in the freezer. Miss one, or do them out of order, and the experiment can be ruined. The crew is highly trained, but they are also tired, interrupted and working in a loud environment.

The obvious approach is to train a video model to recognise the whole procedure. That gives you a black box with a confidence score. It cannot tell a controller which step was skipped, why the alert fired, or what evidence it saw. In orbit, an unexplainable alert is almost as bad as no alert at all.

02

The decision

I treated the procedure as a state machine. Each step has a visual precondition and a visual completion signal. A detector finds the objects; a vision-language model answers narrow questions about what is visible; a deterministic layer decides whether the current step is complete, pending or out of order.

That separation matters. The models are allowed to be uncertain. The state machine owns the logic, so the system can explain itself: this is the step I expected, this is the evidence I saw, this is why I spoke.

The models can be uncertain. The procedure cannot. Separating the two is what made the system trustworthy.
03

The build

The perception layer is a YOLO11 detector trained on real ISS footage, paired with Qwen3-VL for plain-English questions about a frame. A sliding window keeps the last few seconds of evidence so a single ambiguous frame does not trigger an alert.

Everything runs locally on one laptop GPU. The on-board side writes a hash-chained log and sends a signed report of roughly one kilobyte over the downlink. A React dashboard lets a ground controller replay the procedure, inspect the evidence and see exactly where the state machine diverged.

  • YOLO11 detector trained on real ISS footage, mAP50 0.79
  • Qwen3-VL answers nine visual questions in about 0.6 seconds
  • Sliding-window evidence with a deterministic procedure state machine
  • Hash-chained log and a signed ~1 KB downlink report
04

The evidence

The system passed 8 of 8 verification checks and 276 automated tests. In the recorded procedure it tracked 8 of 8 steps in the correct order with zero false alerts. The dashboard can replay the run frame by frame, with the detector boxes and the state transition that followed.

The important result is not the mAP score. It is that the system stayed quiet when it should have, and spoke with a reason when it should have. That is the behaviour a crew would actually have to trust.

05

What I learned

I learned to stop asking a model to be the system. A model is a sensor. The system is the contract around it: what counts as evidence, what counts as done, and what happens when the evidence is ambiguous.

That pattern now runs through everything I build. Keep the intelligence narrow, keep the logic inspectable, and make the machine say why.

Built with
PythonYOLO11Qwen3-VLPyTorchReact
Next chapter · 02
Susurrus