Veris
On-device assistive vision for blind and low-vision users
The software side of a privacy-first wearable that describes the world out loud from a fine-tuned vision-language model running entirely on a Jetson Orin Nano. No image ever leaves the device.
Recorded runs, always up, no hardware needed
- 13.5s → 3.6s
- time to first spoken word
- ½
- invented details on unanswerable images
- 15 W
- full pipeline power budget on the Orin
- 653
- curated training examples for Veris-3B-v1
Why it exists
Cloud-based visual assistants send every camera frame off the device, which is a hard sell for something worn on your face all day. Veris keeps video local: a head-mounted camera feeds a Jetson Orin Nano in a small pack, and an open-ear headset speaks scene descriptions. The only thing that leaves the device is phone GPS, which a cloud context engine turns into a hint about where you are and what to look for. Backed by AIM, UMD's AI institute, with ongoing funding.
What I built
- Drove perceived latency from 13.5 s to 3.6 s cold (and 0.15 s on repeated scenes) with an image-token budget sweep, prompt KV caching, and clause-by-clause streaming text-to-speech. Rejected approaches are documented alongside the ones that worked.
- Fine-tuned Veris-3B-v1 with QLoRA on 653 curated examples: judged distillation from a 7B teacher, 344 hallucinations rescued into honesty targets, and mirror-augmented spatial pairs. Trained overnight on a single RTX 3070, quantized to GGUF Q4_K_M, and served on the Orin at 15 W. Invented details halved with zero over-refusal and no speed penalty.
- Gated training on behavioral evals for laterality, shuffled-image shortcut learning, and context bleed. The v1 adapter took laterality from 86% to 100% and context-bleed failures from 9 to 0.
- Designed a dual-LLM architecture: phone GPS goes to a Dockerized, spend-guarded FastAPI context engine (Google Maps plus a small LLM, with an OpenStreetMap and rules fallback) that returns a situation, suggested mode, and things to look for. The on-device model is primed with that context, with a regression test guaranteeing priming never invents unseen objects.
- Built a data flywheel: every interaction and thumbs-up or thumbs-down rating lands in session logs, syncs off the board, and is mined back into training rows for the next fine-tune.
Also in the box
- Four description modes (outdoor nav, indoor nav, text reading, describe) with a state-machine arbiter that handles hysteresis, reflex interrupts, and stale frames.
- On-device voice questions through whisper.cpp.
- Boot-resilient systemd stack with a watchdog and an always-on front door that keeps the public link working while the board is off.
- Full screen-reader pass on the demo: ARIA live-region streaming, focus management, WCAG AA contrast, and a keyboard-navigable walkthrough.
- Description style and every training-data rule trace to a cited evidence base on blind and low-vision user preferences.
In pictures
Where Veris is going
Everything above runs on a devkit on a desk. The next year is about turning it into something a person can actually wear out the door, and making the model fast and honest enough to trust there.
First wearable prototype
Moving off the devkit into a body-worn pack with an integrated camera, open-ear audio, and a battery that lasts a day. The software already runs headless with a watchdog; the work now is enclosure, power, and thermals.
Sensor integration for faster alerts
Microphones, a gyroscope and IMU, and distance sensors feeding the same arbiter that already switches modes. A vision model is slow by nature; a distance sensor can warn about a step down in milliseconds. The goal is one cohesive device where each sensor covers for the others' weaknesses.
Faster time to first word
The cold-start path is at 3.6 s. The next targets are speculative decoding, a smaller visual encoder, and smarter frame selection so the model only runs when the scene has changed.
More accurate, more honest models
Veris-3B-v2 will train on the sessions the data flywheel has been collecting since v1 shipped, with tighter eval gates for laterality, text reading, and abstention on unanswerable images.
Testing with blind and low-vision users
Structured sessions with the community the device is for, so description style and mode design are shaped by real use instead of our guesses.
Funding the next phase
AIM, UMD's AI institute, funds the current work. We are looking for additional backing to cover hardware for a small prototype fleet, compute for training, and user-testing sessions.