DARPA PTG · University of Rochester · 2022-2024
Real-time multimodal assistant on wearable devices (DARPA PTG)
A vision-language-audio assistant that sees what the wearer sees, listens, answers in speech and guides physical tasks step by step; demonstrated live at the DARPA PTG program review at MIT (Oct 2023) and evaluated by MIT Lincoln Laboratory.
- Evaluated by MIT Lincoln Laboratory
- Decision I made a video-to-text prototype that moved the project to language-first models
- Artifacts EAGLE-400K (ACM MM 2024), OSCaR (NAACL 2024 Findings), MISAR (ICCV 2023 AV4D)
- Scale live on HoloLens and a helmet-mounted camera-and-microphone rig
I led the live demonstration at the DARPA PTG program review at MIT (October 2023), on HoloLens and a helmet-mounted Mevo camera-and-microphone rig, and helped pivot the project toward vision-language and reasoning models. The system was formally evaluated by MIT Lincoln Laboratory.
Related work: EAGLE (ACM MM 2024), OSCaR (Findings of NAACL 2024), MISAR (ICCV 2023 AV4D Workshop), Possible Worlds VQA (IEEE TMM 2024).