Mission Brief

• Founded: 2024
• Headquarters: Sunnyvale, California
• Mission: Enable the largest robot army in the world through intelligent physical AI for the US military.
• Core Technology: Fury, the foundation model for defense robotics
• Operational Focus: Developing and training an embodied AI brain that allows machines to perceive the world, understand natural language, and coordinate action autonomously across any domain.

The Challenge: Reviewing and Indexing Copious Amounts of Test Data

Scout AI, a fast-moving autonomy startup, operates at the cutting edge of autonomous vehicle development. Their core strength lies in continuous real-world testing and rapid iteration of software development cycles. Traditional data annotation methods often become a bottleneck, slowing down the development cycle and hindering the ability to quickly retrain autonomy software. An efficient and scalable solution to process and annotate the vast amounts of data generated during these tests directly results in faster improvements to the autonomous system.

Within Scout AI, diverse roles and responsibilities necessitate access to the same test data for distinct use cases:

• Engineers: Review drive logs and quickly triage critical issues.
• Annotators: Critique the autonomous system’s behavior based on Reinforcement Learning from Human Feedback (RLHF) workflows to be used in training.
• Operators: Update what they were thinking after the fact so the autonomous system can learn to think like them.

By rapidly moving through cycles of live testing, event annotation, and model retraining, Scout AI developed Fury, a defense-specific Vision-Language-Action (VLA) foundation model engineered to transform every defense robot into an intelligent, autonomous agent.

Scout AI’s Autonomy Edge: Vision-Language-Action Models

Unlike traditional robotics software, Fury is an embodied AI system capable of perceiving the physical world, interpreting natural language, and issuing real-time motor commands to act decisively even in comms-denied and GPS-denied environments.

Fury’s impressive capabilities and performance rely on an innovative new approach to autonomy development. VLA models are a class of multimodal foundation models that integrate computer vision and natural language to generate autonomous actions. Given an input video of the robot's surroundings and a text instruction, the VLA directly outputs instructions for the robot to execute autonomously.

VLAs are constructed by fine-tuning a vision-language model (VLM, a large language model with vision capabilities) on a large-scale dataset of labels that capture natural language instructions, time aligned with vision inputs and robot telemetry. Google DeepMind pioneered the approach in 2023 with a VLM adapted for end-to-end manipulation tasks, capable of unifying perception, reasoning and control.

The Solution: Event Annotation with Nominal

Scout AI partnered with Nominal to address these challenges, leveraging Nominal's cloud platform for storing, reviewing, and adjusting their saved live audio-transcribed annotations from the field. Nominal is now the source of truth for tagging and tracking training data across all the robots, missions, and tasks. Engineers take advantage of Nominal’s performance and ease of use for rapid search and curation of training sets.

Live Event Annotation during Field Tests

Nominal's Event catalog in Workbooks proved to be a game-changer for Scout AI's team. Engineers can now easily review events recorded from a field test, automatically time-synchronized with the video and telemetry data collected from the vehicle.

Instead of manual tagging or complex UIs, operators would simply speak into a microphone using speech-to-text to transcribe the scene live around the vehicle during a test and save events that would populate in Nominal. Engineers can then go to Nominal to review each recorded audio command provided to the autonomous system to analyze performance around that instance of time.

Nominal makes it very easy for a user to search for the following example commands and indicate exactly where it occurred around the telemetry data in the chart:

• "Reverse into a K-turn”
• "Steep Terrain ahead"
• “Negative Obstacles detected to the left"

In the past few months, Nominal has allowed Scout AI to:

• Increase annotation speed: Engineers could focus on monitoring the test and simply narrate observations, eliminating the need to look away or manipulate devices.
• Capture rich context: Audio annotations provided a deeper understanding of the operator’s intent and the surrounding circumstances, which might be missed by traditional annotation methods.
• Reduce post-test workload: Majority of initial annotations were already captured in real-time during the test, reducing need to retroactively tag data manually which streamlines the review process.
• Collaborate on results: Instantly send links to team members for immediate review of interesting test events without having to re-run post-processing scripts.

Post-Test Refinement and Retraining

After the field tests, Scout AI utilized Nominal's platform to meticulously review and refine the collected data.

• Adjusting Events: Nominal’s intuitive interface allowed engineers to easily go back to specific points in the test data, more precisely adjust the alignment of start and end times of events, and add more granular details to the initial audio annotations. For example, a spoken annotation "Obstacle detected" could be refined to precisely mark the moment the obstacle appeared and disappeared, and categorize it such as "pedestrian - unexpected crossing."
• Re-training Autonomy Software: The refined and high-quality annotated data was then directly fed back into Scout AI's machine learning pipeline. This rapid feedback loop allowed Scout AI to quickly retrain their autonomy software, addressing edge cases and improving the robustness of their algorithms.

The Impact: Faster Iteration, Smarter Autonomy

By integrating Nominal into their organization’s workflows, Scout AI achieved significant improvements in their development cycle:

Everyone in the company is able to use Nominal to self-service data for a wide variety of their own use cases and quickly share workbook links around the entire organization - from vehicle utilization, to operations efficiency, to fleet management, to sensor evaluation.

Looking Ahead

Scout AI continues to leverage Nominal's data platform to push the boundaries of autonomous technology. As early adopters of Nominal in the Autonomy space, their team provides valuable insights and feedback to our product to ensure we can support crucial workflows for robotics and autonomy development.

Through this partnership, Nominal's robust software platform has proven invaluable for large-scale data management, analysis, and annotation. We are proud to support Scout AI's mission by allowing their team to dedicate more time to the demanding process of developing reliable autonomous defense systems, rather than managing infrastructure or tooling.