Anthropic and Andon Labs Publish Project Pilot Research on AI Drone Navigation

Anthropic and research firm Andon Labs released findings from Project Pilot on July 24, 2026, evaluating whether frontier artificial intelligence models can autonomously control a commercial drone to perform indoor surveillance. Benchmarking 15 models across five core operational tasks, researchers found that while AI agents excel at person detection and target tracking, physical flight remains constrained by 3D environment reconstruction errors.

Deconstructing Aerial Surveillance into Drone-Bench Tasks

Project Pilot expands on Anthropic’s previous physical-world evaluations, including Project Vend and Project Fetch, by examining how AI models interact with physical hardware. To measure model capability in real-time hardware control, Andon Labs developed Drone-Bench in consultation with Anthropic.

Anthropic did not receive direct access to the benchmark software, leaving Andon Labs to execute the evaluations. The primary objective required an AI agent to direct a quad-rotor drone through an indoor office layout to locate and track a specific person using a reference photo.

The evaluation framework broke down the overall surveillance objective into five distinct sub-tasks:

• Reconstruct: Convert video footage of the office into a 3D environmental model and generate a 2D floorplan obstacle map.
• Localize: Compare incoming real-time camera frames against reference video poses to identify the drone’s position on the obstacle map.
• Navigate: Plan flight paths between rooms on the 2D map and fly the route, continuously updating position data to account for control noise.
• Detect: Scan the drone’s live video stream to identify the target person using a detection algorithm built from a reference image, outputting bounding boxes around the individual.
• Follow: Issue continuous flight commands based on target bounding boxes to keep the subject centered in frame at a consistent distance during movement.

To establish a baseline, human-AI engineering teams developed specialized algorithms for each component task. Software simulations were created for each sub-task to allow automated, high-speed testing across multiple runs prior to physical drone deployment.

Benchmark Results and Model Problem-Solving

Andon Labs evaluated 15 AI models from three developers: GPT-4o, GPT-4o Nov, o1, o3, Claude Opus 4, Gemini 2.5 Pro, GPT-5, Gemini 3.1 Pro, Opus 4.5, GPT-5.2, Opus 4.7, GPT-5.5, Opus 4.8, Fable 5, and GPT-5.6 Sol.

Test results showed a clear division in capability depending on the sub-task. Across all evaluated systems, models achieved their highest success rates in the target detection and target following phases, while scoring lowest in 3D reconstruction and position localization.

Among the tested systems, Claude Fable 5 achieved the highest overall score, exceeding the reference baseline on four of the five sub-tasks in its best runs. During the simulation phase, Fable 5 demonstrated distinct problem-solving methods before submitting its code implementations:

• Camera Calibration: The model analyzed simulated video frames of an indoor corridor, using grout line alignments on the floor to calculate the scene’s vanishing point. Fable 5 determined the drone camera’s tilt angle to within four degrees of its true value.
• Local Simulation: Fable 5 built a standalone 2D top-down map of the target environment to test and debug its control routines locally before submitting its final execution code.

Physical Hardware Execution and Consistency Gaps

After simulation testing, researchers deployed Fable 5 to operate a physical DJI Tello EDU quad-rotor drone, a hardware unit retailing for $129, inside an office building with a consenting participant.

In physical execution, Fable 5 outperformed the reference baseline algorithm at keeping the target person centered during the follow phase. However, early errors during the 3D reconstruction stage compounded into localization and navigation failures. During room-to-room navigation, the model misidentified a solid wall as an open doorway and flew the drone directly into it.

The evaluation also measured operational consistency across repeated trials. While frontier models matched or exceeded the baseline on at least one trial for four out of five tasks, Fable 5 averaged performance at or above the baseline on only three out of five tasks over 10 simulation runs. Data from the benchmark indicates a roughly six-month lag between a model’s single best performance and its ability to replicate that performance consistently.

Researchers noted several physical constraints during the experiment. The drones operated exclusively at low speeds within a single indoor office floorplan using a small group of known participants, and testing did not include outdoor flights or crowded public environments.