Projects/Flagship2025–

Agentic Robotics Framework

NYU Abu Dhabi · CAIR · PhD research, University of the Aegean

CHALLENGE

Let facility operators command a field robot for inspection without specialist interfaces or task-specific model training.

MY CONTRIBUTION

Designed and built the agent harness, MCP skill servers, and orchestration that connect natural-language commands to Spot inspection skills.

OUTCOME

A working inspection stack on Boston Dynamics Spot, commanded by voice and text through a local model. The paper is under review at the Journal of Field Robotics.

Project overview

This is my current research system for natural-language field inspection. I designed and built a three-layer agent harness — interface, orchestration, and decision-and-skill — joined only by tool calls. Robot and sensor skills are independent Model Context Protocol servers, so a locally served open-weight model chooses actions from tool descriptions without fine-tuning. Every call is checked before it reaches the Boston Dynamics Spot SDK, and the model cannot clear an operator e-stop. The working stack accepts voice and text, keeps session memory, and drives Spot through navigation, vision, and thermal inspection skills. Two Spots with different payloads can split one mission: the arm robot opens a door or turns a valve, while the robot carrying LiDAR and an acoustic camera navigates and acquires the inspection data. The paper is under review at the Journal of Field Robotics.

System Record

My role: CONTRIBConceived CONTRIBDesigned CONTRIBBuilt CONTRIBSystem Integration CONTRIBExperimental Development
Domain: DOMAINEmbodied & Physical AI DOMAINPerception & Sensing
Application: APPIndustrial Inspection
Platforms: PLATFORMBoston Dynamics Spot
Methods: METHODThermal Imaging
Outcomes: OUTCOMEDeployed Prototype
Period: 2025–

Development & Validation

Three layers — a messaging interface, n8n orchestration, and a language-model skill loop — exchange natural language upward and structured tool calls downward.
Spot locomotion, vision, and thermal sensing are separate MCP servers. The model reads descriptions, not robot code, and is not trained on the inspection task.
Each tool server validates arguments before the Spot SDK runs them. The agent can request a stop; it cannot release an operator e-stop.
Voice and text over Telegram, plus browser and webhook entry points, share one orchestration path and PostgreSQL session memory.
The deployment runs on Spot with an arm and thermal payload, with on-robot perception and the language model served locally.
Two Spots carry different payloads and can cooperate on one mission. The arm robot opens a door or turns a valve. The robot with LiDAR and an acoustic camera handles navigation and data acquisition.