Why AI planning and real-time robot control belong in separate layers
Foundation models can interpret open-ended tasks; robot execution still depends on bounded timing, constraints and defined failure behavior. Reliable Physical AI systems give each layer a different job.
The short answer: language and vision models are useful precisely because they can work with ambiguity. A manipulator or mobile base, however, needs measurable deadlines, predictable execution and explicit behavior when something goes wrong. Separating those concerns preserves AI flexibility without placing open-ended inference inside a high-frequency control loop.
One instruction becomes at least four engineering layers
| Layer | Primary input | Output | Question it must answer |
|---|---|---|---|
| Intent | Language and task context | Structured goal and constraints | What outcome is requested, and what must not be violated? |
| Perception | Cameras, state and workpiece data | Object pose and scene state | Where is the target, and is the evidence sufficient to act? |
| Task and motion planning | Goal, robot model and scene | Reviewable stages and trajectories | Is the motion reachable, collision-aware and within constraints? |
| Real-time execution | Trajectory, feedback and safety state | Joint/base commands and feedback | Do cycle time, jitter and fault handling meet the application need? |
MoveIt Task Constructor models a complex task as interdependent stages and passes state between them. Its PlanningScene represents both robot state and the surrounding world. That makes a request such as “pick up the cable” a sequence of inspectable steps—not a single model response. The official MTC documentation also keeps planning and execution as distinct operations.
Low latency is not the definition of real-time
The ROS 2 real-time design material frames the problem around deadlines, predictability and the failure mode after a missed deadline. A call that is fast on average can still be unsuitable for deterministic control if it occasionally blocks. The design guidance discusses separating real-time and non-real-time work and avoiding sources of nondeterminism such as dynamic allocation, page faults and lower-priority work blocking a real-time thread.
The planning layer should produce a constrained task—not motor commands
- Translate language into a structured goal. Name the object, destination, allowed contact, speed limits and stopping conditions.
- Validate the goal against the current scene. Check perception confidence, robot state, collision geometry and tool state.
- Generate reviewable stages and trajectories. Check reachability, joint limits, collision constraints and time parameterization before execution.
- Hand execution to the real-time layer. Read feedback, issue commands and monitor faults on the required cycle.
- Return outcome and exceptions upstream. Success, deviation, grasp failure and safety events become inputs to the next planning decision.
Industrial networking solves a different deterministic problem. EtherCAT Technology Group describes on-the-fly frame processing and distributed clocks for predictable transfer and synchronization. That supports the execution layer; it does not replace task or motion planning.
Architecture review checklist
- Does language output pass through a structured task and constraint check?
- Do perception results carry confidence, timestamps and failure handling?
- Does planning use current robot state and an updated collision scene?
- Are cycle time, jitter, bus synchronization and load conditions measured?
- Are emergency stop, protective stop and limits independent from open-ended AI?
- Does the end-to-end demonstration include recovery, not only one successful run?
Matrix Dimension perspective
The useful product boundary is a clear contract between layers. AI proposes intent and candidate plans; the planning layer produces a trajectory that respects robot and scene constraints; the real-time layer executes only confirmed commands and returns state continuously. Product claims should still be published from tests on a named robot, load, network and task—not inferred from architecture alone.
Safety boundary: ISO 10218-1:2025 addresses safety requirements for the industrial robot, while complete applications and integration also involve ISO 10218-2:2025. This article explains architecture; it does not replace an application risk assessment, functional-safety design or conformity work.
Frequently asked questions
Can a foundation model directly command robot joints?
Open-ended model output should not become joint commands without constraints. A safer architecture converts intent into a structured goal, checks scene, collision, limits and safety state, then hands execution to a real-time layer.
Does real-time simply mean faster?
No. Real-time is about completing work predictably before a defined deadline and specifying what happens if that deadline is missed.
Why does a dual-arm system make layering more important?
Two arms add synchronization, mutual collision, shared-object and contact-force constraints. Planning handles coordination while the execution layer maintains deterministic feedback and synchronization.
Sources
These primary references support the architecture, real-time and safety boundaries discussed above.
Evaluating robot control, bimanual manipulation or a mobile platform?
Talk to Matrix Dimension →