Edge Computing and Inference Budgets

Modern robots run large neural networks, but every millisecond counts. Running a model onboard, at the edge, avoids network delay but is limited by the robot's power and compute. Sending work to a nearby server or the cloud gives far more compute, but adds network time and depends on a good connection. Deciding what runs where is an inference budget problem.

Object detector runs on Motion planner runs on Language model (task planning) runs on Reaction budget for detect and plan Wi-Fi is congested

The budget

Each stage costs its compute time on the chosen device, and each change of device costs a network hop. The fast path, detection then planning, must fit within the reaction budget, while slower tasks like language planning can take longer:

$$t_{\text{fast}} = t_{\text{detect}}(d_1) + t_{\text{net}}(d_0 \to d_1) + t_{\text{plan}}(d_2) + t_{\text{net}}(d_1 \to d_2) + t_{\text{net}}(d_2 \to d_0)$$

All numbers on this page are illustrative orders of magnitude, not measurements of any particular hardware.

What to notice

Keeping detection and planning onboard usually fits the budget, though detection is slower on a small onboard GPU. Moving detection to a nearby server speeds the model up, but the network round trip eats much of the gain, and congested Wi-Fi breaks the budget entirely. The cloud is fine for the language model, which decides what to do next every few seconds, but too slow for the reaction path. A common split keeps anything safety-critical onboard and sends slow, heavy reasoning off-board, with a safe fallback if the connection drops.