Autonomous Driving Pipeline

A self-driving car runs a chain of modules many times per second. Perception turns sensor data into a list of objects. Prediction guesses where each will go. Planning picks a safe, comfortable path among many candidates. Control turns that path into steering and pedal commands. Each stage passes a simpler, more abstract picture to the next.

Show perception (detections) Show prediction (future paths) Show planning (candidate trajectories) A car cuts in from the next lane

Four stages

Perception fuses camera, LiDAR, and radar into tracked objects with position, size, and velocity. Prediction forecasts each object's future, often several possible futures with probabilities. Planning samples or optimizes candidate trajectories and scores each by a cost:

$$J(\tau) = w_{\text{safe}}\,c_{\text{collision}} + w_{\text{lane}}\,c_{\text{offset}} + w_{\text{comfort}}\,c_{\text{jerk}} + w_{\text{progress}}\,c_{\text{slow}}$$

Control tracks the chosen trajectory, for example with pure pursuit or MPC.

What to notice

When a car starts to cut in, its predicted path crosses the ego lane, so candidate trajectories that keep full speed turn red, and the planner chooses one that eases off. The planner never sees raw sensor data, only the predicted objects, so errors early in the chain flow downstream. Some newer systems train one large network end to end, from sensors to trajectory, but most deployed systems still keep these stages separate so each can be tested.