Distribution Shift and DAgger

Behavior cloning fails because the robot ends up in states the expert never showed. DAgger, Dataset Aggregation, fixes that by letting the robot drive with its current policy, then asking the expert to label the correct action for every state the robot visited. Each round adds exactly the data the policy was missing.

Noise when the policy runs

The DAgger loop

Start with a policy trained on expert demonstrations. Then repeat: run the current policy, record the states $s$ it visits, have the expert label each with its action $\pi^*(s)$, add those pairs to the dataset, and retrain:

$$\mathcal{D} \leftarrow \mathcal{D} \cup \{(s,\ \pi^*(s)) : s \sim \pi_{\text{current}}\}, \qquad \pi \leftarrow \text{train}(\mathcal{D})$$

This keeps the training data matched to the states the robot really reaches, which removes the compounding error problem.

What to notice

Round 0 is plain behavior cloning from one demonstration, so many rollouts fail. Each round adds corrections exactly where the robot went wrong, and the success rate climbs within a few rounds. The cost is that the expert must be available to label the robot's states, which is easy in simulation and harder with a human, so practical versions such as HG-DAgger let a person take over only when the robot is about to fail.