Synthetic Data and Data Augmentation

Real robot data is expensive, and it rarely covers every condition the robot will meet. Two cheap fixes are data augmentation, making altered copies of real examples, such as brighter, darker, or shifted, and synthetic data, rendered images with randomized lighting and textures. Both teach a model what does not matter, so it focuses on what does.

Augment the real images (brightness and color jitter) Add synthetic rendered images How dim the test lighting is Real training images per class

Teaching invariance

Each image is reduced here to two features: its hue and its brightness. The two classes, red and orange balls, differ in hue. But in the real training photos the red balls sat in bright light and the orange balls happened to be in shade, so brightness also separates them. A classifier can latch onto that accidental cue, a spurious correlation. Augmentation applies random changes $T$ that should not change the label:

$$\mathcal{D}_{\text{aug}} = \{(T(x),\ y) : (x, y) \in \mathcal{D},\ T \sim \text{brightness, color, crop, blur}\}$$

Synthetic data does the same with rendered images, whose lighting and appearance can be randomized freely.

What to notice

Without augmentation, the classifier does well in bright light but fails as the test lighting dims, since dim red balls look like the shaded orange ones it trained on. Augmenting brightness spreads the training data across lighting, so the boundary follows hue, and test accuracy holds. Synthetic images help similarly and add volume, though a gap between rendered and real appearance can remain. The classifier is a simple nearest-neighbor rule to keep the idea visible.