Point Cloud Basics: Downsampling and Normals

Depth sensors produce huge, uneven clouds of points. The first steps are almost always the same: downsample with a voxel grid so the density is even and the cloud is smaller, then estimate the surface direction, the normal, at each point. Normals are needed for registration, segmentation, and grasping.

Voxel size Neighbors for the normal Sensor noise Show the raw points

Voxel grid downsampling

Split space into cubes of side $r$ and replace all the points in each cube with their average. Dense areas lose many points, sparse areas few, so the result is even.

Normals from neighbors

Take the $k$ nearest neighbors of a point and compute their covariance. The normal is the direction of least spread, the eigenvector with the smallest eigenvalue:

$$C = \frac{1}{k}\sum_{i=1}^{k} (\mathbf{p}_i - \bar{\mathbf{p}})(\mathbf{p}_i - \bar{\mathbf{p}})^\top, \qquad \mathbf{n} = \mathbf{v}_{\min}(C)$$

What to notice

A larger voxel size shrinks the cloud a lot but blurs sharp corners. Normals from too few neighbors are jittery from noise. From too many, they smooth over corners, where the neighbors come from two different surfaces. In practice the neighborhood is chosen relative to the noise and the size of the details that matter.