Semantic Maps and 3D Scene Graphs

A geometric map says where walls are. A semantic map also says what things are. A 3D scene graph organizes that knowledge in layers, building, rooms, places, and objects, linked by relations such as "in" and "on". Robots use it to follow requests like "bring me the mug from the kitchen".

Ask the map

The layers

A 3D scene graph, as in Hydra or Kimera, stacks several layers: a building node, rooms, places the robot can stand, and objects, each object node carrying its class, pose, and size. Edges inside a layer say what connects to what, such as doorways between rooms. Edges between layers say what contains what.

$$\mathcal{G} = (V, E), \qquad V = V_{\text{building}} \cup V_{\text{rooms}} \cup V_{\text{places}} \cup V_{\text{objects}}$$

What to notice

Each question is answered by walking the graph, not the raw map. "Where is the mug" follows the mug up to its table and then to its room. "Which rooms connect" is a shortest path over the room layer. Object labels come from perception, today often from open-vocabulary models, and the graph turns them into knowledge a planner or a language model can reason with.