Spatial Concept Bottleneck for Interpretable Robot Learning

Method

We define a spatial concept as a geometric relationship between entities corresponding to the task and the robot, which can be computed directly from the scene's geometry. The perception backend \(g\) predicts the spatial concept \(\hat{c}\) and the actor/controller head maps the concept and optionally the proprioceptive state \(\rho_t\) to action \(\hat{y}_t = f(\hat{c}_t, \rho_t)\). The robot executes \(\hat{y}_t\), observes \(o_{t+1}\), and the loop repeats.

\[ c^k_t = \psi\big(a^k_t, r\big), \qquad r \in R_t \]

The task mandates an exocentric entity \(a^k_t\), such as a navigation sub-goal or a destination point for the gripper, and the robot provides an egocentric entity \(r\).

Navigation

\[ C_t(u) = \psi\big(a^1_t, X_t(u)\big) = \big\lVert a^1_t - X_t(u) \big\rVert_2 \]

We define a spatial concept bottleneck for navigation as a visual representation of navigation affordances in the robot's current view. The sub-goal pixel is the only anchor, and the egocentric entity is the 3D point \(X_t(u)\) of each pixel \(u\) of the current observation.

Manipulation

\[ c^k_t = \psi (a^k_t, r_t) = \beta\left(a^k_t - r_t\right), \qquad k \in \{1, 2\} \]

For manipulation, we define two exocentric spatial entities: source and destination, corresponding respectively to the pick and place objects. The source object defines where to act as \(a^1_t\) and the destination defines where to place as \(a^2_t\). We use a single egocentric spatial entity that represents the end effector position \(r_t\), and \(\beta\) discretizes each axis of the difference in positions into 13 bins on a logarithmic scale. The concept therefore consists of six values, the \(x\), \(y\) and \(z\) offsets of the source and of the destination from the gripper. The bins are narrow near the gripper and wide far from it, giving centimetre precision close to the object and a coarse direction elsewhere.

The concept along one rollout (LIBERO-Spatial, true concepts). Each box is the bin \(\beta\) puts the source (orange) or the destination (blue) in, drawn around the gripper. Green represents the object's true position. Below each frame, the 13 bins of one axis to scale, with the source's and the destination's bin. The triangle is the gripper.

Results

Navigation

Spatial CBM is on par or better than predicting Direct Actions.

SPL and SSPL on four navigation tasks, with the best non-oracle scores in bold.

Manipulation

With true concepts the spatial bottleneck performs on par with Direct Action, and swapping the source and destination brings the GT policy down to 4.5%.

Success (%) on the ten LIBERO-Spatial tasks over two training seeds (^†one seed).

Intervention

The key benefit of the bottleneck structure of CBMs is that they allow intervention of human-interpretable concepts at test time. They provide insight into the model's intent which it has to use to generate outputs. The intervener is provided all information of \(x_t\) and the predicted spatial concept \(\hat{c}_t\), and the oracle has additional access to the true concept \(c_t\). Based on the design of the spatial concept bottleneck, the intervener provides exocentric entities \(\tilde{a}^k_t\), while the robot's own ego entities are unchanged. We are thus able to re-generate the entire concept on intervention without changing all the exocentric entities.

\[ \tilde{c}^k_t = \psi\big(\tilde{a}^k_t, r\big), \qquad \tilde{y}_t = f(\tilde{c}_t, \rho_t) \]

Manipulation

In manipulation, the intervener provides the source or destination point \(\tilde{a}^k_t\), and the concept is regenerated. We use the oracle as the intervener and test three uses of the same operation. For correction, the oracle provides the true source and destination points at every step. For steering, we train the action expert with a single sentence shared across all ten tasks in place of the instruction, so that the concept is the only input that tells the robot which bowl to pick, and the oracle provides the position of the other, identical bowl as the source. For an unseen object, the oracle provides the position of the ramekin, which is never moved in any demonstration, and we compare against Direct Action whose instruction asks for the ramekin.

Navigation

We split the spatial concept bottleneck of navigation, the costmap, into three sectors: left, center, and right, which map the three most common commands an intervener would likely give. The VLM is provided the last six camera frames taken five steps apart, eight frames from the earlier traversal, the current view, the sub-goal-containing view, and the final destination image. Most importantly, the VLM is provided only the predicted costmap \(\hat{c}_t\) (and never the true costmap \(c_t\)) to propose an intervention. It is required to output “none” if no change is required, or the proposed sector along with a brief reasoning. We use the oracle to detect differences between the predicted and true costmaps and trigger the intervention.

Each video shows one episode without (left) and with (right) intervention: the RGB view and the costmap (blue is low cost, white square: least-cost cell).

More qualitative results

Navigation: how to read these panels

Each panel shows one decision of the VLM. The top lines give the task and step, the VLM's chosen sector next to the oracle's and the predicted costmap's sectors, and the VLM's stated reason. The strip below shows the walkthrough frames the VLM was given, from the robot's position to the destination (green border). The left block is what the VLM saw, the current view with the predicted costmap over it, where the cross marks the costmap's lowest-cost cell. The middle block is the VLM's choice, where the circle marks the new target. The right column shows the reference and destination frames.

Navigation: successful interventions

Navigation: failure cases

The VLM's stated reason shows why. In Alt-goal the goal is not at the end of the recorded route, so relying on the route frames over the destination image misleads the VLM.

Manipulation

Each figure shows two runs from the same start: the top row without the intervention and the bottom row with it. Orange and blue boxes show where the concept places the source and the destination, and green marks the true positions.