# 空间推理

Gemini Robotics ER 模型可以指向对象、在视频中跟踪它们、使用边界框检测它们并生成运动轨迹。

有关完整的可运行代码，请参阅 [机器人食谱](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).

## 指向物体  {% #pointing-to-objects %}

以下示例查找图像中的特定对象并返回其标准化后的对象 `[y, x]` 坐标：

{% tabs %}
{% tab filename="robotics_spatial.py" label="Python" %}
```python
from google import genai

PROMPT = """
          Point to no more than 10 items in the image. The label returned
          should be an identifying name for the object detected.
          The answer should follow the json format: [{"point": <point>,
          "label": <label1>}, ...]. The points are in [y, x] format
          normalized to 0-1000.
        """
client = genai.Client()

uploaded_file = client.files.upload(file="my-image.png")

image_response = client.interactions.create(
    model="gemini-robotics-er-2-preview",
    input=[
        {
            "type": "image",
            "uri": uploaded_file.uri,
            "mime_type": uploaded_file.mime_type
        },
        {"type": "text", "text": PROMPT}
    ],
    generation_config={"thinking_level": "high"},
)

print(image_response.output_text)
```
{% /tab %}

{% tab filename="robotics_spatial.sh" label="REST" %}
```bash
# First, ensure you have the image file locally.
# Encode the image to base64
IMAGE_BASE64=$(base64 -w 0 my-image.png)

curl -X POST \
  "https://generativelanguage.googleapis.com/v1beta/interactions" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-robotics-er-2-preview",
    "input": {
      "parts": [
        {
          "inlineData": {
            "mimeType": "image/png",
            "data": "'"${IMAGE_BASE64}"'"
          }
        },
        {
          "text": "Point to no more than 10 items in the image. The label returned should be an identifying name for the object detected. The answer should follow the json format: [{\"point\": [y, x], \"label\": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000."
        }
      ]
    },
    "generation_config": {
      "thinking_config": {
        "thinking_level": "high"
      }
    }
  }'
```
{% /tab %}
{% /tabs %}

输出将是一个包含对象的 JSON 数组，每个对象都有一个 `point`
（归一化 `[y, x]` 坐标）和一个 `label` 识别对象。

{% tabs %}
{% tab filename="output.json" label="JSON" %}
```json
[
  {"point": [376, 508], "label": "small banana"},
  {"point": [287, 609], "label": "larger banana"},
  {"point": [223, 303], "label": "pink starfruit"},
  {"point": [435, 172], "label": "paper bag"},
  {"point": [270, 786], "label": "green plastic bowl"},
  {"point": [488, 775], "label": "metal measuring cup"},
  {"point": [673, 580], "label": "dark blue bowl"},
  {"point": [471, 353], "label": "light blue bowl"},
  {"point": [492, 497], "label": "bread"},
  {"point": [525, 429], "label": "lime"}
]
```
{% /tab %}
{% /tabs %}

下图是如何显示这些点的示例：

![](https://www.gstatic.com/cloud-cms/9520553bf3464dfdca8d4bf4b4da2951.png)

## 跟踪视频中的对象  {% #object-tracking-video %}

Gemini Robotics ER 2 还可以分析视频帧以跟踪一段时间内的对象。参见 [视频输入](/docs/video-understanding#supported-formats)
查看支持的视频格式列表。

{% tabs %}
{% tab filename="robotics_tracking.py" label="Python" %}
```python
from google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="my-video.mp4")

prompt = """
          Point to the red ball in every frame where it appears.
          The answer should follow the json format: [{"point": [y, x],
          "label": <label>}, ...]. The points are in [y, x] format
          normalized to 0-1000. Return one entry per frame that contains
          the object.
        """

image_response = client.interactions.create(
  model="gemini-robotics-er-2-preview",
  input=[
    {
        "type": "video",
        "uri": uploaded_file.uri,
        "mime_type": uploaded_file.mime_type
    },
    {"type": "text", "text": prompt}
  ],
)

print(image_response.output_text)
```
{% /tab %}
{% /tabs %}

## 物体检测和边界框  {% #object-detection-bounding-boxes %}

除了点之外，您还可以提示模型返回 2D 边界框，这为检测到的对象提供更多空间细节。

{% tabs %}
{% tab filename="robotics_bounding_boxes.py" label="Python" %}
```python
from google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="my-image.png")

prompt = """
          Detect all objects in this image and return bounding boxes.
          The answer should follow the JSON format:
          [{"label": <label>, "y": <y_min>, "x": <x_min>,
            "y2": <y_max>, "x2": <x_max>}, ...]
          where coordinates are normalized to 0-1000.
        """

image_response = client.interactions.create(
  model="gemini-robotics-er-2-preview",
  input=[
    {
        "type": "image",
        "uri": uploaded_file.uri,
        "mime_type": uploaded_file.mime_type
    },
    {"type": "text", "text": prompt}
  ],
)

print(image_response.output_text)
```
{% /tab %}
{% /tabs %}

## 轨迹  {% #trajectories %}

Gemini Robotics ER 2 可以生成定义轨迹的点序列，对于引导机器人运动非常有用。

此示例请求将红笔移动到组织器的轨迹，包括中间路径点的估计。代码已缩减为仅显示提示。

{% tabs %}
{% tab filename="robotics_trajectories.py" label="Python" %}
```python
prompt = """
          Generate a trajectory for the robotic arm to pick up the red pen
          and place it in the organizer. Return a list of waypoints as JSON:
          [{"step": <int>, "point": [y, x], "action": <description>}, ...]
          where coordinates are normalized to 0-1000.
        """
```
{% /tab %}
{% /tabs %}

## 为笔记本电脑腾出空间  {% #making-room-laptop %}

此示例显示 Gemini Robotics ER 如何推理空格。该提示要求模型识别需要移动哪个对象来为另一个项目创建空间。

{% tabs %}
{% tab filename="robotics_spatial_reasoning.py" label="Python" %}
```python
from google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="path/to/image-with-objects.jpg")

prompt = """
          Point to the object that I need to remove to make room for my laptop
          The answer should follow the JSON format: [{"point": <point>,
          "label": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000.
        """

image_response = client.interactions.create(
  model="gemini-robotics-er-2-preview",
  input=[
    {
        "type": "image",
        "uri": uploaded_file.uri,
        "mime_type": uploaded_file.mime_type
    },
    {"type": "text", "text": prompt}
  ],
)

print(image_response.output_text)
```
{% /tab %}
{% /tabs %}

响应包含回答用户问题的对象的 2D 坐标，在本例中，该对象应该移动以为笔记本电脑腾出空间。

```json
[
  {"point": [672, 301], "label": "The object that I need to remove to make room for my laptop"}
]
```

![](https://www.gstatic.com/cloud-cms/2e93adec0bc100a1db910fd7ea48e758.png)

## 打包午餐  {% #packing-lunch %}

该模型还可以提供多步骤任务的说明，并指向每个步骤的相关对象。此示例展示了模型如何计划一系列步骤来打包午餐袋。

{% tabs %}
{% tab filename="robotics_packing.py" label="Python" %}
```python
from google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="path/to/image-of-lunch.jpg")

prompt = """
          Explain how to pack the lunch box and lunch bag. Point to each
          object that you refer to. Each point should be in the format:
          [{"point": [y, x], "label": }], where the coordinates are
          normalized between 0-1000.
        """

image_response = client.interactions.create(
  model="gemini-robotics-er-2-preview",
  input=[
    {
        "type": "image",
        "uri": uploaded_file.uri,
        "mime_type": uploaded_file.mime_type
    },
    {"type": "text", "text": prompt}
  ],
)

print(image_response.output_text)
```
{% /tab %}
{% /tabs %}

该提示的响应是一组关于如何根据图像输入打包午餐袋的分步说明。

**输入图像**

![](https://www.gstatic.com/cloud-cms/ad80224a995418224640d876f08c5e1e.png)

**模型输出**

```none
Based on the image, here is a plan to pack the lunch box and lunch bag:

1.  **Pack the fruit into the lunch box.** Place the [apple](apple), [banana](banana), [red grapes](red grapes), and [green grapes](green grapes) into the [blue lunch box](blue lunch box).
2.  **Add the spoon to the lunch box.** Put the [blue spoon](blue spoon) inside the lunch box as well.
3.  **Close the lunch box.** Secure the lid on the [blue lunch box](blue lunch box).
4.  **Place the lunch box inside the lunch bag.** Put the closed [blue lunch box](blue lunch box) into the [brown lunch bag](brown lunch bag).
5.  **Pack the remaining items into the lunch bag.** Place the [blue snack bar](blue snack bar) and the [brown snack bar](brown snack bar) into the [brown lunch bag](brown lunch bag).
```

## 后续步骤  {% #whats-next %}

* [智能体能力](/docs/robotics-agentic) — 代码执行、仪器读取、图像注释。
* [任务编排](/docs/robotics-orchestration) — 使用自定义机器人 API 执行长期任务。
* [机器人流式处理](/docs/robotics-streaming) — 实时双向流（仅限 Gemini Robotics ER 2）。
* [视频理解](/docs/robotics-video-progress) — 时刻查找和进度分类（仅限 Gemini Robotics ER 2）。
