空间推理
Gemini Robotics ER 模型可以指向对象、在视频中跟踪它们、使用边界框检测它们并生成运动轨迹。本页上的所有示例均使用自然语言提示 generateContent.
有关完整的可运行代码,请参阅 机器人食谱.
指向物体 #
以下示例查找图像中的特定对象并返回其标准化后的对象 [y, x] 坐标:
from google import genai
from google.genai import types
PROMPT = """
Point to no more than 10 items in the image. The label returned
should be an identifying name for the object detected.
The answer should follow the json format: [{"point": <point>,
"label": <label1>}, ...]. The points are in [y, x] format
normalized to 0-1000.
"""
client = genai.Client()
# Load your image
with open("my-image.png", 'rb') as f:
image_bytes = f.read()
image_response = client.models.generate_content(
model="gemini-robotics-er-2-preview",
contents=[
types.Part.from_bytes(
data=image_bytes,
mime_type='image/png',
),
PROMPT
],
config=types.GenerateContentConfig(
thinking_config=types.ThinkingConfig(thinking_level="high")
)
)
print(image_response.text)
# First, ensure you have the image file locally.
# Encode the image to base64
IMAGE_BASE64=$(base64 -w 0 my-image.png)
curl -X POST \
"https://generativelanguage.googleapis.com/v1beta/models/gemini-robotics-er-2-preview:generateContent" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"contents": [
{
"parts": [
{
"inlineData": {
"mimeType": "image/png",
"data": "'"${IMAGE_BASE64}"'"
}
},
{
"text": "Point to no more than 10 items in the image. The label returned should be an identifying name for the object detected. The answer should follow the json format: [{\"point\": [y, x], \"label\": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000."
}
]
}
],
"generationConfig": {
"thinkingConfig": {
"thinkingLevel": "high"
}
}
}'
输出将是一个包含对象的 JSON 数组,每个对象都有一个 point (归一化 [y, x] 坐标)和一个 label 识别对象。
[
{"point": [376, 508], "label": "small banana"},
{"point": [287, 609], "label": "larger banana"},
{"point": [223, 303], "label": "pink starfruit"},
{"point": [435, 172], "label": "paper bag"},
{"point": [270, 786], "label": "green plastic bowl"},
{"point": [488, 775], "label": "metal measuring cup"},
{"point": [673, 580], "label": "dark blue bowl"},
{"point": [471, 353], "label": "light blue bowl"},
{"point": [492, 497], "label": "bread"},
{"point": [525, 429], "label": "lime"}
]
下图是如何显示这些点的示例:

跟踪视频中的对象 #
Gemini Robotics ER 2 还可以分析视频帧以跟踪一段时间内的对象。参见 视频输入 查看支持的视频格式列表。
from google import genai
from google.genai import types
client = genai.Client()
# Load your video
with open("my-video.mp4", 'rb') as f:
video_bytes = f.read()
prompt = """
Point to the red ball in every frame where it appears.
The answer should follow the json format: [{"point": [y, x],
"label": <label>}, ...]. The points are in [y, x] format
normalized to 0-1000. Return one entry per frame that contains
the object.
"""
image_response = client.models.generate_content(
model="gemini-robotics-er-2-preview",
contents=[
types.Part.from_bytes(
data=video_bytes,
mime_type='video/mp4',
),
prompt
],
config=types.GenerateContentConfig(
thinking_config=types.ThinkingConfig(thinking_level="high")
)
)
print(image_response.text)
物体检测和边界框 #
除了点之外,您还可以提示模型返回 2D 边界框,这为检测到的对象提供更多空间细节。
from google import genai
from google.genai import types
client = genai.Client()
with open("my-image.png", 'rb') as f:
image_bytes = f.read()
prompt = """
Detect all objects in this image and return bounding boxes.
The answer should follow the JSON format:
[{"label": <label>, "y": <y_min>, "x": <x_min>,
"y2": <y_max>, "x2": <x_max>}, ...]
where coordinates are normalized to 0-1000.
"""
image_response = client.models.generate_content(
model="gemini-robotics-er-2-preview",
contents=[
types.Part.from_bytes(
data=image_bytes,
mime_type='image/png',
),
prompt
],
config=types.GenerateContentConfig(
thinking_config=types.ThinkingConfig(thinking_level="low")
)
)
print(image_response.text)
轨迹 #
Gemini Robotics ER 2 可以生成定义轨迹的点序列,对于引导机器人运动非常有用。
此示例请求将红笔移动到组织器的轨迹,包括中间路径点的估计。代码已缩减为仅显示提示。
prompt = """
Generate a trajectory for the robotic arm to pick up the red pen
and place it in the organizer. Return a list of waypoints as JSON:
[{"step": <int>, "point": [y, x], "action": <description>}, ...]
where coordinates are normalized to 0-1000.
"""
为笔记本电脑腾出空间 #
此示例显示 Gemini Robotics ER 如何推理空格。该提示要求模型识别需要移动哪个对象来为另一个项目创建空间。
from google import genai
from google.genai import types
client = genai.Client()
with open('path/to/image-with-objects.jpg', 'rb') as f:
image_bytes = f.read()
prompt = """
Point to the object that I need to remove to make room for my laptop
The answer should follow the JSON format: [{"point": <point>,
"label": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000.
"""
image_response = client.models.generate_content(
model="gemini-robotics-er-2-preview",
contents=[
types.Part.from_bytes(
data=image_bytes,
mime_type='image/jpeg',
),
prompt
],
config=types.GenerateContentConfig(
thinking_config=types.ThinkingConfig(thinking_level="high")
)
)
print(image_response.text)
响应包含回答用户问题的对象的 2D 坐标,在本例中,该对象应该移动以为笔记本电脑腾出空间。
[
{"point": [672, 301], "label": "The object that I need to remove to make room for my laptop"}
]

打包午餐 #
该模型还可以提供多步骤任务的说明,并指向每个步骤的相关对象。此示例展示了模型如何计划一系列步骤来打包午餐袋。
from google import genai
from google.genai import types
client = genai.Client()
with open('path/to/image-of-lunch.jpg', 'rb') as f:
image_bytes = f.read()
prompt = """
Explain how to pack the lunch box and lunch bag. Point to each
object that you refer to. Each point should be in the format:
[{"point": [y, x], "label": }], where the coordinates are
normalized between 0-1000.
"""
image_response = client.models.generate_content(
model="gemini-robotics-er-2-preview",
contents=[
types.Part.from_bytes(
data=image_bytes,
mime_type='image/jpeg',
),
prompt
],
config=types.GenerateContentConfig(
thinking_config=types.ThinkingConfig(thinking_level="high")
)
)
print(image_response.text)
该提示的响应是一组关于如何根据图像输入打包午餐袋的分步说明。
输入图像

模型输出
Based on the image, here is a plan to pack the lunch box and lunch bag:
1. **Pack the fruit into the lunch box.** Place the [apple](apple), [banana](banana), [red grapes](red grapes), and [green grapes](green grapes) into the [blue lunch box](blue lunch box).
2. **Add the spoon to the lunch box.** Put the [blue spoon](blue spoon) inside the lunch box as well.
3. **Close the lunch box.** Secure the lid on the [blue lunch box](blue lunch box).
4. **Place the lunch box inside the lunch bag.** Put the closed [blue lunch box](blue lunch box) into the [brown lunch bag](brown lunch bag).
5. **Pack the remaining items into the lunch bag.** Place the [blue snack bar](blue snack bar) and the [brown snack bar](brown snack bar) into the [brown lunch bag](brown lunch bag).