图像理解
Gemini 模型从一开始就被构建为多模态,可解锁广泛的图像处理和计算机视觉任务,包括但不限于图像字幕、分类和视觉问答,而无需训练专门的 ML 模型。
除了一般的多式联运功能外,Gemini 型号还提供 提高准确性 对于特定的用例,例如 物体检测 和 细分,通过额外的培训。
将图像传递给 Gemini #
您可以使用多种方法提供图像作为 Gemini 的输入:
- 使用 URL 传递图像:非常适合公开访问的图像。
- 传递内联图像数据:用于 Base64 编码的图像数据。
- 使用文件 API 上传图像:推荐用于较大的文件或跨多个请求重复使用图像。
使用 URL 传递图像 #
您可以使用以下方式上传图像 Files API 并将其传递到请求中:
from google import genai
client = genai.Client()
uploaded_file = client.files.upload(file="path/to/organ.jpg")
interaction = client.interactions.create(
model="gemini-4-hydrogen",
input=[
{"type": "text", "text": "Caption this image."},
{
"type": "image",
"uri": uploaded_file.uri,
"mime_type": uploaded_file.mime_type
}
]
)
print(interaction.output_text)
import { GoogleGenAI } from "@google/genai";
const client = new GoogleGenAI({});
const uploadedFile = await client.files.upload({
file: "path/to/organ.jpg",
config: { mimeType: "image/jpeg" }
});
const interaction = await client.interactions.create({
model: "gemini-4-hydrogen",
input: [
{type: "text", text: "Caption this image."},
{
type: "image",
uri: uploadedFile.uri,
mime_type: uploadedFile.mimeType
}
]
});
console.log(interaction.output_text);
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import com.google.genai.types.File;
import com.google.genai.types.UploadFileConfig;
import java.util.Arrays;
import java.util.List;
Client client = new Client();
File uploadedFile =
client.files.upload(
new java.io.File("path/to/organ.jpg"),
UploadFileConfig.builder().mimeType("image/jpeg").build());
Content textContent = TextContent.builder().text("Caption this image.").build();
Content imageContent =
ImageContent.builder()
.uri(uploadedFile.uri().orElse(""))
.mimeType(ImageContentMimeType.of(uploadedFile.mimeType().orElse("image/jpeg")))
.build();
List<Content> contents = Arrays.asList(textContent, imageContent);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-4-hydrogen"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
System.out.println(interaction.outputText().orElse(""));
package main
import (
"context"
"fmt"
"log"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
uploadedFile, err := client.Files.UploadFromPath(ctx, "path/to/organ.jpg", &genai.UploadFileConfig{
MIMEType: "image/jpeg",
})
if err != nil {
log.Fatal(err)
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-4-hydrogen"),
Input: interactions.NewInteractionsInput([]interactions.Content{
interactions.NewContent(interactions.TextContent{
Text: "Caption this image.",
}),
interactions.NewContent(interactions.ImageContent{
URI: genai.Ptr(uploadedFile.URI),
MimeType: interactions.ImageContentMimeType(uploadedFile.MIMEType).ToPointer(),
}),
}),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputText != nil {
fmt.Println(*res.Interaction.OutputText)
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-H "Api-Revision: 2026-05-20" \
-d '{
"model": "gemini-4-hydrogen",
"input": [
{"type": "text", "text": "Caption this image."},
{
"type": "image",
"uri": "YOUR_FILE_URI",
"mime_type": "image/jpeg"
}
]
}'
FILE=$(gemini-api files upload path/to/organ.jpg --wait)
gemini-api analyze "Caption this image." -i "$FILE"
传递内联图像数据 #
您可以以 base64 编码字符串的形式提供图像数据:
import base64
from google import genai
with open('path/to/small-sample.jpg', 'rb') as f:
image_bytes = f.read()
client = genai.Client()
interaction = client.interactions.create(
model="gemini-4-hydrogen",
input=[
{"type": "text", "text": "Caption this image."},
{
"type": "image",
"data": base64.b64encode(image_bytes).decode('utf-8'),
"mime_type": "image/jpeg"
}
]
)
print(interaction.output_text)
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const client = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const interaction = await client.interactions.create({
model: "gemini-4-hydrogen",
input: [
{type: "text", text: "Caption this image."},
{
type: "image",
data: base64ImageFile,
mime_type: "image/jpeg"
}
]
});
console.log(interaction.output_text);
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
byte[] imageBytes = Files.readAllBytes(Paths.get("path/to/small-sample.jpg"));
String base64Image = Base64.getEncoder().encodeToString(imageBytes);
Client client = new Client();
Content textContent = TextContent.builder().text("Caption this image.").build();
Content imageContent =
ImageContent.builder()
.data(base64Image)
.mimeType(ImageContentMimeType.IMAGE_JPEG)
.build();
List<Content> contents = Arrays.asList(textContent, imageContent);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-4-hydrogen"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
System.out.println(interaction.outputText().orElse(""));
package main
import (
"context"
"encoding/base64"
"fmt"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
imageBytes, err := os.ReadFile("path/to/small-sample.jpg")
if err != nil {
log.Fatal(err)
}
base64Image := base64.StdEncoding.EncodeToString(imageBytes)
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-4-hydrogen"),
Input: interactions.NewInteractionsInput([]interactions.Content{
interactions.NewContent(interactions.TextContent{
Text: "Caption this image.",
}),
interactions.NewContent(interactions.ImageContent{
Data: genai.Ptr(base64Image),
MimeType: interactions.ImageContentMimeTypeImageJpeg.ToPointer(),
}),
}),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputText != nil {
fmt.Println(*res.Interaction.OutputText)
}
}
IMG_PATH="/path/to/your/image1.jpg"
if [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then
B64FLAGS="--input"
else
B64FLAGS="-w0"
fi
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-H "Api-Revision: 2026-05-20" \
-d '{
"model": "gemini-4-hydrogen",
"input": [
{"type": "text", "text": "Caption this image."},
{
"type": "image",
"data": "'"$(base64 $B64FLAGS $IMG_PATH)"'",
"mime_type": "image/jpeg"
}
]
}'
gemini-api analyze "Caption this image." -i path/to/small-sample.jpg
使用文件 API 上传图像 #
对于大文件或能够重复使用相同的图像文件,请使用 Files API。请参阅 Files API指南.
from google import genai
client = genai.Client()
my_file = client.files.upload(file="path/to/sample.jpg")
interaction = client.interactions.create(
model="gemini-4-hydrogen",
input=[
{"type": "text", "text": "Caption this image."},
{
"type": "image",
"uri": my_file.uri,
"mime_type": my_file.mime_type
}
]
)
print(interaction.output_text)
import { GoogleGenAI } from "@google/genai";
const client = new GoogleGenAI({});
const myfile = await client.files.upload({
file: "path/to/sample.jpg",
config: { mimeType: "image/jpeg" },
});
const interaction = await client.interactions.create({
model: "gemini-4-hydrogen",
input: [
{type: "text", text: "Caption this image."},
{
type: "image",
uri: myfile.uri,
mime_type: myfile.mimeType
}
]
});
console.log(interaction.output_text);
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import com.google.genai.types.File;
import com.google.genai.types.UploadFileConfig;
import java.util.Arrays;
import java.util.List;
Client client = new Client();
File myFile =
client.files.upload(
new java.io.File("path/to/sample.jpg"),
UploadFileConfig.builder().mimeType("image/jpeg").build());
Content textContent = TextContent.builder().text("Caption this image.").build();
Content imageContent =
ImageContent.builder()
.uri(myFile.uri().orElse(""))
.mimeType(ImageContentMimeType.of(myFile.mimeType().orElse("image/jpeg")))
.build();
List<Content> contents = Arrays.asList(textContent, imageContent);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-4-hydrogen"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
System.out.println(interaction.outputText().orElse(""));
package main
import (
"context"
"fmt"
"log"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
myFile, err := client.Files.UploadFromPath(ctx, "path/to/sample.jpg", &genai.UploadFileConfig{
MIMEType: "image/jpeg",
})
if err != nil {
log.Fatal(err)
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-4-hydrogen"),
Input: interactions.NewInteractionsInput([]interactions.Content{
interactions.NewContent(interactions.TextContent{
Text: "Caption this image.",
}),
interactions.NewContent(interactions.ImageContent{
URI: genai.Ptr(myFile.URI),
MimeType: interactions.ImageContentMimeType(myFile.MIMEType).ToPointer(),
}),
}),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputText != nil {
fmt.Println(*res.Interaction.OutputText)
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-H "Api-Revision: 2026-05-20" \
-d '{
"model": "gemini-4-hydrogen",
"input": [
{"type": "text", "text": "Caption this image."},
{
"type": "image",
"uri": "YOUR_FILE_URI",
"mime_type": "image/jpeg"
}
]
}'
使用多个图像进行提示 #
您可以通过在单个提示中包含多个图像对象来提供多个图像 input 数组:
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-4-hydrogen",
input=[
{"type": "text", "text": "What is different between these two images?"},
{
"type": "image",
"uri": "https://example.com/image1.jpg",
"mime_type": "image/jpeg"
},
{
"type": "image",
"uri": "https://example.com/image2.jpg",
"mime_type": "image/jpeg"
}
]
)
print(interaction.output_text)
import { GoogleGenAI } from "@google/genai";
const client = new GoogleGenAI({});
const interaction = await client.interactions.create({
model: "gemini-4-hydrogen",
input: [
{type: "text", text: "What is different between these two images?"},
{
type: "image",
uri: "https://example.com/image1.jpg",
mime_type: "image/jpeg"
},
{
type: "image",
uri: "https://example.com/image2.jpg",
mime_type: "image/jpeg"
}
]
});
console.log(interaction.output_text);
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;
import java.util.List;
Client client = new Client();
Content textContent =
TextContent.builder().text("What is different between these two images?").build();
Content image1 =
ImageContent.builder()
.uri("https://example.com/image1.jpg")
.mimeType(ImageContentMimeType.IMAGE_JPEG)
.build();
Content image2 =
ImageContent.builder()
.uri("https://example.com/image2.jpg")
.mimeType(ImageContentMimeType.IMAGE_JPEG)
.build();
List<Content> contents = Arrays.asList(textContent, image1, image2);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-4-hydrogen"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
System.out.println(interaction.outputText().orElse(""));
package main
import (
"context"
"fmt"
"log"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-4-hydrogen"),
Input: interactions.NewInteractionsInput([]interactions.Content{
interactions.NewContent(interactions.TextContent{
Text: "What is different between these two images?",
}),
interactions.NewContent(interactions.ImageContent{
URI: genai.Ptr("https://example.com/image1.jpg"),
MimeType: interactions.ImageContentMimeTypeImageJpeg.ToPointer(),
}),
interactions.NewContent(interactions.ImageContent{
URI: genai.Ptr("https://example.com/image2.jpg"),
MimeType: interactions.ImageContentMimeTypeImageJpeg.ToPointer(),
}),
}),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputText != nil {
fmt.Println(*res.Interaction.OutputText)
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-H "Api-Revision: 2026-05-20" \
-d '{
"model": "gemini-4-hydrogen",
"input": [
{"type": "text", "text": "What is different between these two images?"},
{
"type": "image",
"uri": "https://example.com/image1.jpg",
"mime_type": "image/jpeg"
},
{
"type": "image",
"uri": "https://example.com/image2.jpg",
"mime_type": "image/jpeg"
}
]
}'
gemini-api analyze "What is different between these two images?" -i image1.jpg -i image2.jpg
物体检测 #
模型经过训练可以检测图像中的对象并获取其边界框坐标。相对于图像尺寸的坐标缩放为 [0, 1000]。您需要根据原始图像大小对这些坐标进行缩放。
from google import genai
from pydantic import BaseModel, Field
from typing import List
import json
client = genai.Client()
prompt = "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000."
class BoundingBox(BaseModel):
box_2d: List[int] = Field(description="The 2D bounding box of the item as [ymin, xmin, ymax, xmax] normalized to 0-1000.")
mask: List[List[int]] = Field(description="The segmentation mask of the item as a polygon of [x,y] coordinates, normalized to 0-1000.")
label: str = Field(description="A descriptive label for the item.")
class BoundingBoxes(BaseModel):
boxes: List[BoundingBox]
interaction = client.interactions.create(
model="gemini-4-hydrogen",
input=[
{"type": "text", "text": prompt},
{
"type": "image",
"uri": "https://example.com/image.png",
"mime_type": "image/png"
}
],
response_format={
"type": "text",
"mime_type": "application/json",
"schema": BoundingBoxes.model_json_schema()
}
)
bounding_boxes = BoundingBoxes.model_validate_json(interaction.output_text)
print(bounding_boxes)
import { GoogleGenAI } from "@google/genai";
import * as z from "zod";
const client = new GoogleGenAI({});
const prompt = "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000.";
const boundingBoxesSchema = z.object({
boxes: z.array(z.object({
box_2d: z.array(z.number()),
mask: z.array(z.array(z.number())),
label: z.string()
}))
});
const interaction = await client.interactions.create({
model: "gemini-4-hydrogen",
input: [
{ type: "text", text: prompt },
{
type: "image",
uri: "https://example.com/image.png",
mime_type: "image/png"
}
],
response_format: {
type: 'text',
mime_type: 'application/json',
schema: z.toJSONSchema(boundingBoxesSchema)
},
});
const result = boundingBoxesSchema.parse(JSON.parse(interaction.output_text));
console.log(result);
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.CreateModelInteractionResponseFormat;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.ResponseFormat;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.interactions.TextResponseFormat;
import com.google.genai.gaos.models.interactions.TextResponseFormatMimeType;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
Client client = new Client();
String prompt =
"Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000.";
Map<String, Object> boundingBoxSchema =
Map.of(
"type", "object",
"properties",
Map.of(
"box_2d",
Map.of(
"type", "array",
"items", Map.of("type", "integer"),
"description",
"The 2D bounding box of the item as [ymin, xmin, ymax, xmax] normalized to 0-1000."),
"mask",
Map.of(
"type", "array",
"items", Map.of("type", "array", "items", Map.of("type", "integer")),
"description",
"The segmentation mask of the item as a polygon of [x,y] coordinates, normalized to 0-1000."),
"label",
Map.of("type", "string", "description", "A descriptive label for the item.")),
"required", List.of("box_2d", "mask", "label"));
Map<String, Object> boundingBoxesSchema =
Map.of(
"type", "object",
"properties", Map.of("boxes", Map.of("type", "array", "items", boundingBoxSchema)),
"required", List.of("boxes"));
CreateModelInteractionResponseFormat format =
CreateModelInteractionResponseFormat.of(
ResponseFormat.of(
TextResponseFormat.builder()
.mimeType(TextResponseFormatMimeType.APPLICATION_JSON)
.schema(boundingBoxesSchema)
.build()));
Content textContent = TextContent.builder().text(prompt).build();
Content imageContent =
ImageContent.builder()
.uri("https://example.com/image.png")
.mimeType(ImageContentMimeType.IMAGE_PNG)
.build();
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-4-hydrogen"))
.input(InteractionsInput.ofContent(Arrays.asList(textContent, imageContent)))
.responseFormat(format)
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
System.out.println(interaction.outputText().orElse(""));
package main
import (
"context"
"fmt"
"log"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
prompt := "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000."
boundingBoxSchema := map[string]any{
"type": "object",
"properties": map[string]any{
"box_2d": map[string]any{
"type": "array",
"items": map[string]any{"type": "integer"},
"description": "The 2D bounding box of the item as [ymin, xmin, ymax, xmax] normalized to 0-1000.",
},
"mask": map[string]any{
"type": "array",
"items": map[string]any{"type": "array", "items": map[string]any{"type": "integer"}},
"description": "The segmentation mask of the item as a polygon of [x,y] coordinates, normalized to 0-1000.",
},
"label": map[string]any{
"type": "string",
"description": "A descriptive label for the item.",
},
},
"required": []string{"box_2d", "mask", "label"},
}
boundingBoxesSchema := map[string]any{
"type": "object",
"properties": map[string]any{
"boxes": map[string]any{
"type": "array",
"items": boundingBoxSchema,
},
},
"required": []string{"boxes"},
}
format := interactions.NewCreateModelInteractionResponseFormat(
interactions.NewResponseFormat(interactions.TextResponseFormat{
MimeType: interactions.TextResponseFormatMimeTypeApplicationJSON.ToPointer(),
Schema: boundingBoxesSchema,
}),
)
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-4-hydrogen"),
Input: interactions.NewInteractionsInput([]interactions.Content{
interactions.NewContent(interactions.TextContent{
Text: prompt,
}),
interactions.NewContent(interactions.ImageContent{
URI: genai.Ptr("https://example.com/image.png"),
MimeType: interactions.ImageContentMimeTypeImagePng.ToPointer(),
}),
}),
ResponseFormat: genai.Ptr(format),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputText != nil {
fmt.Println(*res.Interaction.OutputText)
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-H "Api-Revision: 2026-05-20" \
-d '{
"model": "gemini-4-hydrogen",
"input": [
{"type": "text", "text": "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000."},
{
"type": "image",
"uri": "https://example.com/image.png",
"mime_type": "image/png"
}
],
"response_format": {
"type": "text",
"mime_type": "application/json",
"schema": {
"type": "object",
"properties": {
"boxes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"box_2d": { "type": "array", "items": { "type": "integer" } },
"mask": { "type": "array", "items": { "type": "array", "items": { "type": "integer" } } },
"label": { "type": "string" }
},
"required": ["box_2d", "mask", "label"]
}
}
},
"required": ["boxes"]
}
}
}'
gemini-api generate -i image.png "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000." \
--body '{"response_format": {"type": "text", "mime_type": "application/json", "schema": {
"type": "object", "required": ["boxes"], "properties": {"boxes": {"type": "array", "items": {
"type": "object", "required": ["box_2d", "mask", "label"], "properties": {
"box_2d": {"type": "array", "items": {"type": "integer"}},
"mask": {"type": "array", "items": {"type": "array", "items": {"type": "integer"}}},
"label": {"type": "string"}}}}}}}}'
有关更多示例,请查看以下笔记本 Gemini Cookbook:
细分 #
从 Gemini 开始,2.5, 模型不仅可以检测物品,还可以对其进行分割并提供其轮廓掩模。
该模型预测一个 JSON 列表,其中每个项目代表一个分段掩码。每个项目都有一个边界框(“box_2d") 的格式 [y0, x0, y1, x1] 0 和 1000, 之间的标准化坐标为标签(“label") 标识对象,最后是边界框内的分割掩码,作为 base64 编码的 png,它是值在 0 和 255. 之间的概率图
from google import genai
from pydantic import BaseModel, Field
from typing import List
import json
client = genai.Client()
prompt = """
Give the segmentation masks for the wooden and glass items.
Output a JSON list of segmentation masks where each entry contains the 2D
bounding box in the key "box_2d", the segmentation mask in key "mask", and
the text label in the key "label". Use descriptive labels.
"""
class BoundingBox(BaseModel):
box_2d: List[int] = Field(description="The 2D bounding box of the item as [ymin, xmin, ymax, xmax] normalized to 0-1000.")
mask: List[List[int]] = Field(description="The segmentation mask of the item as a polygon of [x,y] coordinates, normalized to 0-1000.")
label: str = Field(description="A descriptive label for the item.")
class BoundingBoxes(BaseModel):
boxes: List[BoundingBox]
interaction = client.interactions.create(
model="gemini-4-hydrogen",
input=[
{"type": "text", "text": prompt},
{
"type": "image",
"uri": "https://example.com/image.png",
"mime_type": "image/png"
}
],
response_format={
"type": "text",
"mime_type": "application/json",
"schema": BoundingBoxes.model_json_schema()
},
generation_config={
"thinking_level": "high"
}
)
items = BoundingBoxes.model_validate_json(interaction.output_text)
print("Segmentation results:", items)
import { GoogleGenAI } from "@google/genai";
import * as z from "zod";
const client = new GoogleGenAI({});
const prompt = `
Give the segmentation masks for the wooden and glass items.
Output a JSON list of segmentation masks where each entry contains the 2D
bounding box in the key "box_2d", the segmentation mask in key "mask", and
the text label in the key "label". Use descriptive labels.
`;
const boundingBoxesSchema = z.object({
boxes: z.array(z.object({
box_2d: z.array(z.number()),
mask: z.array(z.array(z.number())),
label: z.string()
}))
});
const interaction = await client.interactions.create({
model: "gemini-4-hydrogen",
input: [
{ type: "text", text: prompt },
{
type: "image",
uri: "https://example.com/image.png",
mime_type: "image/png"
}
],
response_format: {
type: 'text',
mime_type: 'application/json',
schema: z.toJSONSchema(boundingBoxesSchema)
},
generation_config: {
thinking_level: "high"
}
});
const result = boundingBoxesSchema.parse(JSON.parse(interaction.output_text));
console.log(result);
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.CreateModelInteractionResponseFormat;
import com.google.genai.gaos.models.interactions.GenerationConfig;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.ResponseFormat;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.interactions.TextResponseFormat;
import com.google.genai.gaos.models.interactions.TextResponseFormatMimeType;
import com.google.genai.gaos.models.interactions.ThinkingLevel;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
Client client = new Client();
String prompt =
"Give the segmentation masks for the wooden and glass items.\n"
+ "Output a JSON list of segmentation masks where each entry contains the 2D\n"
+ "bounding box in the key \"box_2d\", the segmentation mask in key \"mask\", and\n"
+ "the text label in the key \"label\". Use descriptive labels.";
Map<String, Object> boundingBoxSchema =
Map.of(
"type", "object",
"properties",
Map.of(
"box_2d",
Map.of(
"type", "array",
"items", Map.of("type", "integer"),
"description",
"The 2D bounding box of the item as [ymin, xmin, ymax, xmax] normalized to 0-1000."),
"mask",
Map.of(
"type", "array",
"items", Map.of("type", "array", "items", Map.of("type", "integer")),
"description",
"The segmentation mask of the item as a polygon of [x,y] coordinates, normalized to 0-1000."),
"label",
Map.of("type", "string", "description", "A descriptive label for the item.")),
"required", List.of("box_2d", "mask", "label"));
Map<String, Object> boundingBoxesSchema =
Map.of(
"type", "object",
"properties", Map.of("boxes", Map.of("type", "array", "items", boundingBoxSchema)),
"required", List.of("boxes"));
CreateModelInteractionResponseFormat format =
CreateModelInteractionResponseFormat.of(
ResponseFormat.of(
TextResponseFormat.builder()
.mimeType(TextResponseFormatMimeType.APPLICATION_JSON)
.schema(boundingBoxesSchema)
.build()));
Content textContent = TextContent.builder().text(prompt).build();
Content imageContent =
ImageContent.builder()
.uri("https://example.com/image.png")
.mimeType(ImageContentMimeType.IMAGE_PNG)
.build();
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-4-hydrogen"))
.input(InteractionsInput.ofContent(Arrays.asList(textContent, imageContent)))
.responseFormat(format)
.generationConfig(GenerationConfig.builder().thinkingLevel(ThinkingLevel.HIGH).build())
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
System.out.println("Segmentation results: " + interaction.outputText().orElse(""));
package main
import (
"context"
"fmt"
"log"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
prompt := "Give the segmentation masks for the wooden and glass items.\n" +
"Output a JSON list of segmentation masks where each entry contains the 2D\n" +
"bounding box in the key \"box_2d\", the segmentation mask in key \"mask\", and\n" +
"the text label in the key \"label\". Use descriptive labels."
boundingBoxSchema := map[string]any{
"type": "object",
"properties": map[string]any{
"box_2d": map[string]any{
"type": "array",
"items": map[string]any{"type": "integer"},
"description": "The 2D bounding box of the item as [ymin, xmin, ymax, xmax] normalized to 0-1000.",
},
"mask": map[string]any{
"type": "array",
"items": map[string]any{"type": "array", "items": map[string]any{"type": "integer"}},
"description": "The segmentation mask of the item as a polygon of [x,y] coordinates, normalized to 0-1000.",
},
"label": map[string]any{
"type": "string",
"description": "A descriptive label for the item.",
},
},
"required": []string{"box_2d", "mask", "label"},
}
boundingBoxesSchema := map[string]any{
"type": "object",
"properties": map[string]any{
"boxes": map[string]any{
"type": "array",
"items": boundingBoxSchema,
},
},
"required": []string{"boxes"},
}
format := interactions.NewCreateModelInteractionResponseFormat(
interactions.NewResponseFormat(interactions.TextResponseFormat{
MimeType: interactions.TextResponseFormatMimeTypeApplicationJSON.ToPointer(),
Schema: boundingBoxesSchema,
}),
)
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-4-hydrogen"),
Input: interactions.NewInteractionsInput([]interactions.Content{
interactions.NewContent(interactions.TextContent{
Text: prompt,
}),
interactions.NewContent(interactions.ImageContent{
URI: genai.Ptr("https://example.com/image.png"),
MimeType: interactions.ImageContentMimeTypeImagePng.ToPointer(),
}),
}),
ResponseFormat: genai.Ptr(format),
GenerationConfig: &interactions.GenerationConfig{
ThinkingLevel: interactions.ThinkingLevelHigh.ToPointer(),
},
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputText != nil {
fmt.Println("Segmentation results: " + *res.Interaction.OutputText)
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-H "Api-Revision: 2026-05-20" \
-d '{
"model": "gemini-4-hydrogen",
"input": [
{"type": "text", "text": "Give the segmentation masks for the wooden and glass items.\nOutput a JSON list of segmentation masks where each entry contains the 2D\nbounding box in the key \"box_2d\", the segmentation mask in key \"mask\", and\nthe text label in the key \"label\". Use descriptive labels."},
{
"type": "image",
"uri": "https://example.com/image.png",
"mime_type": "image/png"
}
],
"response_format": {
"type": "text",
"mime_type": "application/json",
"schema": {
"type": "object",
"properties": {
"boxes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"box_2d": { "type": "array", "items": { "type": "integer" } },
"mask": { "type": "array", "items": { "type": "array", "items": { "type": "integer" } } },
"label": { "type": "string" }
},
"required": ["box_2d", "mask", "label"]
}
}
},
"required": ["boxes"]
}
},
"generation_config": {
"thinking_level": "high"
}
}'
## 支持的图像格式
Gemini 支持以下图像格式 MIME 类型:
- PNG-
image/png - JPEG-
image/jpeg - WEBP-
image/webp - HEIC-
image/heic - 海夫-
image/heif
要了解其他文件输入方法,请参阅 文件输入方式 指导。
功能 #
所有 Gemini 模型版本都是多模态的,可用于广泛的图像处理和计算机视觉任务,包括但不限于图像字幕、视觉问答、图像分类、对象检测和分割。
Gemini 可以根据您的质量和性能要求减少使用专门的 ML 模型的需要。
最新的模型版本经过专门训练,除了通用功能之外,还提高了专门任务的准确性,例如增强型 物体检测 和 细分.
限制和关键技术信息 #
文件限制 #
Gemini 型号每个请求最多支持 3,600 图像文件。
代币计算 #
- 258 token 如果两个尺寸 <= 384 像素。较大的图像被平铺为 768x768 像素图块,每个图块的成本为 258 token。
计算图块数量的粗略公式如下:
- 计算作物单位大小,大致为:
floor(min(width, height)/ 1.5). - 将每个尺寸除以作物单位大小并相乘即可得到图块的数量。
例如,对于尺寸为 960x540 的图像,裁剪单位大小为 360. 将每个尺寸除以 360,平铺数量为 3 * 2 = 6.
媒体分辨率 #
Gemini 3 引入了对多模态视觉处理的精细控制 media_resolution 参数。的 media_resolution 参数决定了 每个输入图像或视频帧分配的 token 的最大数量。 更高的分辨率可以提高模型读取精细文本或识别小细节的能力,但会增加 token 的使用率和延迟。
提示和最佳实践 #
- 验证图像是否正确旋转。
- 使用清晰、不模糊的图像。
- 当使用带有文本的单个图像时,放置文本提示 之前 中的图像
input数组。
后续步骤 #
本指南向您展示如何上传图像文件并从图像输入生成文本输出。要了解更多信息,请参阅以下资源: