使用 Gemini Omni Flash 生成和编辑视频
Gemini Omni 1.2 Flash (gemini-omni-1.2-flash)是一个多模态模型,专为快速视频生成、对话式编辑和电影控制而构建。主要能力包括:
- 原生多模态: Omni 处理文本、图像和视频输入,生成长达 15 秒的同步视频和本机音频。
- 对话编辑和扩展: 通过 Interactions API,您可以使用自然语言在多个回合中精炼、编辑和扩展视频。描述您想要更改的内容或场景应如何继续,Omni 会更新视频,同时保留场景的其余部分。
- 性格一致性和世界知识: Omni 将对物理和世界知识的理解与视频参考一致性相结合,使用最多 6 参考图像来保持角色和物体在镜头中的一致性。
文本到视频生成 #
根据文本提示生成视频。该模型根据您的文本描述生成带音频的视频。编写包含场景描述、摄像机移动、灯光和情绪等详细信息的提示,以获得最佳效果。
import base64
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input="A marble rolling fast on a chain reaction style track, continuous smooth shot."
)
with open("marble.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: 'A marble rolling fast on a chain reaction style track, continuous smooth shot.',
});
if (interaction.output_video?.data) {
fs.writeFileSync('marble.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Base64;
Client client = new Client();
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(
InteractionsInput.of(
"A marble rolling fast on a chain reaction style track, continuous smooth shot."))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("marble.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(
"A marble rolling fast on a chain reaction style track, continuous smooth shot.",
),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("marble.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": "A marble rolling fast on a chain reaction style track, continuous smooth shot."
}'
gemini-api video "A marble rolling fast on a chain reaction style track, continuous smooth shot." \
-m gemini-omni-1.2-flash \
--out marble.mp4
REST 响应架构 #
便利领域 interaction.output_video 是 SDK-仅限。获取视频输出 steps 直接使用 REST API 时的数组。
原始 REST JSON 结构:
{
"steps": [
{ "type": "user_input", "content": [{"type": "text", "text": "..."}] },
{ "type": "thought", "content": [{"text": "...", "type": "thought"}] },
{
"type": "model_output",
"content": [
{
"type": "video",
"mime_type": "video/mp4",
"data": "AAAAIGZ0eXBpc29t..." // Base64 encoded video data
}
]
}
],
"id": "v1_...",
"status": "completed",
"model": "gemini-omni-1.2-flash",
"object": "interaction"
}
控制纵横比 #
设置 aspect_ratio 到 "9:16" 创建肖像视频。横向 (16:9) 是默认设置。
import base64
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input="A futuristic city with neon lights and flying cars, cyberpunk style",
response_format={
"type": "video", # optional
"aspect_ratio": "9:16" # Supported values: "9:16", "16:9"
}
)
with open("example.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: 'A futuristic city with neon lights and flying cars, cyberpunk style',
response_format: {
type: 'video', // optional
aspect_ratio: '9:16' // Supported values: '9:16', '16:9'
},
});
if (interaction.output_video?.data) {
fs.writeFileSync('example.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.CreateModelInteractionResponseFormat;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.ResponseFormat;
import com.google.genai.gaos.models.interactions.VideoResponseFormat;
import com.google.genai.gaos.models.interactions.VideoResponseFormatAspectRatio;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Base64;
Client client = new Client();
VideoResponseFormat videoFormat =
VideoResponseFormat.builder()
.aspectRatio(VideoResponseFormatAspectRatio.of("9:16"))
.build();
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(
InteractionsInput.of(
"A futuristic city with neon lights and flying cars, cyberpunk style"))
.responseFormat(CreateModelInteractionResponseFormat.of(ResponseFormat.of(videoFormat)))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("example.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
videoFormat := interactions.VideoResponseFormat{
AspectRatio: interactions.VideoResponseFormatAspectRatioNineHundredAndSixteen.ToPointer(),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(
"A futuristic city with neon lights and flying cars, cyberpunk style",
),
ResponseFormat: genai.Ptr(interactions.NewCreateModelInteractionResponseFormat(
interactions.NewResponseFormat(videoFormat),
)),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("example.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": "A futuristic city with neon lights and flying cars, cyberpunk style",
"response_format": {
"type": "video",
"aspect_ratio": "9:16"
}
}'
gemini-api video "A futuristic city with neon lights and flying cars, cyberpunk style" \
-m gemini-omni-1.2-flash \
--aspect-ratio 9:16 \
--out example.mp4
输出分辨率 #
使用以下命令控制生成的视频的输出分辨率 resolution 参数输入 response_format。默认分辨率为 720p。
| 价值 | 说明 |
|---|---|
360p | 360p 输出分辨率 |
720p | 720p 输出分辨率(默认) |
1080p | 1080p 输出(升级) |
4k | 4K 输出(升级) |
import base64
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input="A drone shot of a mountain landscape at sunrise.",
response_format={
"type": "video",
"resolution": "1080p",
},
)
with open("hires.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: 'A drone shot of a mountain landscape at sunrise.',
response_format: {
type: 'video',
resolution: '1080p',
},
});
if (interaction.output_video?.data) {
fs.writeFileSync('hires.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.CreateModelInteractionResponseFormat;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.Resolution;
import com.google.genai.gaos.models.interactions.ResponseFormat;
import com.google.genai.gaos.models.interactions.VideoResponseFormat;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Base64;
Client client = new Client();
VideoResponseFormat videoFormat =
VideoResponseFormat.builder()
.resolution(Resolution.of("1080p"))
.build();
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.of("A drone shot of a mountain landscape at sunrise."))
.responseFormat(CreateModelInteractionResponseFormat.of(ResponseFormat.of(videoFormat)))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("hires.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
videoFormat := interactions.VideoResponseFormat{
Resolution: interactions.ResolutionOneThousandAndEightyp.ToPointer(),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput("A drone shot of a mountain landscape at sunrise."),
ResponseFormat: genai.Ptr(interactions.NewCreateModelInteractionResponseFormat(
interactions.NewResponseFormat(videoFormat),
)),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("hires.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": "A drone shot of a mountain landscape at sunrise.",
"response_format": {
"type": "video",
"resolution": "1080p"
}
}'
gemini-api video "A drone shot of a mountain landscape at sunrise." \
-m gemini-omni-1.2-flash \
--resolution 1080p \
--out hires.mp4
控制视频时长 #
您可以设置视频 duration 在 response_format 从 3 到 15 秒的任何整秒(例如, "15s")。当您不设置持续时间时,模型默认为 10 秒。如果您设置持续时间 response_format,模型使用该值而不是文本提示中提到的任何长度。设置超过 15 秒的持续时间将返回无效参数错误。
import base64
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input="A single unbroken tracking shot following a paper airplane gliding through a bustling library.",
response_format={
"type": "video",
"duration": "15s", # Supported values: "3s" to "15s" (default: "10s")
},
)
with open("library_flight.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: 'A single unbroken tracking shot following a paper airplane gliding through a bustling library.',
response_format: {
type: 'video',
duration: '15s', // Supported values: '3s' to '15s' (default: '10s')
},
});
if (interaction.output_video?.data) {
fs.writeFileSync('library_flight.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.CreateModelInteractionResponseFormat;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.ResponseFormat;
import com.google.genai.gaos.models.interactions.VideoResponseFormat;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Base64;
Client client = new Client();
VideoResponseFormat videoFormat =
VideoResponseFormat.builder()
.duration("15s")
.build();
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(
InteractionsInput.of(
"A single unbroken tracking shot following a paper airplane gliding through a bustling library."))
.responseFormat(CreateModelInteractionResponseFormat.of(ResponseFormat.of(videoFormat)))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("library_flight.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
videoFormat := interactions.VideoResponseFormat{
Duration: genai.Ptr("15s"),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(
"A single unbroken tracking shot following a paper airplane gliding through a bustling library.",
),
ResponseFormat: genai.Ptr(interactions.NewCreateModelInteractionResponseFormat(
interactions.NewResponseFormat(videoFormat),
)),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("library_flight.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": "A single unbroken tracking shot following a paper airplane gliding through a bustling library.",
"response_format": {
"type": "video",
"duration": "15s"
}
}'
gemini-api video "A single unbroken tracking shot following a paper airplane gliding through a bustling library." \
-m gemini-omni-1.2-flash \
--body '{"response_format":{"duration":"15s"}}' \
--out library_flight.mp4
图像到视频生成 #
您可以在文本提示中提供参考图像。根据您的提示,模型将决定如何使用图像。这对于使产品镜头、插图或照片栩栩如生非常有用。
以下示例展示了如何使用鱼跃出水面的参考图像:

出现以下提示:
turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video
生成逼真的绘图视频。
import base64
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input=[
{"type": "image", "data": base64_image, "mime_type": "image/jpeg"},
{"type": "text", "text": "turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video"}
],
)
with open("clownfish.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: [
{ type: 'image', data: base64Image, mime_type: 'image/jpeg' },
{ type: 'text', text: 'turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video' }
]
});
if (interaction.output_video?.data) {
fs.writeFileSync('clownfish.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
Client client = new Client();
byte[] imageBytes = Files.readAllBytes(Paths.get("drawing.jpg"));
String base64Image = Base64.getEncoder().encodeToString(imageBytes);
Content imageContent =
ImageContent.builder()
.data(base64Image)
.mimeType(ImageContentMimeType.IMAGE_JPEG)
.build();
Content textContent =
TextContent.builder()
.text(
"turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video")
.build();
List<Content> contents = Arrays.asList(imageContent, textContent);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("clownfish.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
imageBytes, err := os.ReadFile("drawing.jpg")
if err != nil {
log.Fatal(err)
}
base64Image := base64.StdEncoding.EncodeToString(imageBytes)
contents := []interactions.Content{
interactions.NewContent(interactions.ImageContent{
Data: genai.Ptr(base64Image),
MimeType: interactions.ImageContentMimeTypeImageJpeg.ToPointer(),
}),
interactions.NewContent(interactions.TextContent{
Text: "turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video",
}),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(contents),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("clownfish.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": [
{"type": "image", "data": "'"$BASE64_IMAGE"'", "mime_type": "image/jpeg"},
{"type": "text", "text": "turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video"}
]
}'
gemini-api video "turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video" \
-m gemini-omni-1.2-flash \
-i drawing.jpg \
--out clownfish.mp4
第一帧和最后一帧插值 #
Gemini Omni Flash 支持视频插值,允许您生成在起始图像(第一帧)和结束图像(最后一帧)之间平滑过渡的视频。
提供两张图片 input 在提示中列出并描述所需的转换。模型将从第一帧到结束帧对场景进行动画处理。
import base64
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input=[
{"type": "image", "data": first_frame_b64, "mime_type": "image/jpeg"},
{"type": "image", "data": last_frame_b64, "mime_type": "image/jpeg"},
{"type": "text", "text": "A smooth cinematic transition from a lush green forest at sunrise to a snowy forest under a starry night sky."}
],
)
with open("interpolation.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
const firstFrameB64 = '...';
const lastFrameB64 = '...';
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: [
{ type: 'image', data: firstFrameB64, mime_type: 'image/jpeg' },
{ type: 'image', data: lastFrameB64, mime_type: 'image/jpeg' },
{ type: 'text', text: 'A smooth cinematic transition from a lush green forest at sunrise to a snowy forest under a starry night sky.' }
]
});
if (interaction.output_video?.data) {
fs.writeFileSync('interpolation.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
Client client = new Client();
String firstFrameB64 =
Base64.getEncoder().encodeToString(Files.readAllBytes(Paths.get("first_frame.jpg")));
String lastFrameB64 =
Base64.getEncoder().encodeToString(Files.readAllBytes(Paths.get("last_frame.jpg")));
Content firstFrame =
ImageContent.builder()
.data(firstFrameB64)
.mimeType(ImageContentMimeType.IMAGE_JPEG)
.build();
Content lastFrame =
ImageContent.builder()
.data(lastFrameB64)
.mimeType(ImageContentMimeType.IMAGE_JPEG)
.build();
Content prompt =
TextContent.builder()
.text(
"A smooth cinematic transition from a lush green forest at sunrise to a snowy forest under a starry night sky.")
.build();
List<Content> contents = Arrays.asList(firstFrame, lastFrame, prompt);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("interpolation.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
firstBytes, err := os.ReadFile("first_frame.jpg")
if err != nil {
log.Fatal(err)
}
lastBytes, err := os.ReadFile("last_frame.jpg")
if err != nil {
log.Fatal(err)
}
firstFrameB64 := base64.StdEncoding.EncodeToString(firstBytes)
lastFrameB64 := base64.StdEncoding.EncodeToString(lastBytes)
contents := []interactions.Content{
interactions.NewContent(interactions.ImageContent{
Data: genai.Ptr(firstFrameB64),
MimeType: interactions.ImageContentMimeTypeImageJpeg.ToPointer(),
}),
interactions.NewContent(interactions.ImageContent{
Data: genai.Ptr(lastFrameB64),
MimeType: interactions.ImageContentMimeTypeImageJpeg.ToPointer(),
}),
interactions.NewContent(interactions.TextContent{
Text: "A smooth cinematic transition from a lush green forest at sunrise to a snowy forest under a starry night sky.",
}),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(contents),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("interpolation.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": [
{"type": "image", "data": "'"$FIRST_FRAME_B64"'", "mime_type": "image/jpeg"},
{"type": "image", "data": "'"$LAST_FRAME_B64"'", "mime_type": "image/jpeg"},
{"type": "text", "text": "A smooth cinematic transition from a lush green forest at sunrise to a snowy forest under a starry night sky."}
]
}'
gemini-api video "A smooth cinematic transition from a lush green forest at sunrise to a snowy forest under a starry night sky." \
-m gemini-omni-1.2-flash \
-i first_frame.jpg \
-i last_frame.jpg \
--out interpolation.mp4
主题参考 #
您可以生成包含作为参考图像提供的特定主题的视频。 Gemini Omni 1.2 Flash 根据请求最多支持 6 参考图像(每个角色最多 3 图像,最多 2 角色),以保持角色相似性、服装和物体在镜头中保持一致。例如,以下代码提供猫和纱线的 2 图像,以生成猫玩纱线的视频。
import base64
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input=[
{"type": "image", "data": cat_b64, "mime_type": "image/png"},
{"type": "image", "data": yarn_b64, "mime_type": "image/png"},
{"type": "text", "text": "A cat playfully batting at a ball of yarn."}
],
)
with open("cat.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: [
{ type: 'image', data: catData, mime_type: 'image/png' },
{ type: 'image', data: yarnData, mime_type: 'image/png' },
{ type: 'text', text: 'A cat playfully batting at a ball of yarn.' }
]
});
if (interaction.output_video?.data) {
fs.writeFileSync('cat.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
Client client = new Client();
String catB64 = Base64.getEncoder().encodeToString(Files.readAllBytes(Paths.get("cat.png")));
String yarnB64 = Base64.getEncoder().encodeToString(Files.readAllBytes(Paths.get("yarn.png")));
Content catImage =
ImageContent.builder()
.data(catB64)
.mimeType(ImageContentMimeType.IMAGE_PNG)
.build();
Content yarnImage =
ImageContent.builder()
.data(yarnB64)
.mimeType(ImageContentMimeType.IMAGE_PNG)
.build();
Content textContent =
TextContent.builder()
.text("A cat playfully batting at a ball of yarn.")
.build();
List<Content> contents = Arrays.asList(catImage, yarnImage, textContent);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("cat.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
catBytes, err := os.ReadFile("cat.png")
if err != nil {
log.Fatal(err)
}
yarnBytes, err := os.ReadFile("yarn.png")
if err != nil {
log.Fatal(err)
}
catB64 := base64.StdEncoding.EncodeToString(catBytes)
yarnB64 := base64.StdEncoding.EncodeToString(yarnBytes)
contents := []interactions.Content{
interactions.NewContent(interactions.ImageContent{
Data: genai.Ptr(catB64),
MimeType: interactions.ImageContentMimeTypeImagePng.ToPointer(),
}),
interactions.NewContent(interactions.ImageContent{
Data: genai.Ptr(yarnB64),
MimeType: interactions.ImageContentMimeTypeImagePng.ToPointer(),
}),
interactions.NewContent(interactions.TextContent{
Text: "A cat playfully batting at a ball of yarn.",
}),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(contents),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("cat.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": [
{"type": "image", "data": "'"$CAT_B64"'", "mime_type": "image/png"},
{"type": "image", "data": "'"$YARN_B64"'", "mime_type": "image/png"},
{"type": "text", "text": "A cat playfully batting at a ball of yarn."}
]
}'
gemini-api video "A cat playfully batting at a ball of yarn." \
-m gemini-omni-1.2-flash \
-i cat.png \
-i yarn.png \
--out cat.mp4
任务参数 #
默认情况下,模型直接从自然语言提示推断生成模式和媒体角色。如果仅提示未选择预期模式,请设置 task 参数输入 video_config 作为后备(例如, image_to_video).
以下是允许的值:
text_to_videoimage_to_videoreference_to_videoeditextend
以下示例展示了如何为之前显示的图像到视频示例进行设置。
import base64
from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input=[
{"type": "image", "data": base64_image, "mime_type": "image/jpeg"},
{"type": "text", "text": "turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video"}
],
generation_config={
"video_config": {
"task": "image_to_video",
}
},
)
with open("example.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from "@google/genai";
import * as fs from 'fs';
const ai = new GoogleGenAI({});
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: [
{ type: 'image', data: base64Image, mime_type: 'image/jpeg' },
{ type: 'text', text: 'turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video' }
],
generation_config: {
video_config: {
task: 'image_to_video',
}
}
});
if (interaction.output_video?.data) {
fs.writeFileSync('example.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.GenerationConfig;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.Task;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.interactions.VideoConfig;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
Client client = new Client();
byte[] imageBytes = Files.readAllBytes(Paths.get("drawing.jpg"));
String base64Image = Base64.getEncoder().encodeToString(imageBytes);
Content imageContent =
ImageContent.builder()
.data(base64Image)
.mimeType(ImageContentMimeType.IMAGE_JPEG)
.build();
Content textContent =
TextContent.builder()
.text(
"turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video")
.build();
List<Content> contents = Arrays.asList(imageContent, textContent);
GenerationConfig generationConfig =
GenerationConfig.builder()
.videoConfig(VideoConfig.builder().task(Task.IMAGE_TO_VIDEO).build())
.build();
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.ofContent(contents))
.generationConfig(generationConfig)
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("example.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
imageBytes, err := os.ReadFile("drawing.jpg")
if err != nil {
log.Fatal(err)
}
base64Image := base64.StdEncoding.EncodeToString(imageBytes)
contents := []interactions.Content{
interactions.NewContent(interactions.ImageContent{
Data: genai.Ptr(base64Image),
MimeType: interactions.ImageContentMimeTypeImageJpeg.ToPointer(),
}),
interactions.NewContent(interactions.TextContent{
Text: "turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video",
}),
}
generationConfig := &interactions.GenerationConfig{
VideoConfig: &interactions.VideoConfig{
Task: interactions.TaskImageToVideo.ToPointer(),
},
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(contents),
GenerationConfig: generationConfig,
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("example.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": [
{
"type": "image",
"data": "'"$BASE64_IMAGE"'",
"mime_type": "image/jpeg"
},
{
"type": "text",
"text": "turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video"
}
],
"generation_config": {
"video_config": {
"task": "image_to_video"
}
}
}'
gemini-api video "turn this into realistic footage, using the drawing only as a guide for movement, do not show the drawing in the final video" \
-m gemini-omni-1.2-flash \
-i drawing.jpg \
--body '{"generation_config":{"video_config":{"task":"image_to_video"}}}' \
--out example.mp4
有状态视频编辑 #
生成视频并使用后续提示对其进行迭代编辑。每个回合都建立在前一个结果的基础上。该模型会记住视频上下文,应用您的更改,同时保留您未提及的元素。使用 previous_interaction_id 跟踪对话历史记录并编辑生成的视频长达 15 秒,而无需重新上传以前的视频。
以下示例演示了如何生成第一个视频然后对其进行编辑:
import base64
from google import genai
client = genai.Client()
# Turn 1: Generate initial video
res1 = client.interactions.create(model="gemini-omni-1.2-flash", input="A woman playing violin outdoors.")
# Turn 2: Edit the previous video
res2 = client.interactions.create(
model="gemini-omni-1.2-flash",
previous_interaction_id=res1.id,
input="Make the violin invisible."
)
with open("example.mp4", "wb") as f:
f.write(base64.b64decode(res2.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
// Turn 1: Generate initial video
const res1 = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: 'A woman playing violin outdoors.',
});
// Turn 2: Edit the previous video
const res2 = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
previous_interaction_id: res1.id,
input: 'Make the violin invisible.',
});
if (res2.output_video?.data) {
fs.writeFileSync('example.mp4', Buffer.from(res2.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Base64;
Client client = new Client();
// Turn 1: Generate initial video
CreateModelInteraction turn1Params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.of("A woman playing violin outdoors."))
.build();
Interaction res1 =
client.interactions.create(CreateInteractionRequestBody.of(turn1Params)).interaction().get();
// Turn 2: Edit the previous video
CreateModelInteraction turn2Params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.previousInteractionId(res1.id().get())
.input(InteractionsInput.of("Make the violin invisible."))
.build();
Interaction res2 =
client.interactions.create(CreateInteractionRequestBody.of(turn2Params)).interaction().get();
if (res2.outputVideo().isPresent() && res2.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(res2.outputVideo().get().data().get());
Files.write(Paths.get("example.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
// Turn 1: Generate initial video
res1, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput("A woman playing violin outdoors."),
}),
})
if err != nil {
log.Fatal(err)
}
// Turn 2: Edit the previous video
res2, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
PreviousInteractionID: res1.Interaction.ID,
Input: interactions.NewInteractionsInput("Make the violin invisible."),
}),
})
if err != nil {
log.Fatal(err)
}
if res2.Interaction.OutputVideo != nil && res2.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res2.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("example.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"previous_interaction_id": "'"$PREVIOUS_ID"'",
"input": "Make the violin invisible."
}'
ID=$(gemini-api video "A woman playing violin outdoors." -m gemini-omni-1.2-flash --out violin.mp4 --jq '.response_id')
gemini-api video "Make the violin invisible." -m gemini-omni-1.2-flash --previous-interaction-id "$ID" --out example.mp4
初始视频示例:
▶ 观看视频(参考) ↗编辑视频的示例:
▶ 观看视频(参考) ↗对话的每一轮都会产生一个新视频。该模型可以理解先前回合的上下文,让您可以进行增量更改,例如调整照明和交换背景,而无需重新描述整个场景。
编辑您自己的视频 #
使用以下命令上传您的视频(最长 15 秒) Files API 使用 Gemini Omni Flash 编辑它们。
以下示例展示了如何编辑以下原始视频:
▶ 观看视频(参考) ↗import time
import base64
from google import genai
client = genai.Client()
# Upload video using the file API
video_file = client.files.upload(file="Video.mp4")
while video_file.state == "PROCESSING":
print('Waiting for video to be processed.')
time.sleep(10)
video_file = client.files.get(name=video_file.name)
if video_file.state == "FAILED":
raise ValueError(video_file.state)
print(f'Video processing complete: ' + video_file.uri)
# Edit your video
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input=[
{"type": "video", "uri": video_file.uri},
{"type": "text", "text": "When the person touches the mirror, make the mirror ripple beautifully like liquid, and the person's arm turns into reflective mirror material"}
],
)
with open("example.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
// Upload video using the file API
let videoFile = await ai.files.upload({
file: 'Video.mp4',
});
while (videoFile.state === 'PROCESSING') {
console.log('Waiting for video to be processed.');
await new Promise(r => setTimeout(r, 10000));
videoFile = await ai.files.get({ name: videoFile.name || '' });
}
if (videoFile.state === 'FAILED') {
throw new Error(videoFile.state);
}
console.log('Video processing complete: ' + videoFile.uri);
// Edit your video
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: [
{ type: 'video', uri: videoFile.uri },
{ type: 'text', text: "When the person touches the mirror, make the mirror ripple beautifully like liquid, and the person's arm turns into reflective mirror material" }
],
});
if (interaction.output_video?.data) {
fs.writeFileSync('example.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.interactions.VideoContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import com.google.genai.types.File;
import com.google.genai.types.FileState;
import com.google.genai.types.UploadFileConfig;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
Client client = new Client();
// Upload video using the file API
File videoFile =
client.files.upload("Video.mp4", UploadFileConfig.builder().mimeType("video/mp4").build());
while (videoFile.state().isPresent()
&& videoFile.state().get().knownEnum() == FileState.Known.PROCESSING) {
System.out.println("Waiting for video to be processed.");
Thread.sleep(10000);
videoFile = client.files.get(videoFile.name().get(), null);
}
if (videoFile.state().isPresent()
&& videoFile.state().get().knownEnum() == FileState.Known.FAILED) {
throw new IllegalStateException("Video processing failed: " + videoFile.state().get());
}
System.out.println("Video processing complete: " + videoFile.uri().orElse(""));
// Edit your video
Content videoContent = VideoContent.builder().uri(videoFile.uri().get()).build();
Content textContent =
TextContent.builder()
.text(
"When the person touches the mirror, make the mirror ripple beautifully like liquid, and the person's arm turns into reflective mirror material")
.build();
List<Content> contents = Arrays.asList(videoContent, textContent);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("example.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"fmt"
"log"
"os"
"time"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
// Upload video using the file API
videoFile, err := client.Files.UploadFromPath(ctx, "Video.mp4", &genai.UploadFileConfig{
MIMEType: "video/mp4",
})
if err != nil {
log.Fatal(err)
}
for videoFile.State == genai.FileStateProcessing {
fmt.Println("Waiting for video to be processed.")
time.Sleep(10 * time.Second)
videoFile, err = client.Files.Get(ctx, videoFile.Name, nil)
if err != nil {
log.Fatal(err)
}
}
if videoFile.State == genai.FileStateFailed {
log.Fatalf("Video processing failed: %s", videoFile.State)
}
fmt.Printf("Video processing complete: %s\n", videoFile.URI)
// Edit your video
contents := []interactions.Content{
interactions.NewContent(interactions.VideoContent{
URI: genai.Ptr(videoFile.URI),
}),
interactions.NewContent(interactions.TextContent{
Text: "When the person touches the mirror, make the mirror ripple beautifully like liquid, and the person's arm turns into reflective mirror material",
}),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(contents),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("example.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
#!/bin/bash
VIDEO_B64=$(encode_file "$VIDEO_FILE")
curl -sS -w "\n[HTTP %{http_code}]\n" "https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: ${API_KEY}" \
-H "Content-Type: application/json" \
-d @- <<EOF > video_editing_response.json
{
"model": "gemini-omni-1.2-flash",
"input": [
{
"type": "user_input",
"content": [
{
"type": "video",
"mime_type": "video/mp4",
"data": "$VIDEO_B64"
},
{
"type": "text",
"text": "When the person touches the mirror, make the mirror ripple beautifully like liquid, and the person's arm turns into reflective mirror material"
}
]
}
],
"response_format": { "type": "video" }
}
EOF
VIDEO=$(gemini-api files upload Video.mp4 --wait)
gemini-api video "When the person touches the mirror, make the mirror ripple beautifully like liquid, and the person's arm turns into reflective mirror material" \
-m gemini-omni-1.2-flash \
-i "$VIDEO" \
--out example.mp4
编辑视频的示例:
▶ 观看视频(参考) ↗使用 URI 检索视频 #
使用 delivery="uri" 参数输入 response_format 检索生成的大于 4MB 的视频。这会返回一个 Google 托管的 URI,您可以轮询该 URI,直到视频被 ACTIVE 下载之前。
import time
from google import genai
client = genai.Client()
# 1. Request video via URI delivery
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input="A beautiful sunset.",
response_format={"type": "video", "delivery": "uri"}
)
# 2. Extract file name and poll for ACTIVE state
video_output = interaction.output_video
file_name = video_output.uri.split("/")[-1] # Extract ID
print("Waiting for video processing...")
while True:
f_info = client.files.get(name=f"files/{file_name}")
if f_info.state.name == "ACTIVE":
break
elif f_info.state.name == "FAILED":
raise RuntimeError("Generation failed.")
time.sleep(5)
# 3. Download the final video
client.files.download(file=video_output.uri, destination="output.mp4")
import { GoogleGenAI } from '@google/genai';
const ai = new GoogleGenAI({});
// 1. Request video using URI delivery
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: 'A beautiful sunset.',
response_format: { type: 'video', delivery: 'uri' },
});
// 2. Extract filename and poll for ACTIVE state
const videoOutput = interaction.output_video;
const fileId = videoOutput?.uri?.match(/files\/([a-zA-Z0-9]+)/)?.[1] || '';
const name = `files/${fileId}`;
console.log("Waiting for video processing...");
while (true) {
const fInfo = await ai.files.get({ name });
if (fInfo.state === 'ACTIVE') break;
if (fInfo.state === 'FAILED') throw new Error("Generation failed.");
await new Promise(r => setTimeout(r, 5000));
}
// 3. Download the final video
await ai.files.download({
file: name,
downloadPath: 'output.mp4',
});
console.log("💾 Saved video to output.mp4");
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.CreateModelInteractionResponseFormat;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.ResponseFormat;
import com.google.genai.gaos.models.interactions.VideoContent;
import com.google.genai.gaos.models.interactions.VideoResponseFormat;
import com.google.genai.gaos.models.interactions.VideoResponseFormatDelivery;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import com.google.genai.types.File;
import com.google.genai.types.FileState;
Client client = new Client();
// 1. Request video using URI delivery
VideoResponseFormat videoFormat =
VideoResponseFormat.builder()
.delivery(VideoResponseFormatDelivery.URI)
.build();
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.of("A beautiful sunset."))
.responseFormat(CreateModelInteractionResponseFormat.of(ResponseFormat.of(videoFormat)))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
// 2. Extract filename and poll for ACTIVE state
VideoContent videoOutput = interaction.outputVideo().get();
String uri = videoOutput.uri().get();
String[] parts = uri.split("/");
String fileName = parts[parts.length - 1];
System.out.println("Waiting for video processing...");
while (true) {
File fileInfo = client.files.get("files/" + fileName, null);
if (fileInfo.state().isPresent()
&& fileInfo.state().get().knownEnum() == FileState.Known.ACTIVE) {
break;
} else if (fileInfo.state().isPresent()
&& fileInfo.state().get().knownEnum() == FileState.Known.FAILED) {
throw new RuntimeException("Generation failed.");
}
Thread.sleep(5000);
}
// 3. Download the final video
client.files.download(uri, "output.mp4", null);
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"time"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
// 1. Request video using URI delivery
videoFormat := interactions.VideoResponseFormat{
Delivery: interactions.VideoResponseFormatDeliveryURI.ToPointer(),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput("A beautiful sunset."),
ResponseFormat: genai.Ptr(interactions.NewCreateModelInteractionResponseFormat(
interactions.NewResponseFormat(videoFormat),
)),
}),
})
if err != nil {
log.Fatal(err)
}
// 2. Extract filename and poll for ACTIVE state
uri := *res.Interaction.OutputVideo.URI
parts := strings.Split(uri, "/")
fileName := parts[len(parts)-1]
fmt.Println("Waiting for video processing...")
var fileInfo *genai.File
for {
fileInfo, err = client.Files.Get(ctx, "files/"+fileName, nil)
if err != nil {
log.Fatal(err)
}
if fileInfo.State == genai.FileStateActive {
break
} else if fileInfo.State == genai.FileStateFailed {
log.Fatal("Generation failed.")
}
time.Sleep(5 * time.Second)
}
// 3. Download the final video
videoBytes, err := client.Files.Download(ctx, genai.NewDownloadURIFromFile(fileInfo), nil)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("output.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
#!/bin/bash
# 1. Initial request to generate the video
RESPONSE=$(curl -s -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-omni-1.2-flash",
"input": "A beautiful sunset over a calm ocean.",
"response_format": {"type": "video", "delivery": "uri"}
}')
# Extract FILE_ID from the URI (e.g., "files/abc-123" -> "abc-123")
FILE_URI=$(echo $RESPONSE | jq -r '.output_video.uri')
FILE_ID=$(echo $FILE_URI | cut -d'/' -f2)
echo "Video requested (ID: $FILE_ID). Waiting for processing..."
# 2. Polling loop
while true; do
# Get current file status
STATUS_JSON=$(curl -s -X GET "https://generativelanguage.googleapis.com/v1beta/files/$FILE_ID?key=$API_KEY")
STATE=$(echo $STATUS_JSON | jq -r '.state')
if [ "$STATE" == "ACTIVE" ]; then
echo "Processing complete! Downloading..."
break
elif [ "$STATE" == "FAILED" ]; then
echo "Error: Generation failed."
exit 1
else
echo "Current state: $STATE... (waiting 5s)"
sleep 5
fi
done
# 3. Final download
curl -L -X GET "https://generativelanguage.googleapis.com/v1beta/files/$FILE_ID:download?alt=media&key=$API_KEY" \
--output "output.mp4"
echo "Done! Video saved to output.mp4"
原始 REST JSON 结构 (URI):
{
"steps": [
{ "type": "user_input", "content": [{"type": "text", "text": "..."}] },
{ "type": "thought", "content": [{"text": "...", "type": "thought"}] },
{
"type": "model_output",
"content": [
{
"type": "video",
"mime_type": "video/mp4",
"uri": "https://generativelanguage.googleapis.com/v1beta/files/...:download?alt=media"
}
]
}
],
"id": "v1_...",
"status": "completed",
"model": "gemini-omni-1.2-flash",
"object": "interaction"
}
视频延伸 #
通过在剪辑的末尾生成无缝延续来扩展现有视频。例如,描述您希望视频如何在提示中继续 "Extend this video" 或 "Continue the scene: the camera pans across the mountains"。该模型使用输入视频的最后 10 秒来生成 3 到 10 秒延续(默认为 10 秒,或使用 duration 在 response_format),视频总长度最多为 30 秒。
您可以扩展:
- 模型生成的视频(多轮):通过引用之前生成的视频来扩展它
previous_interaction_id. - 上传的视频:提供上传的视频文件(通过 Files API)以及您的扩展提示。
import base64
from google import genai
client = genai.Client()
# Upload your video using the Files API
video_file = client.files.upload(file="my_video.mp4")
# Extend the video using prompt-based extension
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input=[
{"type": "video", "uri": video_file.uri},
{"type": "text", "text": "Continue the scene."}
],
)
with open("extended.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
// Upload your video using the Files API
let videoFile = await ai.files.upload({
file: 'my_video.mp4',
});
while (videoFile.state === 'PROCESSING') {
await new Promise(r => setTimeout(r, 10000));
videoFile = await ai.files.get({ name: videoFile.name || '' });
}
// Extend the video using prompt-based extension
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: [
{ type: 'video', uri: videoFile.uri },
{ type: 'text', text: 'Continue the scene.' }
],
});
if (interaction.output_video?.data) {
fs.writeFileSync('extended.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.interactions.VideoContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import com.google.genai.types.File;
import com.google.genai.types.UploadFileConfig;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
Client client = new Client();
// Upload your video using the Files API
File videoFile =
client.files.upload(
"my_video.mp4", UploadFileConfig.builder().mimeType("video/mp4").build());
// Extend the video using prompt-based extension
Content videoContent = VideoContent.builder().uri(videoFile.uri().get()).build();
Content textContent = TextContent.builder().text("Continue the scene.").build();
List<Content> contents = Arrays.asList(videoContent, textContent);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("extended.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
// Upload your video using the Files API
videoFile, err := client.Files.UploadFromPath(ctx, "my_video.mp4", &genai.UploadFileConfig{
MIMEType: "video/mp4",
})
if err != nil {
log.Fatal(err)
}
// Extend the video using prompt-based extension
contents := []interactions.Content{
interactions.NewContent(interactions.VideoContent{
URI: genai.Ptr(videoFile.URI),
}),
interactions.NewContent(interactions.TextContent{
Text: "Continue the scene.",
}),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(contents),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("extended.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" -H "Content-Type: application/json" -d '{
"model": "gemini-omni-1.2-flash",
"input": [
{"type": "video", "uri": "'"$VIDEO_URI"'"},
{"type": "text", "text": "Continue the scene."}
]
}'
VIDEO=$(gemini-api files upload my_video.mp4 --wait)
gemini-api video "Continue the scene." -m gemini-omni-1.2-flash -i "$VIDEO" --out extended.mp4
使用参考媒体进行扩展 #
您可以在中提供参考图像 input 数组以及在扩展视频中引入新字符或元素的提示:
import base64
from google import genai
client = genai.Client()
# Upload base video and reference image using the Files API
video_file = client.files.upload(file="my_video.mp4")
character_img = client.files.upload(file="character.png")
# Extend the video while introducing the reference character
interaction = client.interactions.create(
model="gemini-omni-1.2-flash",
input=[
{"type": "video", "uri": video_file.uri},
{"type": "image", "uri": character_img.uri},
{"type": "text", "text": "Extend this video: have the character shown in the reference image enter the scene and wave."}
],
)
with open("extended_with_character.mp4", "wb") as f:
f.write(base64.b64decode(interaction.output_video.data))
import { GoogleGenAI } from '@google/genai';
import * as fs from 'fs';
const ai = new GoogleGenAI({});
// Upload base video and reference image using the Files API
let videoFile = await ai.files.upload({ file: 'my_video.mp4' });
let characterImg = await ai.files.upload({ file: 'character.png' });
while (videoFile.state === 'PROCESSING' || characterImg.state === 'PROCESSING') {
await new Promise(r => setTimeout(r, 10000));
videoFile = await ai.files.get({ name: videoFile.name || '' });
characterImg = await ai.files.get({ name: characterImg.name || '' });
}
// Extend the video while introducing the reference character
const interaction = await ai.interactions.create({
model: 'gemini-omni-1.2-flash',
input: [
{ type: 'video', uri: videoFile.uri },
{ type: 'image', uri: characterImg.uri },
{ type: 'text', text: 'Extend this video: have the character shown in the reference image enter the scene and wave.' }
],
});
if (interaction.output_video?.data) {
fs.writeFileSync('extended_with_character.mp4', Buffer.from(interaction.output_video.data, 'base64'));
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.Model;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.interactions.VideoContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import com.google.genai.types.File;
import com.google.genai.types.UploadFileConfig;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
Client client = new Client();
// Upload base video and reference image using the Files API
File videoFile =
client.files.upload(
"my_video.mp4", UploadFileConfig.builder().mimeType("video/mp4").build());
File characterImg =
client.files.upload(
"character.png", UploadFileConfig.builder().mimeType("image/png").build());
// Extend the video while introducing the reference character
Content videoContent = VideoContent.builder().uri(videoFile.uri().get()).build();
Content imageContent = ImageContent.builder().uri(characterImg.uri().get()).build();
Content textContent =
TextContent.builder()
.text(
"Extend this video: have the character shown in the reference image enter the scene and wave.")
.build();
List<Content> contents = Arrays.asList(videoContent, imageContent, textContent);
CreateModelInteraction params =
CreateModelInteraction.builder()
.model(Model.of("gemini-omni-1.2-flash"))
.input(InteractionsInput.ofContent(contents))
.build();
Interaction interaction =
client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
if (interaction.outputVideo().isPresent() && interaction.outputVideo().get().data().isPresent()) {
byte[] videoBytes = Base64.getDecoder().decode(interaction.outputVideo().get().data().get());
Files.write(Paths.get("extended_with_character.mp4"), videoBytes);
}
package main
import (
"context"
"encoding/base64"
"log"
"os"
"google.golang.org/genai"
"google.golang.org/genai/interactions/models/interactions"
"google.golang.org/genai/interactions/models/operations"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
// Upload base video and reference image using the Files API
videoFile, err := client.Files.UploadFromPath(ctx, "my_video.mp4", &genai.UploadFileConfig{
MIMEType: "video/mp4",
})
if err != nil {
log.Fatal(err)
}
characterImg, err := client.Files.UploadFromPath(ctx, "character.png", &genai.UploadFileConfig{
MIMEType: "image/png",
})
if err != nil {
log.Fatal(err)
}
// Extend the video while introducing the reference character
contents := []interactions.Content{
interactions.NewContent(interactions.VideoContent{
URI: genai.Ptr(videoFile.URI),
}),
interactions.NewContent(interactions.ImageContent{
URI: genai.Ptr(characterImg.URI),
}),
interactions.NewContent(interactions.TextContent{
Text: "Extend this video: have the character shown in the reference image enter the scene and wave.",
}),
}
res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
Model: interactions.Model("gemini-omni-1.2-flash"),
Input: interactions.NewInteractionsInput(contents),
}),
})
if err != nil {
log.Fatal(err)
}
if res.Interaction.OutputVideo != nil && res.Interaction.OutputVideo.Data != nil {
videoBytes, err := base64.StdEncoding.DecodeString(*res.Interaction.OutputVideo.Data)
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("extended_with_character.mp4", videoBytes, 0644); err != nil {
log.Fatal(err)
}
}
}
curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions?key=$API_KEY" -H "Content-Type: application/json" -d '{
"model": "gemini-omni-1.2-flash",
"input": [
{"type": "video", "uri": "'$VIDEO_URI'"},
{"type": "image", "uri": "'$CHARACTER_IMG_URI'"},
{"type": "text", "text": "Extend this video: have the character shown in the reference image enter the scene and wave."}
]
}'
VIDEO=$(gemini-api files upload my_video.mp4 --wait)
IMAGE=$(gemini-api files upload character.png --wait)
gemini-api video "Extend this video: have the character shown in the reference image enter the scene and wave." \
-m gemini-omni-1.2-flash \
-i "$VIDEO" \
-i "$IMAGE" \
--out extended_with_character.mp4
扩展约束和指南 #
扩展视频时请记住以下规则和限制:
- 延期和总期限限制:您可以将上传或模型生成的视频(包括 15 秒剪辑)每回合延长 3 至 10 秒(默认为 10 秒),视频总长度最多为 30 秒。
- 10-第二回溯窗口:扩展视频时,模型使用输入视频的最后 10 秒作为上下文,同时保留较早的帧。如果字符或视觉元素仅在最后 10 秒之前出现,请在提示中再次提及或将其作为参考图像传递。
- 上传视频的语音对话:目前,您无法扩展上传的有人说话的视频以添加额外的对话(如果角色保持沉默或提示不添加对话,则支持)。
- 多轮语音分机:通过多轮扩展先前生成的视频时,支持生成口语对话或语音(
previous_interaction_id). - 仅剪辑结尾:扩展仅限于附加到视频末尾。您无法添加内容或延长剪辑的中间部分。
- 区域可用性:欧洲经济区 (EEA)、瑞士和英国的用户目前无法扩展上传的视频(所有可用区域均支持扩展模型生成的视频)。
最佳实践 #
- 对大视频使用 URI 传送: 对于大于 4MB 的视频(>720p,如果可用),请使用
delivery="uri"在response_format以避免有效负载大小限制。 - 优化性能: 套装
background=false,store=false, 和stream=false用于更快、同步的一元生成。注意设置store=false意味着生成的视频在后续回合中将无法使用previous_interaction_id. - 提示精度: 参见 及时指导 部分了解详细信息。
限制 #
- 第一帧和最后一帧插值(为开始帧和结束帧提供两个图像)支持从 3 到 10 秒的视频。使用两帧设置超过 10 秒的持续时间会返回无效参数错误。
- 欧洲经济区、瑞士和英国不支持上传和编辑包含未成年人的图像。
- 不支持上传和编辑包含某些可识别人物的图像。
- 目前,欧洲经济区 (EEA)、瑞士和英国的用户无法编辑或扩展上传的视频(支持编辑或扩展模型生成的视频)。
- 上传和模型生成的用于编辑的输入视频必须为 15 秒或更短。视频扩展使用输入视频的最后 10 秒,并且可以将上传或模型生成的视频每回合延长 3 至 10 秒,最高可达 30 秒的总视频长度。
- 视频扩展仅限于附加到视频末尾;不支持在剪辑的中间添加或扩展。
- 您无法扩展上传的有人说话的视频来添加额外的对话(角色可以保持沉默,或者使用多轮扩展)
previous_interaction_id可以使用)。 - 不支持语音编辑。
- 当前版本的 API 不支持上传音频输入或音频参考。
- 参考视频最适合视觉相似性(参考视频中的任何音频都会被忽略),并且仅在生成新视频时受支持,而不是在编辑或扩展现有视频时受支持。支持新视频生成、视频编辑和基于提示的视频扩展的参考图像。
- 从 11 到 15 秒的视频支持 1 参考视频(最长为 3 秒),而从 3 到 10 秒的视频最多支持 3 参考视频(每个最长为 3 秒)。参考图像在所有视频持续时间内最多支持 6 图像。
- 不支持跨多个视频进行引用或推理。尝试多视频提示可能会导致模型性能下降或意外输出。
- 不支持预配置吞吐量。
- 系统指令、温度、
top_p、停止序列和否定提示不受支持(您可以将否定提示放入常规提示中:例如“不要做 X”)。 - 不支持使用 YouTube 视频作为媒体源。
技术细节 #
- 所有生成的视频都包含 SynthID 水印,观看者看不到该水印,但可以通过编程方式检测到水印以进行出处验证。
- 视频生成时间根据持续时间、分辨率和当前 API 负载而变化。更长、更高分辨率的视频需要更多时间来生成。
- Omni 将内容安全过滤器应用于输入提示和生成的视频(因地区而异)。违反使用政策的提示将被阻止。
- 完全支持英语 (EN),但尚未评估其他语言,因此它们可能有效,但结果可能会有所不同。
Gemini Omni Flash提示指南 #
本节包含有关如何有效提示 Gemini Omni Flash 的提示和示例。
单场景 #
默认情况下,Omni Flash 将尝试使用几个不同的镜头创建视频。它将尝试根据提示编写有趣的叙述。
如果您需要输出视频包含单个场景,则必须提示:
- 在一个完整的场景中
- 在单次连续拍摄中
- 没有场景剪切
例如:
Continuous, unbroken handheld shot of a fluffy tabby cat sitting on a sunny windowsill, looking out into a leafy garden. The cat's tail twitches slowly, and its ears rotate slightly toward ambient noises. Sunbeams illuminate dust motes in the air. Sound design: Gentle breeze, distant bird chirps. No dialogue.
删除不需要的元素 #
如果生成的视频包含您不想要的内容,请添加简单的负面提示以避免它们:
- 无对话
- 没有任何装饰
- 没有额外的音效
提示编辑 #
简单的提示最适合视频编辑。过度描述性的提示可能会导致意外的更改。
以下是更多简单编辑提示的示例:
- 将此视频制作成动画
- 给这个人戴上一顶时髦的帽子
- 更改灯光以使其更具戏剧性
- 将标牌上的文字更改为“Omni Flash”
编辑视频的特定方面时,包括 "Keep everything else the same" 以保持视觉一致性。
以下是一些示例来展示如何应用此技术:
避免:
In the video of the man sitting on the sofa, please add a small black cat that runs from the right side of the screen, jumps onto his lap, and then he starts to stroke its head while looking down.* 简化:Add a cat that jumps onto his lap, he begins to pet it. Keep everything else the same.避免:
Please remove the cell phone that the person is holding in their hand and fill in the background so it looks like they are just holding their hand empty.* 简化:Make the phone invisible. Keep everything else the same.
提示音频 #
默认情况下,模型将尝试为视频生成适当的音轨。这可能并不总是您想要的。您可以使用提示来描述您想要的音频类型。如果您想在视频中添加音乐,这一点尤其重要:
- 包括平静的背景音乐
- 该视频具有高能量的科技节拍
- 音频是背景中低沉的无线电广播,正在播放歌曲
计时事件 #
您可以提示视频中特定时间发生的事情,不需要精确的语法,可以使用自然语言。这对于创建您自己的场景剪辑、节奏或快速射击序列特别有用。请参阅以下示例:
- 3 秒后,一名妇女进入场景。
- 5 秒时背景音频中开始合唱。
- 每2秒切入一个新帧。
- 在快速射击序列中,每半秒(24fps 的 12 帧)会将场景更改为新位置。
您还可以使用时间码语法:
[0-5s] A person is walking down a quiet street
[5-10s] They stop and turn around toward a glowing doorway
[10-15s] They start running into the portal
元提示 #
您可以要求Gemini Omni Flash关注视频生成的一般品质或原理:
- 考虑微观细节、表情和时机,创造一个非常丰富、细致但完全自然的场景。
- 对人物和环境的描述要非常详细。将服装设计原则应用于角色。对场景中的人物、物品和物体要非常具体。
- 在背景元素中包含大量适当的细节,使场景感觉真实自然。
- 制作一个快速射击视频,展示不同的稀有物品
[thing]每隔 1 秒,欢快的音乐,包括文本来标记该事物。
视频中的文字 #
您可以提示在视频中包含文本,Gemini Omni 将以正确且可读的方式呈现。如果您的视频中存在自然出现的文本,即使在背景元素中,它也可以帮助定义它应该说的内容。
- 屏幕上一次显示一个字:“你知道吗,Omni,可以,可以,很棒,发短信吗?”每个单词都会以不同的动画风格出现 1 秒。没有对话。
- 有一个路牌上写着:“这是 Omni 的人工智能一代”,有一个店面写着:“你需要人工智能”,有一辆车牌号:“OMNI1.2”
提示延长视频 #
使用 Gemini Omni 1.2 Flash,您可以通过提示来扩展视频,例如, "Extend this video" 或 "The scene continues"。您可以将视频每回合延长 3 至 10 秒(默认为 10 秒),总长度最多为 30 秒。
Omni 创建了一个扩展,通过使用原始视频的最后 10 秒作为上下文来保持视频、动作、角色和音频的连贯性。输入视频中的一些最终帧将被编辑以实现无缝过渡。
扩展时,本指南的所有 Omni 提示提示仍然适用:
- 描述扩展场景中的音频,特别是当您需要更改它时:
"The music continues into the chorus" - 描述场景是否继续,或者是否有镜头切换到新场景(可能具有相同的角色):
"Show the same characters in the next scene" - 包含参考图像以保持输出一致或引入新角色:
"The person shown in the reference image enters the scene" - 如果使用时间戳或时间码语法,0 表示视频扩展部分的开头。如果延长10秒的视频,则该提示中的场景切换将在12秒后发生:
"After 2s cut to a new scene with the same characters"
后续步骤 #
- 通过在以下环境中进行实验来开始使用 Gemini Omni Flash Omni 快速入门 Colab.
- 了解如何使用我们的工具编写更好的提示 提示设计简介.