Gemini 中文指南
获取 API 密钥 ↗

使用 WebSocket 快速入门 Gemini Live API

↓ 下载 MarkdownInteractions API

Gemini Live API 允许与 Gemini 模型进行实时双向交互,支持音频、视频和文本输入以及本机音频输出。本指南介绍了如何使用原始 WebSocket 直接与 API 集成。

尝试 Google AI Studio 中的 Live API

从 GitHub 克隆示例应用程序

使用编码智能体技巧

概览 #

Gemini Live API 使用 WebSocket 进行实时通信。与使用 SDK 不同,此方法涉及直接管理 WebSocket 连接并以 API 定义的特定 JSON 格式发送/接收消息。

关键概念:

认证 #

通过将您的 API 密钥作为查询参数包含在 WebSocket URL 中来处理身份验证。

端点格式为:

代码
wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=YOUR_API_KEY

更换 YOUR_API_KEY 与您实际的 API 密钥一起。

使用临时 token 进行身份验证 #

如果您正在使用 短暂的 token,您需要连接到 v1beta 端点。临时 token 需要作为 access_token 查询参数。

临时密钥的端点格式为:

代码
wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContentConstrained?access_token={short-lived-token}

更换 {short-lived-token} 与实际的临时 token。

连接到 Live API #

要启动实时会话,请与经过身份验证的端点建立 WebSocket 连接。通过 WebSocket 发送的第一条消息必须是 BidiGenerateContentSetup 含有 config。有关完整的配置选项,请参阅 Live API - WebSocket API 参考.

python
import asyncio
import websockets
import json

API_KEY = "YOUR_API_KEY"
MODEL_NAME = "gemini-3.8-live"
WS_URL = f"wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key={API_KEY}"

async def connect_and_configure():
    async with websockets.connect(WS_URL) as websocket:
        print("WebSocket Connected")

        # 1. Send the initial configuration
        setup_message = {
            "setup": {
                "model": f"models/{MODEL_NAME}",
                "responseModalities": ["AUDIO"],
                "systemInstruction": {
                    "parts": [{"text": "You are a helpful assistant."}]
                }
            }
        }
        await websocket.send(json.dumps(setup_message))
        print("Configuration sent")

        # Keep the session alive for further interactions
        await asyncio.sleep(3600) # Example: keep open for an hour

async def main():
    await connect_and_configure()

if __name__ == "__main__":
    asyncio.run(main())

发送短信 #

要发送文本输入,请构造一个 BidiGenerateContentRealtimeInput 消息与 text 场。

python
# Inside the websocket context
async def send_text(websocket, text):
    text_message = {
        "realtimeInput": {
            "text": text
        }
    }
    await websocket.send(json.dumps(text_message))
    print(f"Sent text: {text}")

# Example usage: await send_text(websocket, "Hello, how are you?")

发送音频 #

音频需要作为原始 PCM 数据发送(原始 16 位 PCM 音频,16kHz,小端)。构造一个 BidiGenerateContentRealtimeInput 带有音频数据的消息。这 mimeType 至关重要。

python
# Inside the websocket context
async def send_audio_chunk(websocket, chunk_bytes):
    import base64
    encoded_data = base64.b64encode(chunk_bytes).decode('utf-8')
    audio_message = {
        "realtimeInput": {
            "audio": {
                "data": encoded_data,
                "mimeType": "audio/pcm;rate=16000"
            }
        }
    }
    await websocket.send(json.dumps(audio_message))
    # print("Sent audio chunk") # Avoid excessive logging

# Assuming 'chunk' is your raw PCM audio bytes
# await send_audio_chunk(websocket, chunk)

有关如何从客户端设备(例如浏览器)获取音频的示例,请参阅上的端到端示例 GitHub.

发送视频 #

视频帧作为单独的图像(例如 JPEG 或 PNG)发送。与音频类似,使用 realtimeInput 与一个 Blob,指定正确的 mimeType.

python
# Inside the websocket context
async def send_video_frame(websocket, frame_bytes, mime_type="image/jpeg"):
    import base64
    encoded_data = base64.b64encode(frame_bytes).decode('utf-8')
    video_message = {
        "realtimeInput": {
            "video": {
                "data": encoded_data,
                "mimeType": mime_type
            }
        }
    }
    await websocket.send(json.dumps(video_message))
    # print("Sent video frame")

# Assuming 'frame' is your JPEG-encoded image bytes
# await send_video_frame(websocket, frame)

有关如何从客户端设备(例如浏览器)获取视频的示例,请参阅上的端到端示例 GitHub.

接收回复 #

WebSocket 将发回 BidiGenerateContentServerMessage 消息。您需要解析这些 JSON 消息并处理不同类型的内容。

python
# Inside the websocket context, in a receive loop
async def receive_loop(websocket):
    async for message in websocket:
        response = json.loads(message)
        print("Received:", response)

        if "serverContent" in response:
            server_content = response["serverContent"]
            # Receiving Audio
            if "modelTurn" in server_content and "parts" in server_content["modelTurn"]:
                for part in server_content["modelTurn"]["parts"]:
                    if "inlineData" in part:
                        audio_data_b64 = part["inlineData"]["data"]
                        # Process or play the base64 encoded audio data
                        # audio_data = base64.b64decode(audio_data_b64)
                        print(f"Received audio data (base64 len: {len(audio_data_b64)})")

            # Receiving Text Transcriptions
            if "inputTranscription" in server_content:
                print(f"User: {server_content['inputTranscription']['text']}")
            if "outputTranscription" in server_content:
                print(f"Gemini: {server_content['outputTranscription']['text']}")

        # Handling Tool Calls
        if "toolCall" in response:
            await handle_tool_call(websocket, response["toolCall"])

# Example usage: await receive_loop(websocket)

有关如何处理响应的示例,请参阅上的端到端示例 GitHub.

处理工具调用 #

当模型请求工具调用时, BidiGenerateContentServerMessage 将包含一个 toolCall 场地。您必须在本地执行该函数并将结果发送回WebSocket使用 BidiGenerateContentToolResponse 信息。

python
# Placeholder for your tool function
def my_tool_function(args):
    print(f"Executing tool with args: {args}")
    # Implement your tool logic here
    return {"status": "success", "data": "some result"}

async def handle_tool_call(websocket, tool_call):
    function_responses = []
    for fc in tool_call["functionCalls"]:
        # 1. Execute the function locally
        try:
            result = my_tool_function(fc.get("args", {}))
            response_data = {"result": result}
        except Exception as e:
            print(f"Error executing tool {fc['name']}: {e}")
            response_data = {"error": str(e)}

        # 2. Prepare the response
        function_responses.append({
            "name": fc["name"],
            "id": fc["id"],
            "response": response_data
        })

    # 3. Send the tool response back to the session
    tool_response_message = {
        "toolResponse": {
            "functionResponses": function_responses
        }
    }
    await websocket.send(json.dumps(tool_response_message))
    print("Sent tool response")

# This function is called within the receive_loop when a toolCall is detected.

后续步骤 #

搜索标题、API 名称或正文内容