---

AssemblyAI Voice Agent Starter for Python

Voice agents defined as JSON files. Publish one to your AssemblyAI account, then talk to it in a browser tab or by calling a phone number.

Each file in agents/ is the request body for POST /v1/agents. The starter sends it unchanged, saves the agent ID it gets back to .env, and both deployments connect using that ID. An agent you already have goes the other way, python import_agent.py <agent-id> turns it into one of these files. Built on the AssemblyAI Voice Agent API. Python 3.9 or later, standard library only, so there is nothing to pip install.

There is a JS version of this repo with the same agents and the same steps.

Quickstart

1. Clone

git clone https://github.com/AssemblyAI/voice-agent-starter-python
cd voice-agent-starter-python
cp .env.example .env

2. Add your key

From assemblyai.com/dashboard/api-keys:

# .env
ASSEMBLYAI_API_KEY=your_key_here

3. Get an agent

Publish one of the examples:

python publish.py                       # agents/minimal.jsonc
# AGENT=http-tools python publish.py    # or any other file in agents/

Or import one you already have, shaped in the playground or the dashboard:

python import_agent.py <agent-id>          # writes agents/<its-name>.jsonc

Either way you end up with the same pair: a file in agents/ and its id in .env as AGENT_ID_<NAME>. Publishing again updates that agent rather than creating another, and each file keeps its own, so switching with AGENT= never overwrites the last one.

4. Talk to it

python deployment/browser/server.py

Open http://localhost:3000 and start the call.

5. Put it on a phone number

# .env
TWILIO_ACCOUNT_SID=AC...                          # console.twilio.com, top of the page
TWILIO_AUTH_TOKEN=your_token_here                 # same place, hidden until you click it
TWILIO_PHONE_NUMBER=+15551234567                  # a number already in your account, E.164
TWILIO_TRUNK_DOMAIN=acme-agent.pstn.twilio.com    # a name you invent, must end .pstn.twilio.com

The trunk domain does not exist yet. You are naming the SIP trunk that gets created for you, and the name has to be unique across all of Twilio, so put something specific to you in front of .pstn.twilio.com. The phone number does have to exist already: buy one under Phone Numbers in the Twilio console first.

python deployment/telephony/connect.py

This creates the trunk, routes it to AssemblyAI, attaches your number to it, and binds the agent. Then call the number. Details in deployment/telephony.

---

Core examples

Nine agent files. Four demonstrate a parameter, five demonstrate an integration.

AGENT=	Demonstrates	Requires
minimal	the three required fields, and the defaults applied to the rest
keyterms	biasing transcription toward names and jargon
turn-taking	silence thresholds and interruption handling
byo-llm	Claude through the AssemblyAI gateway, or your own endpoint
http-tools	tools that AssemblyAI calls on the agent's behalf
exa-search	web search during a call	EXA_API_KEY
airtable-crm	reading a caller record and writing one back	AIRTABLE_*
cal-booking	checking availability, then booking a slot	CAL_*
dtmf	PCI compliance: card entry on the keypad, never in the transcript, the logs or the model	DTMF_WEBHOOK_URL

AGENT=exa-search python publish.py
python deployment/browser/server.py

To write your own, copy the closest file: cp agents/http-tools.jsonc agents/my-agent.jsonc. Every field is commented, with a link to the documentation page that defines it.

Importing an agent

The playground is the quickest way to shape an agent. This is how it moves into code without being rebuilt by hand:

python import_agent.py 8f3c1e2a-...

It writes agents/<name>.jsonc, the live agent as a file, headed with the id it came from. It records AGENT_ID_<NAME> in .env, so python publish.py sends a PUT to that same agent instead of creating a second one. It drops id, created_at and updated_at, which are not part of a create request. And it refuses to overwrite an existing file unless you pass AGENT=<other-name> or OVERWRITE=1.

Credentials are the one thing it cannot recover. Tool header values and llm[].api_key are write-only on the API, so they come back blank. The import names the ones to restore, and they belong in .env, referenced from the file as ${VARS}:

Header values are write-only and did not come back for: lookup.
Put them in .env and reference them as ${VARS}.

From there it behaves like any other file in agents/: edit it, publish, call.

Where it answers

Browser	python deployment/browser/server.py	Serves a page with a call button and mints session tokens. The API key stays on the server.
Phone	python deployment/telephony/connect.py	Configures a Twilio SIP trunk and attaches the agent to your number.

Twilio passes the call to AssemblyAI over SIP, so nothing in this repo sits in the audio path.

Hosting the browser app

Render reads render.yaml and prompts for exactly one value, ASSEMBLYAI_API_KEY, because that is the only variable marked sync: false. It sets PORT itself. The other two arrive with defaults you can change under Environment on the service:

Variable	Default	What it does
ASSEMBLYAI_API_KEY	prompted	Stays on the server. Never sent to the page.
AGENT	minimal	Which agents/<name>.jsonc the service publishes when it boots.
AGENT_ID	empty	Paste an id from your .env to serve that exact agent, whichever file it came from.

Leaving AGENT_ID empty is fine. The service publishes AGENT on boot, and on later restarts it updates the agent of that name rather than creating another one.

How it works

  copy an example                     python import_agent.py <id>
  or write your own                   an agent you already have
           │                                   │
           ▼                                   ▼
agents/exa-search.jsonc     body of POST /v1/agents
        + .env              the ${VARS} it references
           │
           ▼  python publish.py
      AGENT_ID_EXA_SEARCH
           ├──  browser/server.py      browser tab
           └──  telephony/connect.py   phone number

The first publish sends POST /v1/agents and stores the returned ID in .env under a key of its own, AGENT_ID_EXA_SEARCH for that file. Later publishes send PUT /v1/agents/{id}, so the browser tab and the phone number both pick up the change on the next call, and publishing a different file leaves this one alone. A bare AGENT_ID overrides every per-file key.

Values written as ${VAR} anywhere in an agent file are substituted at publish time from .env, or from agents/<name>.env for credentials only one agent uses. Both files are gitignored, so the JSON can be committed.

Build with AI coding agents

This repo includes AGENTS.md, which Claude Code, Cursor and Copilot read for its conventions. The Voice Agent API changes, so point coding tools at the current documentation rather than letting them work from memory:

Always fetch https://assemblyai.com/docs/llms.txt before writing AssemblyAI code. The API has changed, do not rely on memorized parameter names.

claude mcp add --transport http --scope user assemblyai-docs https://mcp.assemblyai.com/docs
npx skills add AssemblyAI/assemblyai-skill --global

See Build with AI tools and Coding agent prompts.

Voice Agent API

Product: Voice Agent API · Pricing · Dashboard

Start here: Documentation · Create an agent · Manage agents · Prompting guide · Best practices

Configuration: Voices · Greeting · Turn detection · Keyterms · Languages · Noise suppression · Custom LLM

Tools: Overview · HTTP tools · Client-side tools

Deployment: Deploy · Browser integration · Connect to Twilio · Use your own number · Webhooks

Reference: Session configuration · Events · Message sequence · Session history · Troubleshooting

Cost

Sessions are billed to the API key that published the agent. Anyone with the deployed URL or the phone number can start a session on that key.