This project is for testing load generation scenarios against Temporal. This is primarily used by the Temporal team to benchmark features and situations. Backwards compatibility may not be maintained.
- Running & configuring omes — run commands, load configuration, worker profiles, and running against a specific SDK version.
- Authoring scenarios — writing a new scenario, choosing an executor, exposing options, and authoring conventions.
Omes (pronounced oh-mess) is the Hebrew word for "load" (עומס).
- Go 1.25+
protoc+protoc-gen-goor mise for Kitchen Sink Workflow
- Java 8+
- TypeScript: Node 16+
- Python: uv
- .NET
And if you're running the fuzzer (see below)
SDK and tool versions are defined in versions.env.
This (simplified) diagram shows the main components of Omes:
flowchart TD
subgraph "CLI"
RunWorker["run-worker"]
RunScenario["run-scenario"]
RunScenarioWithWorker["run-scenario-with-worker"]
end
Workers["Worker(s)"]
WorkflowsAndActivities["Workflows and Activities"]
RunScenarioWithWorker --> RunWorker
RunScenarioWithWorker --> RunScenario
RunWorker --> |"start"| Workers
RunScenario --> |"start"| Scenario
Scenario --> |"start"| Executor
Workers --> |"consume"| WorkflowsAndActivities
Executor --> |"produce"| WorkflowsAndActivities
- Scenario: starts an Executor to run a particular load configuration
- Executor: produces concurrent executions of workflows and activities requested by the Scenario
- Workers: consumes the workflows and activities started by the Executor
Omes includes a process metrics sidecar - a Go-based HTTP server that monitors CPU and memory usage of the worker process. The sidecar is started by run-worker (or run-scenario-with-worker) after spawning the worker subprocess.
Endpoints:
/metrics- Prometheus-format metrics (process_cpu_percent,process_resident_memory_bytes)/info- JSON metadata (sdk_version,build_id,language)
Configuration flags:
--worker-process-metrics-address- Address for the sidecar HTTP server (required to enable)--worker-metrics-version-tag- SDK version to report in/info(defaults to--version)
Example:
go run ./cmd/omes run-worker --language python --run-id my-run \
--worker-process-metrics-address :9091 \
--worker-metrics-version-tag v1.24.0Metrics export:
When using the Prometheus instance (--prom-instance-addr), sidecar metrics can be exported to parquet files on shutdown. The export includes SDK version, build ID, and language from the /info endpoint.
--prom-export-worker-metrics- Path to export metrics parquet file--prom-export-process-job- Prometheus job name for process metrics (default:omes-worker-process)--prom-export-worker-info-address- Address to fetch/infofrom during export (e.g.,localhost:9091)
Scenarios are plain Go files in the scenarios folder that register themselves from
init(). The file name becomes the scenario name, and most scenarios use the KitchenSinkExecutor, which
runs the Kitchen Sink workflow — implemented in every worker language — from a declarative action tree.
See docs/authoring-scenarios.md for writing one: registration and the
filename rule, choosing between KitchenSinkExecutor and a custom Executor, exposing --option
values, and the authoring conventions.
See docs/running.md for how to run omes (run-scenario-with-worker,
run-worker, run-scenario, cleanup-scenario), configure the offered load, select a worker profile,
and run against a specific SDK version.
For example, to build a go worker image using v1.24.0 of the Temporal Go SDK:
go run ./cmd/dev build-worker-image --language go --version v1.24.0This will produce an image tagged like <current git commit hash>-go-v1.24.0.
If version is not specified, the SDK version specified in versions.env will be used.
Publishing images is done via CI, using the build-push-worker-image command.
See the GHA workflows for more information.
The project scenario is different from the built-in kitchen-sink style scenarios. It is intended
to be a more ergnomomic way of writing load tests, akin to writing a Temporal sample.
Instead of driving workflows and activities that Omes owns (through a client that Omes owns), the
project scenario enables you to write a project where you configure the worker(s), client, and load
(workflows, activites, Nexus operations, etc.), and pass them as entrypoints to a harness. Omes calls into
this harness repeatedly (i.e. for 'x' --iterations or 'y' --duration) through a gRPC interface to
generate load.
As such, it's a good fit if you:
- need to write a load test
- are more familiar with Temporal-native code than Omes's framework
and are not restricted by the current limitations:
- the load pattern is limited to a steady-rate executor (i.e. "run 'x' times or run for 'y' duration), more nuanced load patterns will need to create their own scenario + executor (the existing method)
Writing a project should be fairly similar to writing a Temporal sample, and requires only a little familiarity with the project harness interface.
To get started:
- projects should be written as apps under
workers/<lang>/apps/<name> - register the app in
workers/<lang>/apps/registry.* - use the
helloworldapp in each worker language as a reference to get started - take a look at the language harness entrypoint (for example
workers/python/harness/src/harness/__init__.py) to see how worker and project-server modes dispatch
On the worker-side, it's fairly similar to how Omes starts and runs workers already:
- Omes builds a program that can start a worker via a command
- Omes executes the command that starts the worker process against the built program The only difference is that the program is now your project + harness.
On the runner-side, it's a bit different than typical Omes scenarios because the runner needs to call into the harness to drive load. Consequently, it too needs to build your project + harness to run. So the flow looks like:
- Omes runner builds a program that can start a gRPC server (
project-server) via a commandproject-serveris language-agnostic interface for runner to tell your project to create load
- Omes runner executes the command that starts the
project-serverin another process - Omes runner calls into gRPC server repeatedly over the course of the load test to spawn load
- Omes runner shuts down
project-serveron conclusion of load test
To run a project:
- Run your project worker:
go run ./cmd/omes run-worker \
--language <lang> \
--app helloworld \
--run-id local-project-test \
--server-address <your server address>Make sure to point your server-address to a running Temporal server. Alternatively, use the
--embedded-server to spin one up for you.
- Run the project scenario
go run ./cmd/omes run-scenario \
--scenario project \
--iterations 1 \
--server-address <your server address> \
--run-id local-project-test \
--option language=<lang> \
--option project-name=helloworld \
--option project-server-ready-timeout=15sFor local all-in-one development, run the worker and scenario together:
go run ./cmd/omes run-scenario-with-worker \
--scenario project \
--iterations 1 \
--language <lang> \
--app helloworld \
--run-id local-project-test \
--embedded-server \
--option language=<lang> \
--option project-name=helloworldTo run a project via Docker:
docker build \
-f dockerfiles/<lang>.Dockerfile \
--build-arg SDK_VERSION=<sdk-version> \
-t omes-<lang> .
docker network create omes-project-net
docker run -d --rm \
--name omes-<lang>-project-worker \
--network omes-project-net \
omes-<lang> \
--app helloworld \
--run-id local-project-test \
--embedded-server-address 0.0.0.0:7233
docker run --rm \
--network omes-project-net \
--entrypoint /app/temporal-omes \
omes-<lang> \
run-scenario \
--scenario project \
--run-id local-project-test \
--server-address omes-<lang>-project-worker:7233 \
--iterations 1 \
--connect-timeout 30s \
--option language=<lang> \
--option project-name=helloworld \
--option prebuilt-project-dir=/app/workers/<lang>/prepared
docker stop omes-<lang>-project-worker
docker network rm omes-project-netWorker images contain a single prepared worker package for their language. Select the app at runtime
with --app for workers and --option project-name=... for project scenarios.
The throughput_stress scenario can be configured to run "sleep" activities with different configurations.
The configuration is done via a JSON file, which is passed to the scenario with the
--option sleep-activity-per-priority-json=@<file> flag. Example:
echo '{"count":{"type":"fixed","value":5},"groups":{"high":{"weight":2,"sleepDuration":{"type":"uniform","min":"2s","max":"4s"}},"low":{"weight":3,"sleepDuration":{"type":"discrete","weights":{"5s":3,"10s":1}}}}}' > sleep.json
go run ./cmd/omes run-scenario-with-worker --scenario throughput_stress --language go --option sleep-activity-json=@sleep.json --run-id default-run-id
This runs 5 sleep activities per iteration, where "high" has a weight of 2 and sleeps for a random duration between 2-4s, and "low" has a weight of 3 and sleeps for either 5s or 10s.
Look at DistributionField to learn more about different kinds of distrbutions.
The throughput_stress scenario can generate Nexus load if the scenario is started with --option nexus-endpoint=my-nexus-endpoint:
temporal operator nexus endpoint create \
--name my-nexus-endpoint \
--target-namespace default \ # Change if needed
--target-task-queue throughput_stress:default-run-id
- Start the scenario with the given run-id:
go run ./cmd/omes run-scenario-with-worker --scenario throughput_stress --language go --option nexus-endpoint=my-nexus-endpoint --run-id default-run-id
The throughput_stress scenario can generate standalone-activity load (activities started outside
any workflow context via StartActivityExecution) with --option include-standalone-activity=true.
This requires server support for standalone activities (dynamic config activity.enableStandalone).
Implemented for the Go, Python, TypeScript, .NET, Java, and Ruby workers.
The fuzzer scenario makes use of the kitchen sink workflow (see below) to exercise a wide
range of possible actions. Actions are pre-generated by the kitchen-sink-gen tool, written in
Rust, and are some combination of actions provided to the workflow as input, and actions to be
run by a client inside the scenario executor.
You can run the fuzzer with new random actions like so:
go run ./cmd/omes run-scenario-with-worker --scenario fuzzer --iterations 1 --language csThe fuzzer automatically creates a Nexus endpoint and generates Nexus operations. To use an existing endpoint instead:
go run ./cmd/omes run-scenario-with-worker --scenario fuzzer --iterations 1 --language go --option nexus-endpoint=my-endpointBy default, the scenario will spit out a last_fuzz_run.proto binary file containing the generated
actions. To re-run the same set of actions, you can pass in such a file like so:
go run ./cmd/omes run-scenario-with-worker --scenario fuzzer --iterations 1 --language cs --option input-file=last_fuzz_run.protoOr you can run with a specific seed (seeds are printed at the start of the scenario):
go run ./cmd/omes run-scenario-with-worker --scenario fuzzer --iterations 1 --language cs --option seed=131962944538087455However, the fuzzer is also sensitive to its configuration, and thus the seed will only produce the exact same set of actions if the config has also not changed. Thus you should prefer to save binary files rather than seeds.
Please do collect interesting fuzz cases in the scenarios/fuzz_cases.yaml file. This file
currently has seeds, but could also easily reference stored binary files instead.
The Kitchen Sink workflows accepts a DSL generated by the kitchen-sink-gen Rust tool, allowing us
to test a wide variety of scenarios without having to imagine all possible edge cases that could
come up in workflows. Input may be saved for regression testing, or hand written for specific cases.
Build by running go run ./cmd/dev build kitchensink.
Test by running go test -v ./loadgen -run TestKitchenSink.
Prefix with env variable SDK=<sdk> to test a specific SDK only.
A scenario can only fail if an Execute method returns an error, that means the control is fully in the scenario
authors's hands. For enforcing a timeout for a scenario, use options like workflow execution timeouts or write a
workflow that waits for a signal for a configurable amount of time.
- Nicer output that includes resource utilization for the worker (when running all-in-one)
- Ruby worker
Use the dev command for development tasks:
go run ./cmd/dev install # Install tools (default: all)
go run ./cmd/dev lint-and-format # Lint and format workers (default: all)
go run ./cmd/dev test # Test workers (default: all)
go run ./cmd/dev build # Build worker images (default: all)
go run ./cmd/dev clean # Clean worker artifacts (default: all)
go run ./cmd/dev build-proto # Build kitchen-sink protoOr target specific languages: go run ./cmd/dev build go java python
All versions are defined in versions.env.