Live cycle · What it does · Architecture · Physics & rig · Calibration · Quick start · Docs

Isaac uArm Twin is a real-time, physics-driven digital twin of a three-arm PCB assembly cell in NVIDIA Isaac Sim 5.x. Live servo angles from three physical uArm Swift Pro robots stream over HTTP into calibrated PhysX joint drives, while closed-loop parallel linkages, surface-velocity conveyor belts, sphere-sweep light barriers (S1–S4) and pick-as-is fixed joints reproduce the complete assembly cycle. It runs standalone or attaches directly to the companion TIA S7 Copilot robot controller.

Relationship to TIA S7 Copilot

This project extends TIA S7 Copilot with a physics-driven digital twin in NVIDIA Isaac Sim. TIA S7 Copilot handles the operator interface, PLC communication and physical robot coordination; Isaac uArm Twin adds calibrated servo-to-PhysX following, closed-loop robot linkages, conveyors and sensor simulation.

The integration uses an optional telemetry hook in the copilot's robot controller, allowing the twin to read servo angles without opening additional serial connections. A standalone runner is also provided.

See Connecting the Plant for setup instructions.

One Cycle in the Digital Twin

_{3D mesh replay rendered from the cell's real waypoint programs (sequences.py), firmware inverse kinematics and closed-loop linkage solver (kinematics.py). Illustrative software render for documentation · 2400 px high-resolution viewport still.}

1. Infeed pickup (uarm1, vacuum): picks a PCB holder from the supply table and sets it onto Conveyor 1 at light barrier S1.
2. Conveyor 1 transport (S1 → S2): S1 latches Belt 1 on (+0.15 m/s surface velocity); S2 stops the holder directly under uarm2.
3. Grip & wrist presentation (uarm2, gripper + wrist): closes on the holder, lifts it, rotates JointGripperExt and holds it at the assembly hand-off point.
4. PCB mounting (uarm3, vacuum): picks a PCB from the tray and releases it onto the held holder pocket.
5. Outfeed transport (S3 → S4): uarm2 places the finished assembly at S3, which starts Belt 2 (-0.15 m/s) until S4 stops it at the exit.

What It Does

Live servo-to-PhysX following Polls GET /api/servo_angles at 120 Hz and updates PhysX joint position targets on every 60 Hz simulation step through two-point calibration, slew-rate limiting, exponential smoothing ($\alpha = 0.28$) and a $0.05^\circ$ deadband. No inverse kinematics or trajectory planner runs in the loop.	Closed-loop parallel linkage rig Each uArm Swift Pro has 9 revolute joints and 3 closed kinematic loops. Only Joint1, Joint2, Joint6 (and JointGripperExt on arm 2) carry PD drives; the 6 passive joints follow 3 excluded-from-articulation loop-closure joints solved by PhysX.
Zero serial-port contention Arms 1 and 3 query servo angles asynchronously with a single-in-flight _pending guard so telemetry never backs up behind blocking motion commands. Arm 2 reads servos 0–2 and appends its software-tracked wrist angle.	Conveyors, light barriers & pick-as-is tools OmniGraph ScriptNodes drive kinematic conveyor belts via PhysxSurfaceVelocityAPI, detect parts with sweep_sphere_all light barriers (S1–S4), and attach parts with pose-preserving fixed joints that never snap or yaw-jump.
Clean room-scale OpenUSD generator tools/build_scene.py generates scene/cell.usda (1,702 lines, Z-up, metres) from scratch using primitive colliders and derived waypoint coordinates—no proprietary factory CAD is shipped.	Two-point hardware calibration helper isaac_uarm_twin.calibrate detaches individual servos so an arm can be posed by hand at its extended and retracted stops, captures the physical servo angles and re-engages holding torque.

How It Works

flowchart LR
    subgraph PLANT["Physical Cell"]
        A1["uArm 1 · COM3<br/>vacuum"]
        A2["uArm 2 · COM5<br/>gripper + wrist"]
        A3["uArm 3 · COM4<br/>vacuum"]
    end
    subgraph PC["Robot PC · FastAPI :8001"]
        CTRL["TIA S7 Copilot controller<br/>or isaac_uarm_twin.cell_demo"]
        TEL["ServoTelemetry<br/>GET /api/servo_angles"]
    end
    subgraph ISAAC["NVIDIA Isaac Sim 5.x · scene/cell.usda"]
        POLL["HTTP poll thread<br/>120 Hz latest sample"]
        STEP["PhysX step callback · 60 Hz<br/>calibrate → clamp → rate limit → EMA"]
        ART["3× Articulation + 3 loop joints<br/>PD force drives (32 pos / 1 vel)"]
        OG["OmniGraph ScriptNodes<br/>S1–S4 sweeps · belts · pick-as-is"]
    end
    A1 & A2 & A3 <-->|"USB serial"| CTRL
    CTRL --> TEL
    TEL -->|"JSON 1 Hz"| POLL
    POLL --> STEP --> ART
    ART <--> OG

    classDef nv fill:#14181c,stroke:#76b900,color:#ffffff
    classDef hw fill:#1f262d,stroke:#6b7785,color:#ffffff
    class POLL,STEP,ART,OG,TEL nv
    class A1,A2,A3,CTRL hw

Loading

Stage	Rate	Role
Hardware telemetry (ServoTelemetry)	1 Hz default	Samples physical servo angles without blocking serial motion commands
Follower HTTP poller (poll_http)	120 Hz	Keeps the latest JSON payload in memory; network hiccups hold the last pose
PhysX step callback (on_physics_step)	60 Hz	Applies two-point calibration, slew-rate limit, EMA ($\alpha = 0.28$) and $0.05^\circ$ deadband

Building the Physics

The uArm Swift Pro is a closed-loop mechanism, whereas a PhysX articulation must be a tree. isaac/tools/rig_swiftpro.py configures each imported robot so PhysX solves the real mechanical coupling instead of faking it in code:

Joint Group	Joints	Axis	Limits (deg)	PhysX Configuration
Base turret	Joint1	Z	-90 .. +90	Force PD drive: k = 10,000, d = 1,000, maxForce = 1e7, friction 0.05
Shoulder	Joint2	Y	-30 .. +90	Force PD drive: k = 10,000, d = 1,000, maxForce = 1e7, friction 0.05
Elbow crank	Joint6	Y	-17 .. +110	Force PD drive: k = 2,000, d = 200, maxForce = 1e6, friction 0.05
Gripper wrist (arm 2)	JointGripperExt	Z	-90 .. +90	Force PD drive: k = 20,000, d = 5,000, maxForce = 1e5, friction 0.02
Passive linkage	Joint3, 4, 5, 7, 8, 9	Y	J3: ±83.1	DriveAPI removed; free revolute joints with friction 0.1
Loop closures	Loop_Link5_Link4, Loop_Link7_Link3, Loop_Link9_Link4	Y	unconstrained	UsdPhysics.RevoluteJoint with excludeFromArticulation = True

Checked at Joint2 = 14.798° and Joint6 = 36.026°, the analytical solver solve_linkage() in isaac_uarm_twin/kinematics.py matches the passive joint angles settled by PhysX (Joint3 = 21.51°, Joint4 = -14.79°, Joint5 = 14.81°, Joint7 = -21.17°, Joint8 = -35.99°, Joint9 = 36.06°) within 0.42°, and forward kinematics reaches every SDK waypoint in sequences.py within 2.03 mm.

Calibration and 60 Hz Smoothing

Physical uArm servo readings do not share the same zero or direction as the URDF joint axes (Joint2 is inverted: -6.44° servo corresponds to +89.954° in Isaac Sim). Each driven joint is mapped from two measured anchor poses (servo_ext, servo_ret → isaac_ext, isaac_ret) captured with isaac_uarm_twin.calibrate:

Driven DOF	Physical Servo Extended → Retracted	Isaac Sim DOF Extended → Retracted	Slew-Rate Limit
Joint1 (base)	0.0° → 180.0°	-90.0° → +90.0°	360°/s
Joint2 (shoulder)	-6.44° → 129.966°	+89.954° → -29.972°	220°/s
Joint6 (elbow crank)	-22.0° → 113.0°	-17.0° → +110.0°	260°/s
JointGripperExt (arm 2 wrist)	180.0° → 0.0°	+90.0° → -90.0°	200°/s

Quick Start

1 · Set up the Python environment and generate the stage

py -3.13 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python tools/fetch_swiftpro_meshes.py
python tools/build_scene.py

Import assets/swiftpro/swiftpro_twin.urdf and assets/swiftpro/swiftpro_twin_gripper.urdf with the Isaac Sim URDF Importer, run isaac/tools/rig_swiftpro.py in the Script Editor and save them to scene/robot/swiftpro_twin/swiftpro_twin.usd and scene/robot/swiftpro_twin_gripper/swiftpro_twin_gripper.usd (see docs/building-the-twin.md).

2 · Stream telemetry from the plant (two options)

Option A — Attach to TIA S7 Copilot (recommended when the PLC cell is running):

Add two lines to the copilot's robot_controller.py before uvicorn.run(...) so the process that already owns COM3, COM5 and COM4 also serves /api/servo_angles:

from isaac_uarm_twin.telemetry import ServoTelemetry

ServoTelemetry({"arm1": arm_1, "arm2": arm_2, "arm3": arm_3}).attach(app)

Option B — Standalone cell demo (without the PLC/copilot):

$env:TWIN_RUN_MOTION = "0"   # set to "1" only when the physical cell is clear to run sequences
python -m isaac_uarm_twin.cell_demo

3 · Launch the Isaac Sim digital twin

python -m isaac_uarm_twin.launcher

The launcher waits until GET http://127.0.0.1:8001/api/servo_angles responds, then starts Isaac Sim with isaac/bootstrap.py, which opens scene/cell.usda, loads isaac/follower.py into the simulation and presses Play.

Caution

cell_demo runs physical robot motion by default unless TWIN_RUN_MOTION=0 is set, and calibrate can detach holding torque so an arm drops under gravity. Keep the hard-wired emergency stop active and read SECURITY.md before connecting to hardware.

Configuration

All settings are read from environment variables or .env (see .env.example):

Variable	Default	Purpose
ARM1_PORT · ARM2_PORT · ARM3_PORT	COM3 · COM5 · COM4	Serial ports of the three uArm Swift Pro arms
TWIN_HOST · TWIN_PORT	127.0.0.1 · 8001	Bind address and port for the telemetry service
TWIN_TELEMETRY_HZ	1.0	Per-arm servo sampling rate (Hz)
TWIN_RUN_MOTION	1	1 = run sequences.py in cell_demo; 0 = telemetry only
TWIN_TELEMETRY_URL	http://127.0.0.1:8001/api/servo_angles	Endpoint polled by isaac/follower.py and launcher.py
TWIN_CALIBRATION_PORT	8002	Port for python -m isaac_uarm_twin.calibrate
ISAAC_SIM_EXE · TWIN_SCENE	isaacsim · scene/cell.usda	Simulator executable and stage path used by launcher.py

Project Layout

isaac-uarm-twin/
├── isaac_uarm_twin/
│   ├── telemetry.py             FastAPI /api/servo_angles router (async arms 1 & 3, sync + wrist arm 2)
│   ├── kinematics.py            Swift Pro firmware IK + closed-loop planar linkage solver
│   ├── sequences.py             3-arm assembly waypoint sequences from the physical cell
│   ├── cell_demo.py             standalone runner (sequences + telemetry on :8001)
│   ├── calibrate.py             limp-servo pose capture service (:8002)
│   ├── launcher.py              waits for telemetry, launches Isaac Sim non-blocking
│   └── settings.py              environment / .env configuration
├── isaac/
│   ├── bootstrap.py             opens the stage, loads follower.py and starts Play
│   ├── follower.py              120 Hz HTTP poller + 60 Hz PhysX PD target follower
│   ├── graphs/
│   │   ├── conveyor_controller.py   OmniGraph ScriptNode: S1–S4 sphere sweeps + surface velocity
│   │   └── vacuum_autograb.py       OmniGraph ScriptNode: pose-preserving fixed-joint pickup + UI
│   └── tools/
│       ├── rig_swiftpro.py      configures PD drives, passive joints, 3 loop joints & solver counts
│       ├── surface_gripper.py   alternative Isaac SurfaceGripper suction setup
│       ├── grab_part.py         manual pick-as-is fixed-joint test helper
│       └── release_part.py      manual fixed-joint release helper
├── assets/swiftpro/             URDFs for suction arms (1, 3) and gripper+wrist arm (2)
├── scene/cell.usda              generated Z-up OpenUSD cell stage (1,702 lines)
├── tools/                       scene generator, SHA-256 mesh fetcher & 3D artwork renderer
├── tests/                       196 hardware-free pytest unit & integration tests
└── docs/                        build guide, plant integration, physics & engineering design log

Verification

python -m pip install -r requirements-dev.txt
python tools/build_scene.py --check
python -m ruff check .
python -m pytest

The 196 hardware-free tests verify:

• two-point calibration anchors, clamping, slew-rate limiting, EMA convergence, deadband and late-arm discovery in isaac/follower.py
• non-overlapping async callbacks, wrist tracking, fallback handling and FastAPI routes in isaac_uarm_twin/telemetry.py
• closed-loop linkage solver agreement with PhysX passive joint states (< 0.5°) and IK→FK tool-tip accuracy (< 2.5 mm) across all cell waypoints
• pose-preserving pickup anchor math (local_hold and local_part coincide without snapping under arbitrary part rotations and stage scales)
• fail-fast bootstrap startup, USDA syntax/target integrity, URDF tree structure and 3D renderer depth-buffer invariants

Documentation

Document	Contents
Building the Twin	Fetching meshes, URDF import settings, running rig_swiftpro.py and first-run checklist
Connecting the Plant	Attaching ServoTelemetry to TIA S7 Copilot, JSON payload format and startup behavior
Physics & Calibration	Parallel-linkage constraints, PD gains, two-point calibration, conveyor/sensor physics and limits
Engineering Design Log	Evolution from ROS 2 / WSL bridges to the final HTTP + PhysX architecture and lessons learned
Reproducing the Artwork	Regenerating the 3D mesh renders, animations and calibration charts
Security & Safety	Machine safety, serial port ownership, network exposure and data hygiene

License and Provenance

Project-authored code, OpenUSD scene generation and documentation are released under the MIT License.

• No proprietary CAD or factory USD assets are included. The table, riser, conveyors, light barriers and parts in scene/cell.usda are generated from scratch by tools/build_scene.py.
• uArm Swift Pro meshes (Base.STL, Link1.STL–Link9.STL, Suction.STL, Gripper.STL) are not redistributed in this repository; tools/fetch_swiftpro_meshes.py downloads them from a pinned commit of uArm-Developer/RosForSwiftAndSwiftPro and verifies each file's SHA-256 hash. Review upstream licensing notes in assets/swiftpro/swiftpro_twin.urdf before redistributing fetched meshes.
• Visualizations in docs/assets/ are deterministic Python software renders produced from the verified public uArm meshes and procedural cell geometry.

NVIDIA, Isaac Sim, Omniverse, PhysX, OpenUSD, uArm and other product names are trademarks of their respective owners. This is an independent project and is not affiliated with or endorsed by any of them.