Add OpenVINO operator profiling to perf - #1377
Conversation
# Conflicts: # src/winml/modelkit/commands/perf.py
| # normalized) — an exact `==` can't reconcile the two. A bounded range keeps the | ||
| # 1.24.x series while resolving to the published wheel. | ||
| "onnxruntime-windowsml>=1.24.5,<1.25 ; sys_platform == 'win32'", | ||
| "onnxruntime-windowsml>=1.24.5,<1.28 ; sys_platform == 'win32'", |
There was a problem hiding this comment.
This range now resolves to ORT 1.27.1, but the pinned windowsml==2.0.300 declares onnxruntime-windowsml==1.24.5.202604171637 as its matching runtime; windowsml==2.3.42 is the package paired with ORT 1.27.1. Please update the WindowsML package in lockstep rather than installing a mismatched WinML C API/runtime pair.
| self._device = normalized_device | ||
| self._output_dir = Path(output_dir) if output_dir is not None else Path.cwd() | ||
| self._output_dir.mkdir(parents=True, exist_ok=True) | ||
| self._profile_prefix = (self._output_dir / "onnxruntime_profile").resolve() |
There was a problem hiding this comment.
Could we make this prefix unique per monitor? output_dir is normally the save directory or cwd, and ORT only appends a millisecond timestamp. Overlapping perf processes can therefore create the same filename or have _find_fresh_profile() select the other process's newer profile. A per-run UUID in the prefix would keep discovery scoped to this trace.
|
|
||
| def configure_session_options(self, session_options: ort.SessionOptions) -> None: | ||
| """Enable ORT profiling on the monitored inference session.""" | ||
| session_options.enable_profiling = True |
There was a problem hiding this comment.
For an NPU session, ORT OpenVINO 1.27.1 explicitly falls back to OV CPU when compilation fails unless session.disable_cpu_ep_fallback is set. Because that fallback stays inside OpenVINOExecutionProvider, these events still pass the parser and are returned as status="ok", device="npu" even though they are CPU timings. Please disable this fallback for NPU profiling or validate/report the actual OV device.
| tracing_backend="onnxruntime", | ||
| num_samples=self._measured_iterations | ||
| or max(operator.sample_count for operator in operators), | ||
| summary={"accel_execute_us": total_us}, |
There was a problem hiding this comment.
accel_execute_us is a per-inference mean in the existing basic/QNN schema, but total_us sums every retained run. The added 4 us / 6 us test therefore displays 10 us, and this value grows with --iterations even when latency is unchanged. Please compute the mean of the per-inference accelerator totals here.
| if self._measured_iterations is not None | ||
| else None | ||
| ) | ||
| retained[parent_name] = occurrences[self._warmup_iterations : stop] |
There was a problem hiding this comment.
These occurrences are partition executions, not necessarily top-level inference runs. A partition inside Loop or Scan can execute multiple times per inference, so this slice removes only the first loop execution from the warmup and can retain the rest of the warmup as measured samples. Please group spans by their enclosing inference run before applying the warmup/sample window.
2 participants
Summary
ORT compatibility
ORT 1.27 validates supplied and default local-function graph attributes even when the function body does not reference or execute them. FP16 conversion now checks those attributes before conversion and rejects captured tensors whose outer binding becomes FP16 while the skipped graph remains FP32. This prevents the
Tensor element type mismatch. 10 != 1session-loading failure and remains compatible with ORT 1.24 by applying the stricter validation conservatively.