hal: C++ API (hal.hh) and pybind11 bindings on the new HAL API

[grandixximo]

@rene-dev @BsAtHome - this is the unification pass we discussed in the #4099 meeting: Rene's C++/pybind11 interface rebuilt on Bertho's getter/setter API, satisfying both constraints (no HAL internals access; simple, low-maintenance, self-documenting C++ surface).

Rebased onto master after #4565, as a single commit. It fills the slot left by the removal of the old hal.hh and is otherwise additive: _hal/hal.py are untouched and nothing is replaced. The two earlier commits for hal.query registration and the Type/Dir IntEnums are gone, since master now carries both.

What this adds

• src/hal/hal.hh - type-safe C++ layer, header-only, in namespace linuxcnc::hal, exported to include/:
  • hal::pin<T> typed handles via traits<T> over rtapi_bool, rtapi_sint, rtapi_uint and rtapi_real. 64-bit only, matching the API break: there are no 32-bit handles. Handles re-read the shmem slot per access, so hal_link() rewrites stay visible (same semantics as C pin pointers). Handles are move-only; HAL objects are unique.
  • hal::pin_t variant + hal::anypin: runtime-typed access multiplexes on the variant tag (the stored-type pattern required for name-keyed collections).
  • hal::component: pins + params, newpin/newparam (typed and runtime-typed), add_pin for the struct-member idiom, item access, prefix handling.
  • hal::port: move-only handle for HAL_PORT pins, created with newport() or the runtime-typed newpin(). All-or-nothing read/peek/peek_commit/write plus readable/writable/size/clear. The buffer belongs to the linking signal and is sized with set_signal() (sets); an unlinked port has no buffer, so its reads and writes fail without log noise. By name and through the runtime-typed item a port reads as its buffer size, like hal.query, and cannot be set.
  • hal::stream: move-only wrapper around hal_stream_t, create or attach, with the same conversion and range checks as the by-name setters.
  • ULAPI by-name functions (get_value, set_value, set_signal, component_exists, pin_has_writer, signal management) implemented entirely on hal_get_p/hal_set_p/hal_get_s/hal_set_s/hal_comp_by_name. No hal_priv.h, no direct shmem access, no re-implemented parsers.
• src/hal/halpybind.cc - pybind11 module (halpp.so), adapted from @rene-dev's bindings, extended with params, ports, streams and error paths. Type and direction tags are the _hal.Type/_hal.Dir IntEnum classes, the same objects hal.query uses: arguments accept members or plain ints, results come back as members. Port pins take and return bytes (str is written as UTF-8); read()/peek() return None when fewer than the requested bytes are readable. Built alongside _hal/hal.py; module replacement is a separate (3.0) discussion.
• tests/halpp/ - Python and native C++ smoke suites, plus a stream create/attach pair across two processes.

Safety properties worth noting

• Query callbacks are strictly exception-free: conversion errors are reported by flag and thrown after the library releases the HAL mutex. (Throwing through a callback wedges the session-wide mutex; found the hard way, and tested.)
• Set-side coercion (by name, runtime-typed items and stream samples) uses the item's type with range checks; an out-of-range value raises instead of truncating or wrapping.
• Typed access compiles to the same inline accessor calls as the C API, so there is no hot-path cost.

Removed compared to the earlier revision of this PR

• The compatibility aliases for the old header (hal_comp, hal_pin<T>, hal_dir, PyPin): the old hal.hh was removed from master and nothing in the tree uses those names.
• The HAL_* integer constants in halpp: halpp.Type/halpp.Dir replace them.

Relationship to the original pybind11 branch

Kept: the variant/map/PyPin structure and the binding shape.
Dropped: the private set_common string parser (superseded by hal_set_p with setps_common_cb), hal_mutex_guard, the get_info_signals stub, waitWritable (was non-functional).

Test evidence

Full tree builds. tests/halpp, tests/halmodule/*, tests/hal-*, tests/halrun-*, tests/halcompile and tests/build/header-sanity pass locally (23/23). The smoke suites cover component lifecycle, typed pins/params (negative and full 64-bit values), signals, linking, by-name get/set with coercion and text parsing, range errors, port pins (unlinked, sizing, resize refusal, read/peek/commit/write/clear, overfull writes), streams and error paths. tests/halpp compiles against the tree, so it skips under SYSTEM_BUILD (checked). cppcheck is clean on the new files.

[grandixximo]

This seems to be working, but I am not sure if it is the right shape we want, these are the direction I took on the issue raised in the meeting, subject to revision.

Problem	Solution
Handle is width-blind	Type lives in traits<T> (compile time) or the variant tag (runtime, from q->pp.type)
hal_link rewrites the handle slot	pin<T> stores a pointer to the slot and re-reads per access (C semantics, staleness impossible)
Throwing through query callbacks wedges the HAL mutex session-wide	Exceptions captured in the callback, rethrown after the library releases the mutex
Handle storage must be shmem	hal_malloc'd hal_refs_u slot per item (halmodule's own pattern)

@BsAtHome A bare hal_sint_t is width-blind, and wrapping foreign raw handles across widths is unsafe in the interim. But the C++ layer never selects from the handle alone. Component-created pins carry the type at construction via traits (int32_t vs int64_t are distinct C++ types, compile-time selection), and the runtime path uses q->pp.type from your query API to pick the variant alternative, the stored-type-tag multiplex you described in #4099. The only unsafe path is explicitly reinterpreting a raw handle, which can be guarded with a query lookup. So the layer works transitionally, it just gets simpler after the break.

[rene-dev]

oh, that was quick. I think this should be separate from the other big PR, as this can go in master fairly quickly. the only reason I stopped was that buster had no pybind11 package. this should also replace the current pyhal/halmodule instead of adding a third module. and, the same cpp api should be used in xhc, task and other places which are implemented in cpp.

[grandixximo]

Buster is basically not supported anymore with master, I will fix the review points tomorrow, thank you both ☺️
My bad on missing the include, getting late time for bed...

[BsAtHome]

There is another issue. The new hal_lib uses a reference counted init/exit and has separated out hal_lib_init() and hal_lib_exit(). The reason to take this out is that you do not need a component for pin/param/signal read/write or query interactions. You only need to have mapped shared memory.

Calling hal_init() to create a component (if you want to create pins) now calls the factored out hal_lib_init() and hal_exit()calls the factored out hal_lib_exit().

The python class must initialize using hal_lib_init() and terminate by calling hal_lib_exit(). If the user has not called hal_exit() on any created components, you will now get a proper error message (because the user forgot to terminate correctly).

[rene-dev]

can you rename the module to halmodule, so that the UIs and the tests that are already in place use the new bindings, and remove the old ones?

[grandixximo]

can you rename the module to halmodule, so that the UIs and the tests that are already in place use the new bindings, and remove the old ones?

Replacing halmodule is the goal, agreed, but renaming now would break things for two separate reasons.

First, a hard technical one: on master the new bindings cannot cover the existing Python surface. get_info_pins/signals/params, pin_has_writer and component_exists are used by qtvcp (qt_halobjects, hal_selectionbox) and hal_glib, and they need the query API (hal_list_*, hal_get_p) from #4247. Without it, the options are a strict subset module (GUIs break immediately) or private shmem access (exactly what this API change abolishes).

Second, the compatibility tail: hal.error exception type, KeyboardInterrupt on halcmd unload (documented in hal.py's docstring, user comps wrap their main loop in it), the exact get_info_* dict shapes (tests/halmodule.0 compares verbatim), the hal.stream object API (tests/halmodule.1), setitem coercion quirks, pin .dir/.type attributes, and the hal.py wrapper shims. Matching all of that is what makes the swap a user-visible event, and it deserves its own PR with a full parity pass, not a rename here.

So the proposal stands as: land this additive now (hal.hh + halpp alongside, nothing replaced), and do the rename + replacement as a follow-up once #4247 is in. Your old branch's consumer sweep (hal_glib, qtvcp core, raster, tests) is the starting checklist for that PR. Is this acceptable? Open for discussion...

[rene-dev]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

[BsAtHome]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

[rene-dev]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

What is halquery?

[BsAtHome]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

What is halquery?

That is the new interface to all of HAL's internals nicely wrapped up from the hal query interface without someone needing to dig into HAL's internals. Hal_lib's internals are fully isolated in #4247 and nobody should be writing code that circumvents that. Inclusion of hal_priv.h is strictly forbidden in any code that is not part of hal_lib.
Halmodule deals with components and everything else was "bolted on". You do not want one big blob of code to do it all. You want modularization, which halquery provides.

[rene-dev]

ah, found it. I would much prefer if it would be a c++ api, which can be used to automatically generate the python bindings.
most of userspace code is c++ anyway. I have maintained the c bindings in the past(pyhton2->3 migration), and do not ever want to deal with them again.

[BsAtHome]

It is a pure C version, so it does not have the same problems as C++ interfacing C-based Python ;-)

[grandixximo]

This is exactly the split #4251 (this PR) implements: the query API stays pure C in the library (Bertho's layer, no bypass possible), and the C++ API sits on top of it (hal.hh), which is where Python bindings get generated (halpybind) , no hand-written CPython code. halquery.c proves the query API is sufficient; the same functions can be exposed as pybind wrappers over hal_get_p/hal_list_* once #4247 lands. So the endgame can be: C query API in hal_lib, C++ API above it, Python bindings generated from the C++ layer, and the hand-written C modules retired on the same schedule. Happy to demonstrate the halquery-equivalent functions in halpybind as a follow-up once the base is in.

[BsAtHome]

Fine by me to get it C++ wrapped. It does not yet have priority one.

There is one thing that should be fixed in both halmodule and halquery: use IntEnum types for both type and direction. It may be necessary to have a haltype module with the enums so you it can be properly shared between the other modules. Not sure how this can be handled properly in using consistent namespace names and hierarchy.

[grandixximo]

A shared pure-Python haltype.py with HALType(IntEnum) and HALDir(IntEnum) is the cleanest shape. IntEnum members are ints, so the C boundary stays plain int and everything interops both ways; hal, halquery and the pybind module import and re-export from it, keeping hal.HAL_FLOAT working unchanged. I would keep the namespace flat (sibling of hal.py) rather than a linuxcnc.* package, which breaks every existing import hal. One pybind11 note: py::enum_ does not emit real IntEnum objects, that needs py::native_enum (pybind11 >= 3.0, trixie ships 2.13), so for now the bindings take constants from haltype.py and we flip to native_enum when the distro catches up.

[BsAtHome]

A shared pure-Python haltype.py with HALType(IntEnum) and HALDir(IntEnum) is the cleanest shape. IntEnum members are ints, so the C boundary stays plain int and everything interops both ways; hal, halquery and the pybind module import and re-export from it, keeping hal.HAL_FLOAT working unchanged.

That would be the simple way. If it works, then it should be nice. The real benefit is in halquery to have the iteration result show the right name instead of a number. There it is big value. Probably also need to add the same for other constants.

I would keep the namespace flat (sibling of hal.py) rather than a linuxcnc.* package, which breaks every existing import hal.

Global namespace naming and fixing hierarchy is a problem we need to fix in a later version. This would be a version 3.0 thing, I guess ;-)

One pybind11 note: py::enum_ does not emit real IntEnum objects, that needs py::native_enum (pybind11 >= 3.0, trixie ships 2.13), so for now the bindings take constants from haltype.py and we flip to native_enum when the distro catches up.

Ha, and so much for "C++ is easy to maintain" ;-)
The version and support problem always pops up regardless of language and library. You always end in hell once you need to support over long periods of time and different distros and distro versions. At least it is warm in hell.

[grandixximo]

Going back to draft, waiting for master new API to materialize ;-)

[grandixximo]

Branch rewritten and pushed (33012db => 63c8a05), four commits on current master:

• hal.hh replaces the deprecated upstream header with the new C++ API (typed pin/param handles, runtime-typed anypin, component, streams), built strictly on the public C API and the query API
• haltype.py: stdlib-only HALType/HALDir IntEnums plus tests/haltype.0 as drift guard against the _hal constants
• streams: linuxcnc::hal::stream and halpp.stream with _hal stream semantics; tests/halpp now runs under runtests
• hal.query registration prototype: empty _hal.query submodule, hal.py publishes it in sys.modules, tests/halquery.0 checks the four import paths

Build notes: the pybind11 bindings are skipped with a note when the headers are absent, not failed, so the tree still builds without python3-pybind11; tests/halpp self-skips in that case. hal.hh is exported to include/ alongside hal.h. tests/halpp/test.sh compiles cpp_test.cc against the tree and runs it in a live HAL session, so the native C++ side has CI coverage too.

Still draft on purpose: #4247 moves the same ground (hal.hh, the type system, HAL isolation), so this waits for that to land before the query bindings follow as hal.query.

[grandixximo]

Branch reworked after feedback Bertho sent me in a private mail (63c8a05 => 7beaaf3). The haltype.py commit is gone, replaced by:

• src/hal/halenum.hh builds the two IntEnum classes from the hal.h constants through the plain Python C API: one native source of truth, nothing to drift, and any extension module can instantiate the same classes, pybind11 or not.
• _hal registers the shared classes as _hal.type/hal.dir. hal.py serves hal.type.REAL, hal.dir.IN etc. (HAL* spellings are aliases), plus the HALType/HALDir class names so pickle round-trips.
• halpp drops its private py::enum_ copies and casts hal_type_t/hal::dir arguments and results through the shared classes, so tags print with their names and compare equal to the plain constants, which stay exported for compatibility.
• Canonical members are the platform-stable spellings (BOOL, REAL, S32, U32, PORT, S64, U64 and IN..RW); SINT/UINT follow the platform width and are aliases.

pybind11 note: real enum.IntEnum via py::native_enum needs pybind11 >= 3.0 and trixie ships 2.13, so construction goes through the enum module's functional API. The classes are genuine IntEnums either way, so moving to py::native_enum later (TODO in the header) is invisible to user code.

Query results stay plain dicts as discussed; the future hal.query bindings tag type/dir with these classes, no per-function wrappers. tests/haltype.0 is now a sanity suite for the classes, tests/halpp checks halpp shares them. All HAL suites pass.

[grandixximo]

@BsAtHome one API question from adding port pins: hal_pin_new(3) says hal_get_p(3) cannot be called on a HAL_PORT pin, but the library returns success with value 0 for it, and hal.query reports the pin's buffer size. hal.hh's get_value() currently relies on hal_get_p() succeeding for port pins (a callback fills in the buffer size). Which behavior is intended? If get_p on a port pin should fail, I'll switch get_value() to another route and the man page stays as is; otherwise the man page needs a line.

[BsAtHome]

Tricky question... The port pins are only end-point nodes. They do not have any real meaning besides read/write. The size of the buffer should ideally be retrieved using hal_get_s because the port pins can never stand alone and need a binding signal. The hal_set_s on that signal sets the buffer size, which makes it logical to retrieve the size using hal_get_s.

The fact that the hal_get_p on the pins also return the size is due to legacy IIRC. There is no "harm" in that you can call hal_get_p on a port pin. However, the question in the room is whether hal_get_p on an output pin really should be retrieving one byte of the queue and hal_set_p on the input pin writes one byte into the queue. The queue's size is managed by the signal.

[grandixximo]

Thanks, that settles it. I'll keep the pin side out of it: in hal.hh a port pin gets no scalar value (get_value() on a port pin and the runtime-typed get() raise), the buffer size comes from the signal via hal_get_s(), and port::size() stays as the explicit call on the handle. That leaves hal_get_p()/hal_set_p() on port pins free for whatever you decide, including byte I/O.

Small correction for the record: the library's hal_get_p() on a port pin returns 0, not the size (get_common() is called with getport=0 for pins); only hal.query computes the size for pins itself.