Fix function pointer subtyping

[kimjune01]

Relates function pointer signatures using function subtyping instead of asserting structural equality. This handles alpha-renamed bound regions in #1718 and default refinement differences in #1185.

This follows the approach discussed in the Zulip thread on moving function subtyping into flux-infer.

Adds regression coverage for generic inputs and outputs, nested function pointers, higher-ranked lifetimes, multiple binders, mutable references, and unsafe function pointers. All new cases ICE on main and pass with this change.

Fixes #1718.
Fixes #1185.

[ranjitjhala]

Thanks @kimjune01 !! I'll take a look first thing tomorrow!

[nilehmann]

This is the right approach to solve issues #1718 and #1184, but I'm a bit concerned that this duplicates the code for function subtyping.

It's a bit tricky to abstract this because we have to support subtyping with two levels of binders (EarlyBinder and Binder), and binders on the right have to be peeled out before binders on the left. I haven't given it a ton of cycles, but not having a convincing way to avoid code duplication is the main reason I haven't implemented function pointer subtyping.

[ranjitjhala]

In the spirit of "not letting the perfect be the enemy of the good" I'd rather we merge and try to coalesce later.

[ranjitjhala]

Plus Having two concrete chunks that we want to abstract/share likely makes it easier to do the refactor..

[nilehmann]

Agree, but historically I've been the one doing these cleanups, and since I don't have much time to do it anymore, merging this PR means the code is going to stay like that unless someone else takes on the burden.

[ranjitjhala]

Sure thing, I'll do it. We want to refactor the two subtyping impls to be the same, yes?

…

On Mon, Aug 31, 2026 at 1:54 PM Nico Lehmann ***@***.***> wrote: *nilehmann* left a comment (flux-rs/flux#1737) <https://urldefense.com/v3/__https://github.com/flux-rs/flux/pull/1737*issuecomment-5484421691__;Iw!!Mih3wA!A6F7NEqcigbTouiFaA9jNFtlFW5kkrjV7IfeKHBTpK3hnvsz-9-cOR3iFayVwFZ1cnHKhuU7JS1wyLIWzAkIWqQM$> Agree, but historically I've been the one doing these cleanups, and since I don't have much time to do it anymore, merging this PR means the code is going to stay like that unless someone else takes on the burden. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/flux-rs/flux/pull/1737?email_source=notifications&email_token=AAMS4OHLMGW765OGSGMWC3T5MXQYTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBYGQ2DEMJWHEY2M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM*issuecomment-5484421691__;Iw!!Mih3wA!A6F7NEqcigbTouiFaA9jNFtlFW5kkrjV7IfeKHBTpK3hnvsz-9-cOR3iFayVwFZ1cnHKhuU7JS1wyLIWzOeWY5DL$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAMS4OHBYE4R25WGG6O5G5T5MXQYTAVCNFSNUABFKJSXA33TNF2G64TZHMZTIMBQGQ4DQMJTHNEXG43VMU5TKMRXGQZDGMJTGUZKC5QC__;!!Mih3wA!A6F7NEqcigbTouiFaA9jNFtlFW5kkrjV7IfeKHBTpK3hnvsz-9-cOR3iFayVwFZ1cnHKhuU7JS1wyLIWzHTi8kOl$> . You are receiving this because you commented.Message ID: ***@***.***>

-- - Ranjit.

[nilehmann]

We want to refactor the two subtyping impls to be the same, yes?

Ideally, but it's tricky because we need to move things around, since check_fn_subtyping depends on stuff defined in refineck.rs. That function also uses a different reason for the generated constraints. But also, check_fn_subtyping does extra stuff like unfolding pointers, which may be overkill for function pointer subtyping. Unclear...

[ranjitjhala]

Ah ok, so then we'd want to

1. Pass the extra reason as a parameter?
2. Just do the unfolding and then, if it turns out to be unnecessary, somehow "gate" it?

[nilehmann]

Yes, but if we end up gating parts of the implementation, I don't know if that makes the overall code better.

[ranjitjhala]

What other option is there (other than not supporting fnptr subtyping, which seems worse?)

…

On Mon, Aug 31, 2026 at 2:24 PM Nico Lehmann ***@***.***> wrote: *nilehmann* left a comment (flux-rs/flux#1737) <https://urldefense.com/v3/__https://github.com/flux-rs/flux/pull/1737*issuecomment-5484811666__;Iw!!Mih3wA!B7b4Oov6rPRW66l6tTialJ9EiiV4l_61fL3hVmsc5TZ0wnObZzZXN0JAqZQAqSuwoH4UarpXemdXcodtl4GWIwHr$> Yes, but if we end up gating parts of the implementation, I don't know if that makes the overall code better. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/flux-rs/flux/pull/1737?email_source=notifications&email_token=AAMS4OGCTBN6LZEBUCARC3L5MXUJXA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBYGQ4DCMJWGY3KM4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM*issuecomment-5484811666__;Iw!!Mih3wA!B7b4Oov6rPRW66l6tTialJ9EiiV4l_61fL3hVmsc5TZ0wnObZzZXN0JAqZQAqSuwoH4UarpXemdXcodtl9KlQ3sd$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAMS4OEVMFBHKPYS4QUQ2AT5MXUJXAVCNFSNUABFKJSXA33TNF2G64TZHMZTIMBQGQ4DQMJTHNEXG43VMU5TKMRXGQZDGMJTGUZKC5QC__;!!Mih3wA!B7b4Oov6rPRW66l6tTialJ9EiiV4l_61fL3hVmsc5TZ0wnObZzZXN0JAqZQAqSuwoH4UarpXemdXcodtl-IyN8CT$> . You are receiving this because you commented.Message ID: ***@***.***>

-- - Ranjit.