Drop legacy prime-generation callback in setup_certificate

[hsbt]

The BN_GENCB block only printed progress dots in verbose mode, but it re-enters Ruby from inside OpenSSL's keygen.

Under ZJIT that re-entry intermittently aborts generation with EVP_PKEY_keygen. The callback form is obsolete in OpenSSL 3.x, so just generate the key without it.

[rhenium]

Do you have the crash logs available? I suspect there may be a bug somewhere. The callback doesn't use any obsolete OpenSSL C APIs, and as far as I know, ruby/openssl isn't doing anything unusual here.

That said, the change makes sense to me. Generating a 2048-bit RSA key shouldn't take long enough for progress reporting to be useful.

[hsbt]

It happened with https://github.com/ruby/ruby/actions/runs/27654847453/job/81786813949

I'm working to fix flaky tests instead of triggering re-run button now. I'm not sure that's caused by ZJIT or something. It's good to fix that at root cause.

[rhenium]

Thanks for the link. I'd have expected to see additional context from OpenSSL's ERR_raise() in the error message, so this is a bit strange.

Setting up libssl3:amd64 (3.0.2-0ubuntu1.25)
[...]
===============================================================================
Error: test_07_break_18(DRbTests::TestDRbSSLAry): OpenSSL::PKey::PKeyError: EVP_PKEY_keygen
/home/runner/work/ruby/ruby/build/.ext/common/openssl/pkey.rb:382:in 'OpenSSL::PKey.generate_key'
/home/runner/work/ruby/ruby/build/.ext/common/openssl/pkey.rb:382:in 'OpenSSL::PKey::RSA.generate'
/home/runner/work/ruby/ruby/build/.ext/common/openssl/pkey.rb:392:in 'OpenSSL::PKey::RSA.new'
/home/runner/work/ruby/ruby/src/gems/src/drb/lib/drb/ssl.rb:173:in 'DRb::DRbSSLSocket::SSLConfig#setup_certificate'
/home/runner/work/ruby/ruby/src/gems/src/drb/lib/drb/ssl.rb:290:in 'DRb::DRbSSLSocket.open_server'
/home/runner/work/ruby/ruby/src/gems/src/drb/lib/drb/drb.rb:795:in 'block in DRb::DRbProtocol.open_server'
/home/runner/work/ruby/ruby/src/gems/src/drb/lib/drb/drb.rb:793:in 'Array#each'
/home/runner/work/ruby/ruby/src/gems/src/drb/lib/drb/drb.rb:793:in 'DRb::DRbProtocol.open_server'
/home/runner/work/ruby/ruby/src/gems/src/drb/lib/drb/drb.rb:1491:in 'DRb::DRbServer#initialize'
/home/runner/work/ruby/ruby/src/gems/src/drb/test/drb/test_drbssl.rb:41:in 'Class#new'
/home/runner/work/ruby/ruby/src/gems/src/drb/test/drb/test_drbssl.rb:41:in 'DRbTests::DRbSSLService#start'
/home/runner/work/ruby/ruby/src/gems/src/drb/test/drb/drbtest.rb:24:in 'DRbTests::DRbService#initialize'
/home/runner/work/ruby/ruby/src/gems/src/drb/test/drb/test_drbssl.rb:75:in 'Class#new'
/home/runner/work/ruby/ruby/src/gems/src/drb/test/drb/test_drbssl.rb:75:in 'DRbTests::TestDRbSSLAry#setup'
===============================================================================