Describe the bug
Houdini 21: resolver's statically linked OpenSSL gets hijacked by Houdini's own bundled libssl, breaking every TLS request
Related to the crash I filed over on ayon-cpp-api (the null deref in serialCorePost ynput/ayon-cpp-api#52) but this is the actual root cause underneath it, and it's specific to how the resolver is linked, not the AyonApi code itself.
Every resolve in Houdini 21 was failing with SSLServerVerification, 100% of the time, regardless of what we tried on the network side. We ruled out the obvious stuff first: the CA bundle and cert both check out fine with curl and openssl s_client against the same host, IPv6 wasn't it, no proxy involved. Eventually broke out gdb and set a breakpoint on SSL_get_verify_result, and this is where it gets interesting:
Thread 1 hit Breakpoint, SSL_get_verify_result ()
from .../hfs/dsolib/libssl.so.1.1
#1 httplib::ClientImpl::send_(...) at ayonUsdResolver.so
So the call originates from our resolver code, but it's actually executing inside Houdini's own bundled OpenSSL 1.1, not the OpenSSL 3 the resolver statically links. Checked with nm and sure enough:
$ nm -D ayonUsdResolver.so | grep SSL_get_verify_result
0000000000244560 T SSL_get_verify_result
That symbol is exported globally from the .so instead of being hidden. Houdini loads its own libssl.so.1.1 well before the resolver plugin gets loaded, and since our OpenSSL 3 symbols aren't hidden, the dynamic linker just resolves any call to that symbol name using whichever definition loaded first for the whole process, which is Houdini's older one. Different major version, incompatible internal struct layouts, so every SSL call the resolver makes ends up executing against the wrong version's code. Explains why it fails 100% of the time regardless of the actual network/cert state, since it's not really a network problem at all.
This doesn't affect Maya, since it doesn't have this OpenSSL version clash.
Describe the bug
Houdini 21: resolver's statically linked OpenSSL gets hijacked by Houdini's own bundled libssl, breaking every TLS request
Related to the crash I filed over on ayon-cpp-api (the null deref in serialCorePost ynput/ayon-cpp-api#52) but this is the actual root cause underneath it, and it's specific to how the resolver is linked, not the AyonApi code itself.
Every resolve in Houdini 21 was failing with
SSLServerVerification, 100% of the time, regardless of what we tried on the network side. We ruled out the obvious stuff first: the CA bundle and cert both check out fine with curl and openssl s_client against the same host, IPv6 wasn't it, no proxy involved. Eventually broke out gdb and set a breakpoint onSSL_get_verify_result, and this is where it gets interesting:So the call originates from our resolver code, but it's actually executing inside Houdini's own bundled OpenSSL 1.1, not the OpenSSL 3 the resolver statically links. Checked with nm and sure enough:
That symbol is exported globally from the .so instead of being hidden. Houdini loads its own libssl.so.1.1 well before the resolver plugin gets loaded, and since our OpenSSL 3 symbols aren't hidden, the dynamic linker just resolves any call to that symbol name using whichever definition loaded first for the whole process, which is Houdini's older one. Different major version, incompatible internal struct layouts, so every SSL call the resolver makes ends up executing against the wrong version's code. Explains why it fails 100% of the time regardless of the actual network/cert state, since it's not really a network problem at all.
This doesn't affect Maya, since it doesn't have this OpenSSL version clash.