Handling TLS Hostname

[Malte-Neu]

I have a problem with calling the OpenSSL function SSL_set_tlsext_host_name in the unit IdSSLOpenSSL.pas. I want to connect to a server that I host, which has a self-signed certificate. The names in the certificate are not all correctly specified for the names the server can have. This results in an SSL handshake error during the connection setup.
When I comment out the line that calls SSL_set_tlsext_host_name(fSSL, fHostName), it works with my server.

How can I best solve this issue so that the invocation of the parameter can be controlled explicitly?

[rlebeau]

The TLS SNI extension is required by a lot of TLS 1.2+ servers, so removing the call to SSL_set_tlsext_host_name() is not a good idea. Indy passes in the host name that it is actually trying to connect to (which is the whole purpose of SNI), so if that does not match your certificate and you need to validate the certificate yourself (ie you can't get a better certificate for your server), then use the TIdSSLIOHandlerSocketOpenSSL.OnVerifyPeer event, which gives you access to the actual server certificate and lets you return whether you accept it or not.

[Malte-Neu]

Unfortunately, that doesn’t work. In my case, an error is already raised after SSL_connect. VerifyPeer isn’t even called.

[rlebeau]

Can you be more specific? What error, exactly?

The OnVerifyPeer event should be called while SSL_connect() is running, unless there is another problem with your TLS handshake before certificate validation occurs.

In any case, without OnVerifyPeer, there is no good solution. The host name that is passed to SSL_set_tlsext_host_name() is based on the TIdSSLIOHandlerSocketBase.URIToCheck, TIdSSLIOHandlerSocketBase.TransparentProxy.Host, and TIdSSLIOHandlerSocketOpenSSL.Host properties (in that order). Depending on which TCP component you are using, those values may not be changable in user code. TIdSSLIOHandlerSocketBase.URIToCheck would be your best bet, unless you are using TIdHTTP, which sets its own URIToCheck value.

Can you provide more context about your situation?

[Malte-Neu]

In my test I forgot to set VerifyMode, which is why verifyPeer was not called.
I’ve corrected that. I’m still getting an error — here’s the call flow:

SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2]
SSLOptions.VerifyMode= [sslvrfPeer]
OnVerifyPeer := onVerifyPeerCallback

After Thttp.post call:

• TIdSSLSocket.Connect is called
• SSL_set_tlsext_host_name is set
• SSL_connect is called
• VerifyCallback is triggered
  • DoVerifyPeer is triggered --> result true
  • VerifyCallback returns 1
• ssl_connect returns -1
• EIdOpenSSLAPISSLError.RaiseExceptionCode is raised with AErrCode=1, aRetCode=-1, AMsg='Error connecting with SSL.'
• EIdOpenSSLAPICryptoError.RaiseExceptionCode is raised with AErr=336703553 AMsg="Error connecting with SSL" LMsg = "Error connecting with SSL.'#$D#$A'error:1411B041:SSL routines:ssl3_get_new_session_ticket:malloc failure"

If I skip the call to SSL_set_tlsext_host_name, everything works as expected.

[rlebeau]

error:1411B041:SSL routines:ssl3_get_new_session_ticket:malloc failure - that is not a certificate error. Something else is going wrong during the TLS handshake. When I search online for that error message, I see it reported a few times going back at least 5 years. So, I suspect either you are using an outdated version of OpenSSL, or you have found a new bug in it. Either way, not an Indy issue.