Starting with Java™ SE Development Kit 11.0.9 (JDK 11.0.9), Oracle has introuduced a new feature:

https://www.oracle.com/java/technologies/javase/11all-relnotes.html#JDK-8245417

 Improve Certificate Chain Handling
A new system property, jdk.tls.maxHandshakeMessageSize, has been added to set the maximum allowed size for the handshake message in TLS/DTLS handshaking. The default value of the system property is 32768 (32 kilobytes).

This might cause troubles when a client does a SSL handshake with the remote the server using JDK 11.0.9 above. The error message would be something like below:

javax.net.ssl.SSLProtocolException: The size of the handshake message (39270) exceeds the maximum allowed size (32768)

To workaround this issue, the client needs to add the property mentioned above, for example: to allow a value of 128k, add the following property to the client startup options and restart the client.

-Djdk.tls.maxHandshakeMessageSize=131072

If you cannot modify the client (java application) startup option, you can try the OS environment variable (assuming on a Linux host):

export JAVA_TOOL_OPTIONS="-Djdk.tls.maxHandshakeMessageSize=131072"

One way to verify this is to use “openssl” with the option “-msg” to check the SSL handshake process:

openssl s_client -connect server.sample.com:port -msg > output.txt

[service@joetest~]$ grep -i handshake output.txt
>>> TLS 1.3, Handshake [length 0122], ClientHello
<<< TLS 1.3, Handshake [length 0055], ServerHello
<<< TLS 1.2, Handshake [length 03a0], Certificate
<<< TLS 1.2, Handshake [length 012c], ServerKeyExchange
<<< TLS 1.2, Handshake [length cffe], CertificateRequest
<<< TLS 1.2, Handshake [length 0004], ServerHelloDone
>>> TLS 1.2, Handshake [length 0007], Certificate
>>> TLS 1.2, Handshake [length 0025], ClientKeyExchange
>>> TLS 1.2, Handshake [length 0010], Finished
<<< TLS 1.2, Handshake [length 0010], Finished
SSL handshake has read 56461 bytes and written 424 bytes

The other options like “-prexit” or “-debug” might be helpful as well. In the example above, the handshake of “CertificateRequest” has the length affe which is 53246 in decimal which exceeds the default size 32768.

This could happen when the server has a large Trusted Root Certificate Store with hundreds of trusted root certificates. These entries are negotiated with a client in case there is a need for the client side authentication and they tell the client from which root authorities the certificate presented by the client can be trusted. When using openssl, you will see a big list of certificates returned under the section of “Acceptable client certificate CA names”.

openssl s_client -connect server.sample.com:port

If a server is a Windows server, you can see that list from “Control Panel” -> “Manage computer certificates” -> “Certificates – Local Computer” -> “Trusted Root Certification Authorities” -> “Certificates”

• Uncategorized