Java Mutual TLS with Apache HTTP Client and MockServer

Recently I had to implement a feature where we wanted to add user-configurable client authentication for an HTTPS connection between two services. The user was to specify the certificate as well as the corresponding credentials in the UI and they were later used in the background to make the REST call. Turns out, testing that properly became way more effort than I initially thought. I’ll quickly go over the general problem to be solved, give you a rough overview of the implementation to be tested, and then describe my struggle to get the test done.

The Problem

The initial requirement is as follows: two of our services are running on different servers and are exposed as different URLs. These two services can be connected to each other by an administrator (i.e. the user) via the UI. Apart from other transport mechanisms we also support the use of HTTPS for easy communication between the two services. However, especially in customer environments, firewalls often prohibit a direct, unauthenticated connection between servers. As such we had to implement the use of client certificates for TLS authentication.

During implementation, unit tests of course cover the bits where it’s easily possible to isolate the functionality. Nevertheless, before providing a pull request for the feature, I wanted to make sure that everything definitely works as expected and have a proper local setup for testing. That’s where the initial estimation then got blown up a tiny bit.

Implementation to be tested

To give you a basic overview of what we are talking about, let’s take a look at the included components. We have a HttpClientFactory that uses a CertificateProvider to load the stored client certificate from different sources (might be a classpath resource or a file on disk). The HttpClientFactory then creates our HTTP client which makes authenticated requests to the server which then verifies the given certificate.

The signature of the CertificateProvider is quite simple – given below without JavaDoc for simplicity.

public interface CertificateProvider {

    @Nonnull
    Optional<KeyManager[]> loadCertificate(@Nonnull String certificateName,
                                           @Nonnull String certificatePassword);

}

Consuming these providers – there can be multiple providers registered at runtime – is then the task of the HttpClientFactory. The following code snippet shows the basic structure of the HttpClientFactory and its important methods.

public class HttpClientFactory {

    // ...

    @Nonnull
    CloseableHttpClient create() {
        final HttpClientBuilder builder = HttpClients.custom()
                .setDefaultHeaders(ImmutableList.of(
                        new BasicHeader(HttpsTransport.HEADER_SENDER_TOKEN, senderToken)
                ));

        if (!Strings.isNullOrEmpty(certificateName)) {
            final Optional<SSLContext> sslContext;
            try {
                sslContext = createSslContext();
            } catch (NoSuchAlgorithmException | KeyManagementException e) {
                log.error("Failed to create SSLContext", e);
                throw new HttpClientFactoryException("Failed to create SSLContext", e);
            }
            sslContext.ifPresent(builder::setSSLContext);
        }

        return builder.build();
    }

    // ...
}

As you can see the code basically just creates a HttpClientBuilder and populates it with a default header before trying to create the SSLContext. Always a good thing to remember when calling SSLContext.init(…) is that it will fall back to the default providers for both key and trust managers.

So that’s basically all that we needed to get the basics to load certificates by their name and potential credentials. Got the underlying functionality covered with unit tests – now let’s go for a test run… but against what?! No custom certificate in sight, no server setup available that requires client authentication…

Test Driving

In order to be able to test it fully we’ll need the following:

  1. Client Certificate: we explicitly want to test client authentication, so of course we’ll need a proper client certificate for that.
  2. PKCS12 Key Store: since we’re using Java the easiest way to load our certificate is by providing a PFX file that contains the certificate as a KeyStore.
  3. Server: of course, our communication partner with SSL and TLS enabled and enforcing client authentication.

Creating a Client Certificate

The easiest way to create a client certificate is by using openssl. DigitalOcean has a nice explanation of some OpenSSL Essentials.

In order to create my own self-signed certificate I used the following command (attention: do not use this to create keys for anything else than testing):

openssl req -newkey rsa:4096 -nodes -keyout test-cert.key -x509 -days 7300 -out test-cert.crt

This will ask a few questions and create a new 4096 bit RSA key (-newkey rsa:4096) that is not password protected (-nodes – read: no DES) and self-sign it with a validity of 20 years (-days 7300). The signed certificate will be stored as test-cert.crt (-out test-cert.crt).

Generate PKCS12-based Key Store

Now let’s create a PKCS12-based key store from that to use in our Java application:

openssl pkcs12 -export -out test-cert.pfx -inkey test-cert.key -in test-cert.crt

The above command generates the file test-cert.pfx – our PKCS12-based key store – by reading both our key and signed certificate from the previous command. You will be prompted for a password to encrypt the file with (remember this in order to load it from Java properly).

Setting up the Server

Since I didn’t want to setup a full NGINX server and generate another certificate to use for the server etc. I looked around for something to be used quickly for tests. Turns out MockServer is a great tool for easily testing these things. They provide different ways of using it but for the sake of simplicity I opted to go with their Docker image to have a “real” external server to talk to.

Spinning up MockServer as Docker Image

Well, there’s nothing easier than starting a Docker image right?! Let’s go and follow their official documentation:

$ docker pull mockserver/mockserver
$ docker run -d --rm -P mockserver/mockserver

Two things to note for the run. First, we use the --rm flag that automatically removes the container once it has been stopped. Second, -P will automatically bind all ports exposed by the container to dynamically allocated ports of the host machine. That’s it and it’s up and running.

Untrusted Server Certificates and Free Features

NOTE: MockServer uses their own CA to generate the server-side certificates for SSL, of course. This CA is not trusted by default by the OS.

Yup, of course I missed that. As they state in their documentation, you have to explicitly trust their Mock CA in order to be able to connect via SSL. For our code from above of the HttpClientBuilder that means we need to create the SSLContext with an explicitly provided TrustManager that contains the Mock CA. So after adding this “unplanned feature”, adding the unit tests, integrating it everywhere – … how do we get the “thing” that we can load into Java?

In order to create a TrustManager that contains server-side certificates we need to generate a TrustStore with the Mock CA inside it. Some quick googling turned up a really old looking Oracle documentation page. In the end it’s just a one-liner to create a Java TrustStore from a PEM certificate:

keytool -import -file MockServerCA.pem -alias MockServerCA -keystore test-truststore.jks

It will prompt for a password to be used (again – remember to specify that when loading the store in Java).

Enabling Mutual TLS

Enabling Mutual TLS for MockServer is quite simple as once more outlined in their documentation. When running it as a Docker container you need to set the MOCKSERVER_TLS_MUTUAL_AUTHENTICATION_REQUIRED environment variable to true. The overall Docker command thus is as follows:

docker run -d --rm -e MOCKSERVER_TLS_MUTUAL_AUTHENTICATION_REQUIRED=true -P mockserver/mockserver

Low and behold – that won’t work as I soon figured out. Of course we’ll have to provide our certificate chain to be used for validation of the provided client certificates. The documentation for the required configuration flags is just the next section in the above linked manual. A quick copy and paste of the environment variable and same error… nothing was provided. Turns out I didn’t pay attention and there was an error in the documentation that contained the wrong environment variable MOCKSERVER_FORWARD_PROXY_TLS_X509_CERTIFICATE_CHAIN! 🙁 After quickly checking the source code, the correct one to be used is: MOCKSERVER_TLS_MUTUAL_AUTHENTICATION_CERTIFICATE_CHAIN.

Okay, got that covered – but how to actually get those certificates in there… and what is the root for the chain? I played myself there a little until I finally figured out that just providing the self-signed certificate that we initially generated (test-cert.crt) as the certificate chain is enough (spare yourself the hour to figure that out…). Since I haven’t done that much Docker in the past – probably as one of the only ones – I even struggled with mounting my local file into that Docker container 😀

I stumbled again and again over the issue that my mounted file somehow was detected as a directory when mounted into the container. If you ever encounter that – use an absolute path when specifying the mount:

# DONT:
docker run ... -v ./test-cert.crt:/opt/test-cert.crt ...
# DO:
docker run ... -v $(pwd)/test-cert.crt:/opt/test-cert.crt ...

FINALLY it worked – I could start my Docker container, had mTLS enabled and it used the correct certificate to validate what the client provided. All started with a simple command:

docker run -d --rm \
    -e MOCKSERVER_TLS_MUTUAL_AUTHENTICATION_REQUIRED=true \
    -e MOCKSERVER_TLS_MUTUAL_AUTHENTICATION_CERTIFICATE_CHAIN=/opt/test-cert.crt \
    -v $(pwd)/test-cert.crt:/opt/test-cert.crt \
    -P mockserver/mockserver

And turns out that the original implementation I wanted to test – worked like a charm!

Conclusion

If I had known what I had been doing it would have been a lot faster. As always 😀 Nevertheless, to me using MockServer was still quicker than setting up anything else and now I can easily use it for further tests of these authentication problems. MockServer is capable of doing a lot more things, of course, but I didn’t need those for my quick and dirty test here.

I hope that my post might save someone else some time in figuring things out, let me know if there’s anything else you want to know!

Leave a Reply

Your email address will not be published. Required fields are marked *