Tuesday, February 3, 2009

Using HttpClient with Unique SSL Requirements

In looking at the HtmlUnit library as a means to do headless front end testing as part of our build process.  I quickly ran into to very large problems that I had to overcome.

Firstly, the default HttpClient behaviour when dealing with un-trusted server certificates is to throw an exception when connecting.  Since our development server doesn’t have a trusted certificate and its a waste to buy one for development, I needed a way to be able to get my build server to trust my homemade server certificate.  Luckily, I can tell the webclient to use insecure ssl, but I thought perhaps that this might be a little shot-gunnish.

Secondly, many of our modules require a client side certificate to be able to connect from the web browser…another behaviour which isn’t the default from the HttpClient.

After a little hunting, I found the SSL guide on the HttpClient project site…(image finding it there of all places).  I found that the explanation wasn’t so good on the page except it links to the source file of an AuthSSLProtocolSocketFactory.java that made things quite a bit clearer.  Allow me to explain.

HtmlUnit uses the commons HttpClient to download pages.  The client uses the Protocol class when making connections to servers.  In order for me to be able to change the behavior of a connection, the easiest way to do this is to supply the client with an alternate protocol class to use when connecting with a server…this is a lot easier to do than you think.

I basically replicated the AuthSSLProtocolSocketFactory class mentioned earlier and then I used it to construct an instance of a Protocol class.  Then I registered my new Protocol to handle “https” requests.  I had a bit of trouble using the java keystore for my client certificate, so I had to add a small change to allow me to handle PKCS12 keystore instead, which works just fine.

Once you have the socket factory created, then all you do is create a ‘new’ protocol to handle https and register it with the framework:

  1: final Protocol authhttps = new Protocol(
  2:     "https", 
  3:     new AuthSSLProtocolSocketFactory(
  4:         keystoreUrl,
  5:         "password",
  6:         truststoreUrl,
  7:         "password"), 
  8:     443);
 10: Protocol.registerProtocol("https", authhttps);

Once this is done, you simply code your unit test.  See the ‘Getting Started’ link at http://htmlunit.sourceforge.net.

HttpClient - HttpClient SSL Guide


Mani Jayaram said...

The new Protocol(...) that you use is deprecated. Right? Do you know any other alternative approach?

Steve said...

I like the way you went about implementing this with ssl certs, I have to do this soon and i've bookmarked this so i can come back and read up on what you did when i need to do the job.

Rolf said...

> The new Protocol(...) that you use is deprecated. Right? Do you know any other alternative approach?

I think this is not a problem at all, since the Protocol class has two constructors. One is deprecated, the other is not.

The deprecated one looks like this:

public Protocol(String scheme, SecureProtocolSocketFactory factory, int defaultPort) {

this(scheme, ((ProtocolSocketFactory) (factory)), defaultPort);


So it simply calls the other constructor that is not deprecated.

Stefan said...

Same problem here: Client Certificates and HtmlUnit. In the latest version of HtmlUnit, the trick works slightly different... see http://stefan.ploing.de/2012-04-10-htmlunit-mit-https-und-client-zertifikaten
(written in German, but have a look at the code snippets, they should be pretty self-explanatory)