If you upgrade JDKs, don't forget the security portions

Ran into this today at work - the goal was to upgrade the JDK / JVM on our servers to a newer version with bugfixes for some issues the servers have.

We merrily got the new JDK and slowly promoted it out from dev to qa to staging and then to two servers in prod.

Naturally, there was a problem that wasn't discovered until we got to prod, but on testing other environments was present everywhere. What was the problem? there was a security exception preventing the server from accessing an https URL now over SSL.

Huh? Well, Java is very very strict about SSL / HTTPS connections, so if you try to use a Java program to connect to a server over SSL using an HTTPS URL, and that server doesn't have an SSL certificate that is signed by a CA in your JDK cacerts file, you'll have problems. The normal solution for this is to use keytool to add the CA to the JDK's cacerts file, or to make a keystore using keytool with the certificate in it.

Additionally, it is possible that the java.policy file itself has been changed to grant (or revoke) certain permissions from the JDK.

Both of these files live inside the JDK, so if you upgrade the JDK but forget to copy those files over (like we did) you're going to have some problems.

Finally, for very very old JDKs (1.3.1 and older, pity me) note that you will have other issues if you don't download the JCE and JSSE since without those, the old JDK can't do strong crypto and you'll have trouble too.

In order to discover whether you need to care about this or not, you should like at JAVA_HOME/jre/lib/ext (the JSSE and JCE files go in there) and JAVA_HOME/jre/lib/security/ to see if any files have been updated. Or you can use something like a "find . -mtime " to see what's not stock.

Good luck...

JDK GC tuning for app servers

A quickie post here.

Lets say you get called in to review a server that's getting OutOfMemoryErrors

This is a bad thing, obviously! How do you fix it?

Well, it depends on the root cause, first of all you have to figure out what the problem is.

On old JVMs you turn on -verbose:gc and that's it, on newer ones you can turn on more details, some timestamps etc - all things that help you, but that's not the point of the post.

The point of this post is to send you over to get GCViewer from tagtraum at http://www.tagtraum.com/gcviewer-download.html

It's really the best option for visualizing what's going on.

It's not perfect though, and a trick I have to use nearly every time I'm visualizing the GC behavior in an app server via GCViewer, I have to filter the log file so it has nothing but GC lines:

cat $log | grep '\[' | grep GC > clean-gc.log

Then you fire GCViewer up and see what you've got

usually you get a line going "up and to the right" which is great for finance, but bad for GC signatures. That means you have a memory leak, or you have misconfigured your server to hold on to too many objects in caches, or to accept too many sessions that themselves have too many objects.

Sometimes you realize that you never run out of heap, but if you still get OOM errors, you are either running out of PermGen space or in rare cases are running out of operating system file handles by spawning too many threads (that error gets mapped to OOM, believe it or not)

Without GCViewer, it's difficult to clearly articulate the problem though, and also hard to see trends. Combined with how fast it is to use, I end up reaching for this particular utility every time.

Cheers