Monday, November 5, 2007

Bounce apache

I don't bounce apache enough to remember the command, so here it is:

/usr/local/apache2/bin/./apachectl restart

Tuesday, October 9, 2007

Broken pipes with Tomcat and DBCP connection pooling

Getting "broken pipe" errors showing up in your tomcat logs? Happening every 8 hours or so?I had this problem with MySQL 4.1.21 and I've managed to fix it.

(NB. if you are running 5.0.45 MySQL then you'll get a "com.mysql.jdbc.CommunicationsException: Communications link failure" error which is a symptom of the same problem)

A quick search of the web revealed that lots of people are getting these and there are LOTS of suggested solutions. The most common explanation is that your MySQL connections are getting terminated by MySQL before they have been released by the connection pool. Then the next time you try to get a connection from the pool, you'll get a dead connection which will give a "broken pipe" error.

The first step is to try and reproduce it, without waiting around for 8 hours!

I checked the MySQL timeout setting via "mysql> select @@global.wait_timeout". Mine was set to 28,800 which is, not suprisingly, 8 hours. I then reduced this to one minute by:
  • stopping MySQL ("/usr/local/bin/mysqladmin shutdown"),
  • modifying the /etc/my.inf file and adding a "wait_timeout=60" under the "[mysqld]" heading and saving,
  • restarting MySQL ("/usr/local/mysql/mysqld_safe")
Don't worry if your session timeout hasn't changed ("mysql> select @@session.wait_timeout") - it doesn't seem to affect your terminal session. Your global session will have changed, and new connections will use the global setting.

Next I started my application and saw my connections appear (via "mysql> show processlist;") and then disappear at the 60 second mark. Trying to use the app again resulted in the broken pipe error HOORAY! Retrying the app resulted in one broken pipe error per connection in the pool, then it all works OK.

Now the fix.

Appending ?autoconnect=true to the JDBC driver connection string didn't work.

Killing the idle session before MySQL did was the solution. With the MySQL timeout still set at 60 seconds, I decided to evict idle session after 30 seconds by adding the following to the data source:

minEvictableIdleTimeMillis=30000 (evict after 30 seconds inactivty)
timeBetweenEvictionRunsMillis=10000 (run the evicter every 10 seconds)
numTestsPerEviction=-1 (check every connection)

Now, running the same test above, the connections disappear from the list after about 40 seconds. And retrying the ap just creates a new connection. HOORAY! Fixed!! If you are not happy with your connection pool getting wiped out then you can set "minIdle=n" against the data source. When the items from the pool are evicted, new ones are crested to keep the pool full.

In the end I went for a 6 hour idle eviction, checking very 30 mins.

Friday, September 21, 2007

Testing an Http POST from the command line

Got a REST web service update using an Http POST that you want to test? If so, run this command:

curl --header "Content-Type: text/xml" --data '' "" –i

And get this result:

HTTP/1.1 200 OK
Date: Fri, 21 Sep 2007 00:43:08 GMT
Set-Cookie: JSESSIONID=4EB1673F5BD20FF0495A9FBB38017C38; Path=/myapp
Content-Type: text/xml;charset=ISO-8859-1
Transfer-Encoding: chunked

You can easily time it using:

time curl --header "Content-Type: text/xml" --data '' "" –i

Wednesday, June 20, 2007

"failed to install tomcat5 service" woes?

Trying to install Tomcat 5.x on your XP machine and getting this message:

"failed to install tomcat5 service" Abort, Retry, Ignore...?

The problem could well be that you have already installed Tomcat5 before on this machine and windows is not happy trying to install a second Tomcat5 service. In my case I had installed Tomcat5 before and had deleted it but the old service was still hanging round. The solution is to remove the old service and then the install will continue along happily.

To see if you already have a Tomcat service, you can look in the Service window (under Start-Control Panel-Administrative Tools-Services) and look for a service called "Apache Tomcat". That is the sucker. Right-click and look at the Properties. Check the service name - it will be "Tomcat5". Stop the service if it is running.

Then fire up a command window and run this command:

>sc query tomcat5

This will give you details of the service (if you spell it right). To delete the service:

> sc delete tomcat5

and you should get a nice "[SC] DeleteService SUCCESS" message.

Now you can continue your install!

Friday, May 11, 2007

What jar files to add to your Spring app for Xfire

To expose some of your Spring classes as web services via Xfire you can follow one of the many tutorials on the web. But nobody tells you which jar files into your class path. These worked for me (under Apache Tomcat 5.5 runing Spring 1.2.9):
  • xfire-all-1.2.4.jar - the main library for xfire
  • activation-1.1.jar
  • commons-codec-1.3.jar
  • commons-httpclient-3.0.jar
  • jdom-1.0.jar
  • jsr173_api-1.0.jar
  • mail-1.4.jar
  • stax-1.1.2-dev.jar (this was a pain in the a$%#e to determine!)
  • stax-api-1.0.1.jar
  • wsdl4j-1.6.1.jar
  • xbean-spring-2.7.jar
  • XmlSchema-1.1.jar
This was in addition to usual jar files that MyEclipse will add to your Spring web project when you add Spring capabilities.

Spring and log4j

If you have a Java Spring web application in Apache and are getting this error:

log4j:WARN No appenders could be found for logger (org.apache.commons.digester.Digester.sax).
log4j:WARN Please initialize the log4j system properly.

Then you can fix it by making sure that the file gets deployed to the classes folder under .../webapps/{project}/WEB-INF/classes/

Try sticking it in there and the error should go away. Too easy!

Friday, March 16, 2007

UniVerse and XML

Parsing XML documents in UniVerse just sucks really. I don't know how to describe it any better - it really just sucks!

Oh but validating them against XML schemas works really well! 1-1!

Wednesday, March 14, 2007

JDBC Connection Pooling

For JDC database connection pooling you can use this cool datasource called the BasicDataSource which comes out of the commons-dbcp library (and also requires commons-collections and commons-pool). You can set parameters like the initial pool size, the maximum pool size, whether abandoned connections are removed etc.

They key thing to remember while using the connection pooling is that the datasource object is the scope of the connection pool. ie. if you create a new datasource object then you create a new connection pool. So you need to create one datasource and then either pass it to all your child classes (which something like SPring does for you pretty well), or you refer to a static instance of it from your child classes.

This may sound pretty obvious but I've been working on some code today that had this flaw and and was pretty hard to tell it was there. If you have other users/apps hitting the same DB then it is worth creating a new users for this test. The key is to set the maximum pool size to a low number eg. 10), set the initial pool size to 5, and then run up a 20 or 30 threads. You should not see more than 10 connections to the database (use 'show processlist' in MySQL). If you do, then the pooling isn't working and your code is probably creating datasources as it goes. Nasty!

Thursday, March 8, 2007

Enabling multiple localhost sites in Tomcat

I'm developing code on a Mac under Tomcat for a multi-application website which looks at the domain name to determine which app to run. It was getting damn annoying changing the database tables whenever I wanted to login to a different app. "There must be a better way to do this" he cried! ... and there is!

Rather than just having localhost pointing to your machine, you can have any name you want. eg. dev.localhost, qa.localhost, test.localhost, etc.

Step 1.
Setup the new host names in your hosts file. For me this is under /etc/hosts and I have to sudo to edit it. In there you will already have localhost setup. Add your new ones underneath and save it. localhost qa.localhost dev.localhost test.localhost

Now you should now be able to ping these and see the IP address. In fact, you can try your URL with this server-name instead of localhost and get to your site!

Step 2.
Setup Tomcat to route these hosts to your app. This will run a seperate Tomcat instance for each application. Modify the server.xml file (in the conf folder) by adding the following XML:

<Host name="qa.localhost" appBase="webapps">
<Context path="" docBase=".">

<Host name="dev.localhost" appBase="webapps">
<Context path="" docBase=".">

<Host name="demo.localhost" appBase="webapps">
<Context path="" docBase=".">

I guess if you have other web applications that sit elsewhere you can just modify the appBase path. I didn't need to do this. Note that I was using Eclipse with this setup and when I tried to debug, it started a host for each server and took ages to start; and then died with an out of memory error (this was with 7 aliases setup).

Step 3.
Try it out. Fire up Tomcat, and browse to http:/qa.localhost:8080/{blah blah} and you're away! Nice one brother!

Apache Tomcat URL Limit?

So how long can your URL be before Tomcat spits the dummy and gives you an HTTP 414 error?

On the site I'm currently working on it is 8208 characters. 8207 works fine, 8208 gives the 414 (Request URI too large). And that is one looooonnnnnnnnnnng URL!

That's using Tomcat 5.5.17 and Java 1.5.0_06-b05

Wednesday, March 7, 2007

Software dev can be so frustrating...

This has got to be a typical scenario. I'm working on a system that has been half-written by another team and never quite made it into production. I started work yesterday on a chunk of software dev work that I expected to take about 1/2 a I'd allowed myself a full day to do because we all know to expect the unexpected.

There are four bits to this job, all of them similar. So I'm expecting the first one to take up about half of the time (identify the approach, solve it, test it), and then the other three would flow on pretty easily from there taking the other half of the time. This is one of those cases when 1/4 = 1/2!

So I'm making good progress on the first bit - I'm just testing it out and I'm on target...maybe even ahead of schedule...could be time to nip out for a coffee or a paddle? But then the test results start looking all wrong...further investigation reveals that there is already a bug in the system here and it isn't doing what it should.

Bugger! What do I do? This is a one in ten bug so I really have to fix it. The paddle is definitely off but the coffee could still happen (might have to be a short black though!). So I fix it and find that this is a bit of a b$#%h to test - it requires an exact setup and you've got to do things in the right order. Time ticks on - a good thng I allowed a full day!

But then these tests start to look a bit screwy and lo, it turns out that there is another bug in a fundamental area that this depends on. SH%$#T!

You get the idea, before long I've forgotten what the original job was. By the end of the day (a rather LONG day) all I've succeeded in doing is the first 1/4 of the job, and now I'm faced with writing up all these other little bugs I've found to justify how I've spent the day. And by the time I've done that I'll have lost the flow on the original job and that will end up taking another day.

C'est la vie!

(and it is a good thing I get paid by the hour!) :-)

Monday, March 5, 2007

XSD validation

I spent some time yesterday faffing around with XML and XSD files. In particular, I wanted to use an XML file to send data between systems. The idea being that I could give a sample XML document and and XSD file to someone and they would know enough to interface into the system.

To start with, creating the initial XML document was a pain in the butt! Using my favourite windows text editor TextPad proved to be tedious, so I use Microsoft's XML Notepad 2007. The UI was a bit clunky but it was nice and simple to use.

So XML created, the next step was to generate the XSD. Again, this proved to be a pain in the butt, particularly checking if the XSD syntax was OK. In the end I used Visual Studio 2005 to edit the XMLSchema a it gives the nice red squiggly line under the bits with errors.

So the final step was to validate the XML against the XSD. You think this would be easy right? Well, no.

The browsers (both Firefox2 and IE7) only performed limited validation. They would validate the tags, but not perform the xsd:restrictions. To get this going I had to edit the XML document I wanted to validate and put the name of the XSD file as an attribute of the first element, and then open the XML document via the browser.

eg. <myfirstelement xsi:noNamespaceSchemaLocation="file://c:/xmlschemas/myschema.xsd" xmlns:xsi=""> ...

Good ole XML Notepad seemed the best option. You can choose View-Schemas and then add in your XSD and it will validate against it. Not as simple as I wanted but it worked.

On the Mac I used EditiX (on a 30 day trial) which seemed pretty good, and included the ability to create an XSD from a chunk of XML which can get you started faster. VS2005 also has this feature.

Saturday, March 3, 2007

Post number one...

...and there it was!

Another fascinating post joins the blogosphere and provides thousands of lonely web-crawlers a moment of joy - discovering and indexing another post so that millions of people have something pointless to read.