Java, nosql, persistent queue technologies

Recently I had a task to develop the application which will have large work queue and which need to survive the restarts. The application need to be lightweight. After trying several different persistent engines for Java I''ve chosen to stick with Berkley DB Java edition. This persistent engine is pretty lightweight it is fast, optimized for multi-threaded usage and have no problems with reclaiming free space.

As I needed the fast persistent queue at a cost of possible data loss on system crash I've chosen non-transactional API for Berkley DB. With non-transactional API the great speed can be achieved for persistent queue at a price of loss of some data at system crash. The more data you allow to be lost the greater speed of the queue you will have. Though you can opt to sync to disk each operation on the queue and in that case your data loss will be minimal.

Grails, mongodb technologies

There is some twist to use more specific data storage engines than RDBMS. The mongodb is a modern feature-rich non-RDBMS database. It can be used with grails quite well, though some special care should be taken. In this post we will create base secure grails application powered by mongodb.

Grails, mongodb, nosql technologies

If you are going to use mongodb for production grails application you might stuck with the issue that replica set is really not supported by current version 1.0.0.M1 of mongodb plugin.

It is quite surprising because mongo java driver supports replica set configuration quite fine.

The problem is that GORM plugin is built on top of version 0.5.1 of gmongo library which doesn't support replica set. Only current development version 0.7 of gmongo library supports this feature.

Java, log4j, howto technologies

Log4j doesn't allow to catch stdout and stderr messages out of the box. However, you can still intercept them with a custom output stream, which is especially useful when you have to log data that third-party libraries write to the standard streams.

This has already been done by Jim Moore (have a look at the LoggingOutputStream in the log4j source code). The issue is that this LoggingOutputStream requires org.apache.log4j.Category and org.apache.log4j.Priority which are now partially deprecated.

Distributed, zeromq technologies

If you ever wrote production multi-threaded server in Java you know how it is difficult to implement load balancing between worker threads. You need to fight many issues to have good load balancer:

  1. You need to limit somehow the number of worker threads, because with unlimited thread pool you can have memory exhausted.
  2. You need to implement sophisticated procedure for clean worker shutdown.
  3. If you are using Executors you might now that they are not provide out of the box solution. You need to run into some tricks to have Executors do the load balancing job right.
  4. And for multi-threaded code you have to use those painful synchronized and locks that make your application lock up and/or degrade performance. And the debugging lock issues is a real pain.

ZeroMQ might tremendously help you with solving all these problems and much more. ZeroMQ is a high-performance asynchronous messaging library. It is native, but it has bindings for many languages and to be fair it is worth the hassle of dealing with native code.