Recently I had a task to develop the application which will have large work queue and which need to survive the restarts. The application need to be lightweight. After trying several different persistent engines for Java I''ve chosen to stick with Berkley DB Java edition. This persistent engine is pretty lightweight it is fast, optimized for multi-threaded usage and have no problems with reclaiming free space.
As I needed the fast persistent queue at a cost of possible data loss on system crash I've chosen non-transactional API for Berkley DB. With non-transactional API the great speed can be achieved for persistent queue at a price of loss of some data at system crash. The more data you allow to be lost the greater speed of the queue you will have. Though you can opt to sync to disk each operation on the queue and in that case your data loss will be minimal.