High Failure Rate In Unit Test testSerializability When Using HBase 1.2

[saadmufti]

Hi,

This is not really an issue with the current codebase as it uses HBase 0.94 . I am trying to upgrade it to use HBase 1.2 for our purposes and hoping you can give some guidance on the only failing unit test. I can share my fork if you want but didn't make many changes other than to get rid of the table pool as now HBase Table instances are lightweight, so I create them and HaeinsaTable instances on demand.

All unit tests except HaeinsaComplextTest.testSerializability are passing. That one seems to suffer from a high failure rate but still seems to be making progress towards 100 successful iterations that it requires, but at a very slow rate. Most transactions are failing with the following:

FAILED ITERATION 8 THREAD Serializability-job-thread-4 ERROR: this row is unstable and not expired yet.
ERROR
kr.co.vcnc.haeinsa.exception.NotExpiredYetException: this row is unstable and not expired yet.
at kr.co.vcnc.haeinsa.HaeinsaTable.checkAndIsShouldRecover(HaeinsaTable.java:438)
at kr.co.vcnc.haeinsa.HaeinsaTable.access$000(HaeinsaTable.java:75)
at kr.co.vcnc.haeinsa.HaeinsaTable$ClientScanner.next(HaeinsaTable.java:1030)
at kr.co.vcnc.haeinsa.HaeinsaTable.get(HaeinsaTable.java:160)
at kr.co.vcnc.haeinsa.HaeinsaComplexTest$2.run(HaeinsaComplexTest.java:244)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I see the same in the older codebase but the failure rate is lower and it fairly quickly gets to 100 successes and the test passes. In the new code it runs for hours and most threads are still at iteration number in the 50's, at that point the threads start throwing out of memory exceptions.

If I can get some clues where to look at I'd be happy to contribute any changes reqiured back to the project.

Thanks.