NPE during cleanup up of big blobs

[sgandon]

Hi Roberto,
When removing blobs bigger than 1MB in my repo I get the following NPE.
I have not had a look at your code so I am not sure if it comes from JGit or BFG.
You'll find at the bottom the command line used that generated this exception.

 [echo] Exception in thread "main" java.lang.NullPointerException
 [echo]     at org.eclipse.jgit.internal.storage.file.UnpackedObjectCache$Table.index(UnpackedObjectCache.java:146)
 [echo]     at org.eclipse.jgit.internal.storage.file.UnpackedObjectCache$Table.contains(UnpackedObjectCache.java:109)
 [echo]     at org.eclipse.jgit.internal.storage.file.UnpackedObjectCache.isUnpacked(UnpackedObjectCache.java:64)
 [echo]     at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject1(ObjectDirectory.java:359)
 [echo]     at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObjectImpl1(FileObjectDatabase.java:173)
 [echo]     at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObject(FileObjectDatabase.java:158)
 [echo]     at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
 [echo]     at org.eclipse.jgit.treewalk.CanonicalTreeParser.reset(CanonicalTreeParser.java:201)
 [echo]     at org.eclipse.jgit.treewalk.TreeWalk.parserFor(TreeWalk.java:984)
 [echo]     at org.eclipse.jgit.treewalk.TreeWalk.addTree(TreeWalk.java:468)
 [echo]     at com.madgag.git.package$RichRevTree.walk(package.scala:68)
 [echo]     at com.madgag.git.package$.allBlobsUnder(package.scala:215)
 [echo]     at com.madgag.git.bfg.cleaner.protection.ProtectedObjectCensus$$anonfun$6.apply(ProtectedObjectCensus.scala:79)
 [echo]     at com.madgag.git.bfg.cleaner.protection.ProtectedObjectCensus$$anonfun$6.apply(ProtectedObjectCensus.scala:79)
 [echo]     at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
 [echo]     at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
 [echo]     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
 [echo]     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
 [echo]     at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
 [echo]     at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
 [echo]     at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
 [echo]     at com.madgag.git.bfg.cleaner.protection.ProtectedObjectCensus$.apply(ProtectedObjectCensus.scala:79)
 [echo]     at com.madgag.git.bfg.cli.CLIConfig.objectProtection$lzycompute(CLIConfig.scala:143)
 [echo]     at com.madgag.git.bfg.cli.CLIConfig.objectProtection(CLIConfig.scala:143)
 [echo]     at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:57)
 [echo]     at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:34)
 [echo]     at scala.Option.map(Option.scala:145)
 [echo]     at com.madgag.git.bfg.cli.Main$delayedInit$body.apply(Main.scala:33)
 [echo]     at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
 [echo]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
 [echo]     at scala.App$$anonfun$main$1.apply(App.scala:71)
 [echo]     at scala.App$$anonfun$main$1.apply(App.scala:71)
 [echo]     at scala.collection.immutable.List.foreach(List.scala:318)
 [echo]     at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
 [echo]     at scala.App$class.main(App.scala:71)
 [echo]     at com.madgag.git.bfg.cli.Main$.main(Main.scala:27)
 [echo]     at com.madgag.git.bfg.cli.Main.main(Main.scala)     [echo] Exception in thread "main" java.lang.NullPointerException
 [echo]     at org.eclipse.jgit.internal.storage.file.UnpackedObjectCache$Table.index(UnpackedObjectCache.java:146)
 [echo]     at org.eclipse.jgit.internal.storage.file.UnpackedObjectCache$Table.contains(UnpackedObjectCache.java:109)
 [echo]     at org.eclipse.jgit.internal.storage.file.UnpackedObjectCache.isUnpacked(UnpackedObjectCache.java:64)
 [echo]     at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject1(ObjectDirectory.java:359)
 [echo]     at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObjectImpl1(FileObjectDatabase.java:173)
 [echo]     at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObject(FileObjectDatabase.java:158)
 [echo]     at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
 [echo]     at org.eclipse.jgit.treewalk.CanonicalTreeParser.reset(CanonicalTreeParser.java:201)
 [echo]     at org.eclipse.jgit.treewalk.TreeWalk.parserFor(TreeWalk.java:984)
 [echo]     at org.eclipse.jgit.treewalk.TreeWalk.addTree(TreeWalk.java:468)
 [echo]     at com.madgag.git.package$RichRevTree.walk(package.scala:68)
 [echo]     at com.madgag.git.package$.allBlobsUnder(package.scala:215)
 [echo]     at com.madgag.git.bfg.cleaner.protection.ProtectedObjectCensus$$anonfun$6.apply(ProtectedObjectCensus.scala:79)
 [echo]     at com.madgag.git.bfg.cleaner.protection.ProtectedObjectCensus$$anonfun$6.apply(ProtectedObjectCensus.scala:79)
 [echo]     at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
 [echo]     at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
 [echo]     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
 [echo]     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
 [echo]     at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
 [echo]     at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
 [echo]     at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
 [echo]     at com.madgag.git.bfg.cleaner.protection.ProtectedObjectCensus$.apply(ProtectedObjectCensus.scala:79)
 [echo]     at com.madgag.git.bfg.cli.CLIConfig.objectProtection$lzycompute(CLIConfig.scala:143)
 [echo]     at com.madgag.git.bfg.cli.CLIConfig.objectProtection(CLIConfig.scala:143)
 [echo]     at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:57)
 [echo]     at com.madgag.git.bfg.cli.Main$$anonfun$1.apply(Main.scala:34)
 [echo]     at scala.Option.map(Option.scala:145)
 [echo]     at com.madgag.git.bfg.cli.Main$delayedInit$body.apply(Main.scala:33)
 [echo]     at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
 [echo]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
 [echo]     at scala.App$$anonfun$main$1.apply(App.scala:71)
 [echo]     at scala.App$$anonfun$main$1.apply(App.scala:71)
 [echo]     at scala.collection.immutable.List.foreach(List.scala:318)
 [echo]     at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
 [echo]     at scala.App$class.main(App.scala:71)
 [echo]     at com.madgag.git.bfg.cli.Main$.main(Main.scala:27)
 [echo]     at com.madgag.git.bfg.cli.Main.main(Main.scala)

here is the command line used to launch BFG.

java -jar /home/talend/talend-svn-git-migration/bfg-1.11.1.jar --strip-blobs-bigger-than 1M --protect-blobs-from branch-5_0,branch-5_1,branch-5_2,branch-5_3,branch-5_4,master,release-5_0_0,release-5_0_1,release-5_0_2,release-5_0_3,release-5_1_0,release-5_1_1,release-5_1_2,release-5_1_3,release-5_2_0,release-5_2_1,release-5_2_2,release-5_2_3,release-5_3_0,release-5_3_1,release-5_3_2,release-5_4_0,release-5_4_1

[rtyley]

Interesting - looks like The BFG is passing JGit a null object id, which is pretty surprising. Is the repo publicly available for me to reproduce this?

[rtyley]

Failing that, could you send me the output of this command executed in your repositories directory:

$ git show-ref

[sgandon]

Hello Roberto,
I am sorry but the repos that is failing with BGF contains code I cannot share.
here is the command output you asked for.

git show-ref
80db0f39ea4fc8716ad4cdb8e181a16abfdaa7a0 refs/heads/branch-5_0
7f0e5bf7cf2fd5e5f444de15b2894e875399f554 refs/heads/branch-5_1
febd2ccad46300eba94ec82e01d21ae5e36d0cb2 refs/heads/branch-5_2
de33dc82a948c205532b8593718863f6322e7120 refs/heads/branch-5_3
5620b7ae9102e06135ad43fc4655a43d874a24ce refs/heads/branch-5_4
3a8606c7102302a74464b4ce17f5019be19a12c1 refs/heads/master
8dcdcd7992e97f4fe0b8fda4f934fcda4f2a0df0 refs/tags/release-5_0_0
acc9675265c7ebefa753b12ec9e08ffb2789742c refs/tags/release-5_0_1
c59f6a3803955eca3a42db1ea71cc90f884641c6 refs/tags/release-5_0_2
5369404bca1d80bc0ae7db47d66d96bd12fc651a refs/tags/release-5_0_3
2dbffbd38b962e0669a142c64ec0c35fdff92388 refs/tags/release-5_1_0
038bdbbeab835bca47656b125104e9cf8b7f81b8 refs/tags/release-5_1_1
9a9c7cc5050bb19faaf47665ced816d49bc0ff4b refs/tags/release-5_1_2
57d6633ef8185e0e7021cbde2816808aa1c46e8b refs/tags/release-5_1_3
072287773a0141f4801c940803d5c96b41f11fbe refs/tags/release-5_2_0
fbeffb86cc0703c85c6770b98bfbd0dec18ed9ad refs/tags/release-5_2_1
892fd206c058c9022eba97ee8dc3581945c08557 refs/tags/release-5_2_2
fc063303ba04afec26cee76284e43a4bad455872 refs/tags/release-5_2_3
46e9348aaa7938ad1d464b33c747bee27ceb23e8 refs/tags/release-5_3_0
e968665684c4c23d5f61cf34baccd23fc3af0ef5 refs/tags/release-5_3_1
5941369cce058d765486e842dd8a4bf750f6568b refs/tags/release-5_3_2
8cdab45d3fec7533d9918257e33b8c7da04a4137 refs/tags/release-5_4_0
1c9b2c853a34f47cc95fa577d54ba28e079c5868 refs/tags/release-5_4_1

[sgandon]

I have tried to remove all the tags from the --protect-blobs-from option keeping only branches and master. And it works fine.
I have tried with only keeping one tag and removing all other tags and branches and I have the same issue.
It seems directly related to tags. before the example that you gave on your web page (http://rtyley.github.io/bfg-repo-cleaner/) you mention Tags, so I thought it was supported ?

[rtyley]

@sgandon, you're completely right, this was a bug in The BFG - thanks for reporting it. It only occurred with 'annotated' tags - tag objects that contain full tagging metadata like author & message - which is why I hadn't spotted it before. The fix is in the soon-to-be-available v1.11.2 release of The BFG. Thanks again for getting in contact!

[sgandon]

You are most welcome, you product is a killer especially because of the speed performance.
Do you know how to be notified on a new release? is there any rss feed I could subscribe to ?

[rtyley]

Ah no worries, the new v1.11.2 release is already available and you can download it here:

http://rtyley.github.io/bfg-repo-cleaner/

There's a delay of a few hours between me 'releasing' a release and the new artifact being available for download - this delay is the Maven Central 'sync' delay, where artifacts are copied from feeder repositories (like oss.sonatype.org) to Maven Central proper.

I don't currently have an RSS feed or Twitter account for releases - for most people, the need for The BFG is a one-off, they generally don't need to be kept up to date with updates!

[sgandon]

Thank for everything, I'll test it right away.
You are right this tool is usually a one shot use, but I am preparing a big migration, that will happened in june, so I have plenty of time to make things right.