Doc builder occassionally silently fails due to missing target

[bgamari]

Much of the missing documentation on Hackage appears to be due to cabal failing to find an install plan, due to the target package apparently being missing from the package index. For instance, clock-0.7.2 failed with,

Resolving dependencies...
cabal: Could not resolve dependencies:
next goal: clock (user goal)
rejecting: clock-0.7.1.1, 0.7.1, 0.6.0.1, 0.6.0, 0.5.2, 0.5.1, 0.5.0.1,
0.5.0.0, 0.4.6.0, 0.4.5.0, 0.4.4.0, 0.4.3.0, 0.4.2.0, 0.4.1.3, 0.4.1.2,
0.4.1.1, 0.4.0.1, 0.3, 0.2.0.0, 0.1.2.1, 0.1.2, 0.1.1, 0.1 (global constraint
requires ==0.7.2)
Dependency tree exhaustively searched.

To make matters worse, the package is stuck in "docs pending" state indefinitely after this happens.

[hvr]

Ok, so if I understand this right, the doc-builder polls http://hackage.haskell.org/packages/docs to figure out what to do next,

...and that list can easily contain packages which have not yet entered the 01-index.tar.gz (which is obviously needs some time to be computed and can easily lag behind several seconds; I've seen this myself after package uploads or edits that it takes several seconds).

I think the doc-builder should simply filter out packages from the doc-index for which there isn't a 01-index.tar entry yet. This is easily accomplished.

[bgamari]

Herbert Valerio Riedel notifications@github.com writes:

I think the doc-builder should simply filter out packages from the
doc-index for which there isn't a 01-index.tar entry yet. This is
easily accomplished.

I'll admit I'm not sure I see the "easily" part here. Doesn't the
package repository business live in cabal-install, which we cannot link
against?

Regardless, I still suspect that having some sort of retry policy could
only help things.

[hvr]

@bgamari you don't need to link into cabal-install; cabal-install writes out an easy to parse textual index file ~/.cabal/packages/hackage.haskell.org/00-index.cache:

pkg: Crypto 3.0.3 b# 3
pkg: HDBC 1.0.1 b# 13
pkg: HDBC-odbc 1.0.1.0 b# 15
pkg: HDBC-postgresql 1.0.1.0 b# 18
pkg: HDBC-sqlite3 1.0.1.0 b# 21
pkg: darcs-graph 0.1 b# 24
...

the retry policy is useful in addition of course...

[bgamari]

Herbert Valerio Riedel notifications@github.com writes:

@bgamari you don't need to link into cabal-install; cabal-install writes out an easy to parse textual index file ~/.cabal/packages/hackage.haskell.org/00-index.cache:

Ahh, I see. I don'u suppose there is a parser for this format available
in Cabal?

[hvr]

@bgamari no (because Cabal the lib doesn't know about repository indices), there's a parser inside cabal-install's code though. But that is overkill, since the general idea here is that all you need to parse is lines that begin with pkg: and ignore all else, and for those the syntax is:

"pkg:" <pkgname> <pkgver> {<tag> <val>}

so all you have to do is apply words to each line, and see if the first list element is "pkg:", and if that's the case use the 2nd and 3rd list element as package name & version.

[bgamari]

This strikes me as terribly ugly, but I'll do it. This is precisely the reason why I think it would be helpful if cabal-install would split out its functionality into a few loosely coupled libraries. This sort of reinventing the wheel is fragile and unnecessary.