Sort out doc building

[silky]

There are many issues related building of library docs. Others have been reported on the haskell-cafe and cabal-devel mailing lists.

One seems to outline a plan related to fixing the main system: #282 however it's 2 years old, and no progress for over a year.

can we sort this out?

is there some kind of plan in the works? @hvr ? or should we just get our hands dirty by trying to replicate the problem with local setups of the hackage server? happy to proceed in any direction; but looking for some discussion first.

thanks.

[gbaz]

What we need is a volunteer (assistant) doc czar to help out @davean

The issues are partially in building better, more stable docbuilder code (and honestly rewriting some of how the interactions work).

But keeping the builders up and running is also a sysadmin task, and it needs people to mind it.

Note that ongoing discussion of the docbuilder stuff tends to happen slowly via chats in the #haskell-infrastructure irc channel.

If anyone reading this ticket wants to volunteer and offer a commitment of time and attention over a significant span of time to help mind the docbuilders, and in so doing, to understand their failure modes, please speak up!

[gbaz]

The current round of failures seems due to this issue going on with a cabal ticket: haskell/cabal#3093

[silky]

i'm happy to volunteer, and also to look into rewriting some of the code after getting an understanding for it.

how should i proceed?

[gbaz]

The main thing is to idle in #haskell-infrastructure and volunteer there. We'll try to hook you up with an account on the builder boxes in the next few days (you can email [email protected] too). From there, its investigating their logs, and the coming up with a plan of attack that A) makes the builder failures and status more publicly auditable and B) more resilient.

Part of this relates to being able to run multiple builders and have them co-ordinate, which is about managing shared state of built/unbuilt and reasons, in part.

[silky]

okay, thanks will do.

[crockeea]

Any progress on this? I'd be happy to do (and have been doing) some digging into hackage-build.

Just today, I was using a local hackage server to try to build docs for some pacakges that mysteriously have no docs on the public server. In particular, transformers-0.5.2.0, built in seconds on my computer with GHC 7.10.3.

@gbaz: If the error is related to cabal, why are we not seeing (failed) build-reports on the public server? transformers doesn't have one, but my local server created a report, log, and docs without problems.

[silky]

i've not gotten anywhere.

last i heard @davean was rebuilding things slowly and was making progress, but i'm not sure since then.

[gbaz]

Things got busy, but I'd like to work with @davean 's and your help to just do some interim tweaks to the builder box to at a minimum get better logging and change the queue system to be more inspectable, as an interim step.

[crockeea]

@gbaz Docs are still not building reliably: there are no attempted builds reported for lol-apps. However, they did build successfully for lol. [EDIT: I manually uploaded docs with hackage-build.]

[erantapaa]

fwiw, I've created a report of packages whose latest versions are still in "Docs pending" status:

http://erantapaa.github.io/doc-status-2016-07-08/report.html

There is a lot of information there, but if you click on a number in the left-hand column you'll get an English explanation of the data row. Good examples of recent uploads which should cause concern are rows 41-45. Basically any row where the last column is "Docs available" is an upload which should be investigated.

By looking at the end of the report it seems that the problem started around the end of November 2015.

[crockeea]

@erantapaa Thanks for looking into this. I want to point out that the problem has been going on since at least July 2014, as reported in #244.

[gbaz]

@bgamari is now taking a look at this on the server as well.

[Blaisorblade]

I want to point out that the problem has been going on since at least July 2014, as reported in #244.

An earlier report is from October 2013 in #145.

[bgamari]

I had another look at this today and actually had some trouble finding a failed package. One failure that I was able to find was texmath-0.8.6.5. The log is rather perplexing,

Resolving dependencies...
cabal: Could not resolve dependencies:
next goal: texmath (user goal)
rejecting: texmath-0.8.6.4, texmath-0.8.6.3, texmath-0.8.6.2, texmath-0.8.6.1,
texmath-0.8.6, texmath-0.8.5.1, texmath-0.8.5, texmath-0.8.4.2,
texmath-0.8.4.1, texmath-0.8.4, texmath-0.8.3, texmath-0.8.2.2,
texmath-0.8.2.1, texmath-0.8.2, texmath-0.8.1, texmath-0.8.0.2,
texmath-0.8.0.1, texmath-0.8, texmath-0.7.0.2, texmath-0.7.0.1, texmath-0.7,
texmath-0.6.6.3, texmath-0.6.6.2, texmath-0.6.6.1, texmath-0.6.6,
texmath-0.6.5.2, texmath-0.6.5, texmath-0.6.4, texmath-0.6.3, texmath-0.6.1.5,
texmath-0.6.1.4, texmath-0.6.1.3, texmath-0.6.1.2, texmath-0.6.1.1,
texmath-0.6.1, texmath-0.6.0.6, texmath-0.6.0.5, texmath-0.6.0.4,
texmath-0.6.0.3, texmath-0.6.0.2, texmath-0.6.0.1, texmath-0.6,
texmath-0.5.0.4, texmath-0.5.0.3, texmath-0.5.0.2, texmath-0.5.0.1,
texmath-0.5, texmath-0.4, texmath-0.3.1.5, texmath-0.3.1.4, texmath-0.3.1.3,
texmath-0.3.0.3, texmath-0.3.0.2, texmath-0.3.0.1, texmath-0.3,
texmath-0.2.0.4, texmath-0.2.0.3, texmath-0.2.0.2, texmath-0.2.0.1,
texmath-0.2, texmath-0.1.1, texmath-0.1.0.4, texmath-0.1.0.3, texmath-0.1.0.2,
texmath-0.1.0.1, texmath-0.1, texmath-0.6.7, texmath-0.6.5.1 (constraint from
user target requires ==0.8.6.5)
Dependency tree exhaustively searched.

It appears that we tried to build the target version, but it was present in the local package list. How could this happen?

[crockeea]

@bgamari Where did you find that log file? It doesn't appear to be uploaded on Hackage, so regardless of whether the build should have succeeded or not, that's a bug.

[bgamari]

crockeea [email protected] writes:

@bgamari Where did you find that log file? It doesn't appear to be
uploaded on Hackage, so regardless of whether the build should have
succeeded or not, that's a bug.

That is from
/home/builder/hackage-server/build-cache/results/texmath-0.8.6.5.log
on the builder itself. I believe @davean once told me that failures to
find a workable install plan indeed weren't handled correctly (resulting
in the package being stuck in "documentation pending" state). This does
seem like a bug, albeit a known bug. I don't yet know why this is the
case.

[bgamari]

I've opened #543 to track the issue I mentioned above.

[gbaz]

I think the docbuilder has now been running ok for some time thanks to various changes by bgamari and more general oversight of the system. Going to close out this issue now as if new problems arise there should be new issues rather than a general running tally imho.