missing docs on MANY new packages!!!!

[cartazio]

Have we fixed this in hackage server head? when is it getting rolled out and haddocked regenerated?

http://hackage.haskell.org/package/cabal-debian-3.8.3 is a new example of this problem i just found :(

[blitzcode]

I also noticed missing documentation for a lot of packages like OpenGL and GLFW-b, have to rely on the docs of older versions or the ones I have installed locally. This is rather annoying as Hoogle has links to the main Hackage pages, and I then have to manually locate the alternative ones. A fix would be much appreciated!

[balta2ar]

Confirm the problem. Here is another package with missing docs: http://hackage.haskell.org/package/http-streams-0.7.0.2

[istathar]

To @balta2ar's comment, it's not a problem with the package in question; it renders fine locally (as mirrored at http://research.operationaldynamics.com/projects/http-streams/doc/Network-Http-Client.html) and the part that was spliced out of the package, http-common is fine on Hackage — ie http://hackage.haskell.org/package/http-common-0.7.0.1/docs/Network-Http-Types.html is fine.

AfC

[z0w0]

helm-0.5.0 is missing docs too, they build fine locally.

[funrep]

I confirm this problem too, verbalexpressions which I uploaded recently is missing docs too. Thought I first had to build docs locally, but then realized that cabal's sdist doesnt pack them into the package anyway.

[cartazio]

#56 would be a workaround ... we just need to make it discoverable

[istathar]

Why should it need a workaround? Hackage 1 never had any problems with any of these packages.

[cartazio]

well, then write the patch needed, whatever it is. In the mean time, thats a workaround, upload your own docs :)

[dcoutts]

Note that we're now running a new doc builder client which should resolve some of these issues (it's currently going through a big backlog). For the things not resolved, it does at least generate build logs (see /package/foo-1.0/reports/$n/log) so we can see what's wrong. Mostly I expect this will now be missing C libs which we'll need to add to the builder machine.

In particular, as a test case I've deleted the broken docs for http://hackage.haskell.org/package/verbalexpressions-1.0.0.0, so keep an eye on http://hackage.haskell.org/package/verbalexpressions-1.0.0.0/reports/ and http://hackage.haskell.org/package/verbalexpressions-1.0.0.0/reports/1/log

[dcoutts]

Note also, that if you find packages with broken docs, one can delete them and the doc builder should automatically retry. e.g.

curl -X DELETE http://you:[email protected]/package/helm-0.5.0

You have to be in the maintainer or trustee group to do this. So this would be a good task for trustees, speaking of which, we ought to recruit a few! http://hackage.haskell.org/packages/trustees/

[funrep]

I tried to upload docs to the verbalexpressions package but I got this error:

$ curl -X PUT -d @dist/doc.tar http://usr:[email protected]/package/verbalexpressions-1.0.0.0/docs
Unexpected content-type: For this resource the content-type was expected to be application/x-tar, rather than application/x-www-form-urlencoded

What format should the file I gonna upload be? I just made the dist/doc a tar since I thought that was the accepted format reading the server API.

I also tried the script which was presented in issue #56 but got the following error:

$ sh build-docs.sh 
build-docs.sh: 8: build-docs.sh: Syntax error: "(" unexpected

I changed the parentheses to quotes since I thought it seemed like a string but then I got this error:

$ sh build-docs.sh
http://hackage.haskell.org/package/verbalexpressions-1.0.0.1 404
http://hackage.haskell.org/package/verbalexpressions-1.0.0.1/candidate 404
Found no uploaded version
Resolving dependencies...
Configuring verbalexpressions-1.0.0.1...
build-docs.sh: 72: build-docs.sh: Bad substitution

I really don't know bash so I thought I rather ask here how to upload docs by hand.

[dcoutts]

@klrr to answer the curl question, are indeed uploading the right format (ie a tar file), so all you need to do is tell the server that that is what you're doing. By default curl tells the server that it is sending the results of a html form (application/x-www-form-urlencoded), so you just need to override that and tell it the type you're actually sending (ie tar)

To do that is the -H flag to specify the 'Content-Type header, -H 'Content-Type: application/x-tar':

curl -X PUT -H 'Content-Type: application/x-tar' -d @dist/doc.tar http://usr:[email protected]/package/verbalexpressions-1.0.0.0/docs

[dcoutts]

As for using verbalexpressions as a test case, we can now see that the new doc builder is more useful than the old, because it uploads a report and a build log:

• http://hackage.haskell.org/package/verbalexpressions-1.0.0.0/reports/1/
• http://hackage.haskell.org/package/verbalexpressions-1.0.0.0/reports/1/log

So the report says that regex-pcre failed to build, and verbalexpressions depends on that. Looking at the log we can see why: the pcre C lib is missing on the build machine.

So any of the admins with access to the build machine can fix this by installing that C lib, then grepping the reports for affected packages:

$ grep 'DependencyFailed regex-pcre' /srv/hackage/var/build/results/*.report -l
/srv/hackage/var/build/results/aeson-schema-0.2.0.1.report
/srv/hackage/var/build/results/anatomy-0.4.report
/srv/hackage/var/build/results/atomo-0.4.0.2.report
/srv/hackage/var/build/results/hark-0.2.report
/srv/hackage/var/build/results/hellage-0.1.1.report
/srv/hackage/var/build/results/hyakko-0.6.5.report
/srv/hackage/var/build/results/verbalexpressions-1.0.0.0.report

Then to trigger their rebuild, the admin has to remove these packages from the failed file, and then the doc builder will have another go.

[funrep]

@dcoutts Thanks for the quick reply! I then couldn't upload it since it was in wrong format but I changed it to ustar format as suggested in issue #56. Now I get the following error though which I don't really get since doc.tar is not empty at all.

$ curl -X PUT -H 'Content-Type: application/x-tar' -d @dist/doc.tar http://foo:[email protected]/package/verbalexpressions-1.0.0.0/docs
Warning: Couldn't read data from file "dist/doc.tar", this makes an empty
Warning: POST.
Invalid documentation tarball: truncated tar archive

[dcoutts]

@klrr does dist/doc.tar exist? Is it a valid tarball. The fact that curl cannot read from it suggests it's either missing, or wrong permissions or something.

BTW, you may prefer to just hold off, I've installed the pcre C lib dev files on the builder machine and prodded the builder to try again, and verbalexpressions is on the builder's TODO list again.

[funrep]

The problem was that it was that I specified wrong path, when I ran with correct path to the file it said it was in wrong format although I used the ustar format. I'll just wait for the bot now though, thanks for all help!

[banacorn]

+1

[xich]

How often are the docs rebuilt? I thought it was every night, but I uploaded a package a couple days ago that still doesn't have them, so I'm wondering if something is wrong. I tried to find build logs, but I can't find a link anywhere. Manually visiting the link (suggested above): http://hackage.haskell.org/package/scotty-0.6.0/reports/ gives an empty list.

Or is this just a case of a backlog and I should be more patient? ;-)

[dcoutts]

@xich I've not checked specifically, but the likely answer is it falls into the corner case where no report is generated, which is if the solver cannot find a solution. This doc builder instance is using 7.6.3, and does not allow reinstalling core packages.

Technically the problem here is that the build report format has to be extended to be able to describe the case when no solution can be found (so we don't even know the deps that a package was trying to use for example).

[z0w0]

Helm also didn't have any build reports yet still failed to build. I just deleted the docs using the API and it worked on the next build, so I'd give that a try @xich.

[xich]

@dcoutts If I were to volunteer to try to extend the build report format to handle this case, could you point me to a place to get started?

@z0w0 Thanks... I tried doing that, to no avail. (The deletion seemed to happen, but after a few days, still no docs.)

[cartazio]

@xich the best way to get @dcoutts to explain stuff is to pester him to explain on IRC on #hackage, then paste the notes to a github ticket for yourself or others :)

[m0dev76]

I'm having issues with the following packages right now:

http://hackage.haskell.org/package/haskoin-util-0.0.1
http://hackage.haskell.org/package/haskoin-crypto-0.0.1
http://hackage.haskell.org/package/haskoin-protocol-0.0.1

The documentations are not showing up. I also have no reports displaying under /:package/reports which makes debugging the issue somewhat harder.

I had the tests missing in the tarball of haskoin-util-0.0.1, so I added them and re-uploaded the new package as haskoin-util-0.0.1.1 and the documentation showed up. I removed a benchmarks section from the cabal file of haskoin-crypto-0.0.1 and re-uploaded it as haskoin-crypto-0.0.1.1 and the documentation showed up. Now, hoping to have learned from my previous mistakes, I have uploaded haskoin-protocol-0.0.1 but no documentation is showing up. The tarball builds fine locally, including the documentation.

I just tried to issue a docs delete command:

curl -X DELETE http://PhilippeLaprade:[email protected]/package/haskoin-protocol-0.0.1/docs

I'll be waiting a few hours to see if anything happens. I'll try to find some time to install hackage-server and look at what's going on.

Cheers

[dcoutts]

@xich yes, look at the BuildReport type in Distribution.Server.Features.BuildReports.BuildReport.

As you can see, the current representation expects a list of dependencies that we tried to build the package against. In the case that we could not even find a solution then of course we cannot provide that.

So, we would need to extend the InstallOutcome to add an option for not finding a satisfactory set of dependencies, and in that case the flagAssignment and dependencies should be allowed to be empty, or not present. It's an interesting Q if they should be required to be empty, or if them being partially present is useful to express the case that one tried with certain constraints (e.g. constraining to the versions of the core/platform packages from a specific ghc/HP release).

[m0dev76]

I don't know if this is relevant, but I seem to have fixed my problem by changing the dependency formats in my cabal files. Instead of using == 0.1.* I used >= 0.1 && < 0.2. I uploaded two more packages which built successfully on the first try.

[gbaz]

I think we can close this now, since we have our builders more carefully manged?

[gbaz]

closing in favor of #478 which we've made great strides on