Package JSON API (replacement of #810)

[Kleidukos]

This PR finishes the job of #810.

[gbaz]

@Kleidukos why did you reformat the whole cabal file? It makes this rather messy. Are there any substantive changes to the cabal file, and if not, can you revert that portion?

[Kleidukos]

@gbaz nope, only minimal changes from the original PR, I'll revert the formatting changes.
My editor must have ran cabal-fmt on it automatically.

[gbaz]

If you fix the failing tests I will merge. (I'm pretty sure the failure is the "bad package info" error, and the validation warnings can be disregarded).

[Kleidukos]

The test fails with:

Bad package info for versioned HTML /package/testpackage-1.0.0.0 endpoint: "{\"license\":\"BSD-3-Clause\",\"author\":\"\",\"homepage\":\"\",\"synopsis\":\"test package testpackage\",\"description\":\"\",\"metadata_revision\":0,\"copyright\":\"\"}"

Which means that version HTML endpoints that do not have the .html extension in the URL are actually serving JSON.
@gbaz Looking at the feature initialisation code, I have a hard time seeing the faulty logic. :/

[Kleidukos]

@gbaz ping?

[gbaz]

Been caught up with other things. Taking a look now.

[gbaz]

The issue is not with the server, but with the test. It uses Network.HTTP to make a GET request, and that doesn't include a header specifying the content type, and the framework for some reason picks json instead of html when no content type is specified. There might be a fix to the framework to default the other way, but for the most part I think we can expect any modern client to specify a content type.

[gbaz]

If you try this branch with the changes in the above pr, tests should pass. If so, we can merge that branch, then we can rebase this over that and merge this.

[Kleidukos]

@gbaz Thanks a lot for the help! It should be good to merge now. :)

[andreasabel]

This PR breaks the HighlevelTest on CI, e.g.:

1. https://github.com/haskell/hackage-server/runs/5326441167?check_suite_focus=true

2022-02-24T23:04:42.2257890Z = Server is up
2022-02-24T23:04:42.2258103Z = Server running
2022-02-24T23:04:42.2258328Z = Getting package list
2022-02-24T23:04:42.2258563Z = Getting package index    
2022-02-24T23:04:42.2258803Z = Getting package index with etag
2022-02-24T23:04:42.2259057Z = Getting testpackage info
2022-02-24T23:04:42.2259390Z = Validating "/package/testpackage.html": Got thread killed, so killing
2022-02-24T23:04:42.2259685Z = Server terminated
2022-02-24T23:04:42.2259995Z HighLevelTest: Parse exception: Failed reading: not a valid json value  
2022-02-24T23:04:42.2260202Z 
2022-02-24T23:04:42.2260382Z Test suite HighLevelTest: FAIL

2. https://github.com/haskell/hackage-server/runs/5351961922?check_suite_focus=true

Revert?

Test also broken locally on my machine (macOS):

$ time cabal test --enable-tests -w ghc-8.10.7 HighLevelTest 
Resolving dependencies...
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - hackage-server-0.5.1 (test:HighLevelTest) (ephemeral targets)
Preprocessing test suite 'HighLevelTest' for hackage-server-0.5.1..
Building test suite 'HighLevelTest' for hackage-server-0.5.1..
Running 1 test suites...
Test suite HighLevelTest: RUNNING...
= Initialising
= Setting up test directory
= initialising hackage database
using "/Users/abel/bin/src/hackage-server/dist-newstyle/build/x86_64-osx/ghc-8.10.7/hackage-server-0.5.1/x/hackage-server/noopt/build/hackage-server/hackage-server"
hackage-server: Creating initial state...
finalising data for feature users
finalising data for feature core
finalising data for feature security
hackage-server: Using default administrator account (user admin, passwd admin)
hackage-server: Done
= Forking server thread
= Making a request to see if server is up
= Server thread started
= Server not up yet; will try again shortly
= (result was Left user error (openTCPConnection: failed to connect to 127.0.0.1:8392: Network.Socket.connect: <socket: 14>: does not exist (Connection refused)))
using "/Users/abel/bin/src/hackage-server/dist-newstyle/build/x86_64-osx/ghc-8.10.7/hackage-server-0.5.1/x/hackage-server/noopt/build/hackage-server/hackage-server"
hackage-server: Guessing public URI as http://unknownACBC32A26F85:8392
(you can override with the --base-uri= flag)
hackage-server: Ready! Point your browser at http://unknownACBC32A26F85:8392
= Making a request to see if server is up
= Server is up
= Server running
= Validating "/": Got thread killed, so killing
= Server terminated
HighLevelTest: ConnectionAbruptlyTerminated

Test suite HighLevelTest: FAIL
Test suite logged to:
/Users/abel/bin/src/hackage-server/dist-newstyle/build/x86_64-osx/ghc-8.10.7/hackage-server-0.5.1/t/HighLevelTest/noopt/test/hackage-server-0.5.1-HighLevelTest.log
0 of 1 test suites (0 of 1 test cases) passed.
Error: cabal: Tests failed for test:HighLevelTest from hackage-server-0.5.1.


real	1m14.293s
user	0m4.804s
sys	0m1.420s

[Kleidukos]

= (result was Left user error (openTCPConnection: failed to connect to 127.0.0.1:8392: Network.Socket.connect: <socket: 14>: does not exist (Connection refused)))

Is this the failure you're talking about?

[andreasabel]

@Kleidukos: Atm, I cannot reproduce the failure locally. Unfortunately, I have overwritten the logs with the failure by running the test again.

But the CI logs are still present. E.g. I copied an error from the log of run 1 I quote in #996 (comment):

2022-02-24T23:04:42.2259995Z HighLevelTest: Parse exception: Failed reading: not a valid json value  
2022-02-24T23:04:42.2260202Z 
2022-02-24T23:04:42.2260382Z Test suite HighLevelTest: FAIL

I see now that the same failure occurred on: https://github.com/haskell/hackage-server/runs/5267206134?check_suite_focus=true which was before your PR.

It seems the test fails now and then. Cf. runs on pushes to master: https://github.com/haskell/hackage-server/actions/workflows/ci.yml?query=event%3Apush+branch%3Amaster