RFC: --index-snapshot <posix-time>

[hvr]

This is an early working proof-of-concept (which already works), and I'm submitting this here to solicit an early general discussion/feedback regarding the direction the implementation is going.

This is a forward-ported version of

hvr@3dea219

mostly aimed at use with new-build & cabal.project files.

This introduces a flag --index-snapshot and a respective property index-snapshot: ... for use in cabal.project(.local) files which currently only supports a unix timestamp argument. The --index-snapshot-value acts as a timestamp-filter on the package index. For now, only posix-timestamps are supported as filtering criteria. Additionally an "index serial-number" is planned, which would enumerate all entries in the 01-index-tarball incrementing the serial number for each .cabal and each preferred-versions entry. This is a more exact/granular way to specify the index-state, as there could be more than one index-entry with the same timestamp.

By setting --index-snapshot to a timestamp from the past(!), this effectively freezes the hackage index to the state and thereby allows for simple way to provide (non-curated) deterministic install-plans.

This requires enabling secure repos to have any useful effect (non-secure repos have a 0-timestamp for all entries, therefore all entries lay in the past of positive --index-snapshot values and are not filtered).

Since the index-cache format is extended, this uses a different name 01-index.cache and the contents looks like

pkg: b# 0 t# 1157499455 iconv 0.2
pkg: b# 3 t# 1159125300 Crypto 3.0.3
...
pkg: b# 663524 t# 1460923884 aeson-flatten 0.1.0.0
pkg: b# 663530 t# 1460924055 aeson-flatten 0.1.0.1
pkg: b# 663536 t# 1460924864 aeson-prefix 0.1.0.1
pref-ver: b# 663542 t# 1460924975 aeson-prefix <0.1.0.0 || >0.1.0.0
pref-ver: b# 663544 t# 1460925074 aeson-flatten <0.1.0.0 || >0.1.0.0
pkg: b# 663546 t# 1460925526 aeson-prefix 0.1.0.2
pref-ver: b# 663554 t# 1460925649 aeson-prefix <0.1.0.0 || >0.1.0.0 && <0.1.0.1 || >0.1.0.1
pkg: b# 663556 t# 1460927930 data-default-instances-base 0.1.0
pkg: b# 663560 t# 1460928015 data-default 0.6.0
pkg: b# 663565 t# 1460955096 ether 0.4.0.1
pkg: b# 663578 t# 1460955492 ether 0.4.0.1
...

where b# denotes the block-offset into the index-tarball and t# denotes the posix-time of that entry

PS: This patch is based on 1.24 as I started working on this rebased patch a couple of weeks ago already and was planning to have this merged in time for 1.24.1

/cc @ezyang @dcoutts @23Skidoo

[hvr]

PS: some variations are easily computable (i.e. consider this as different modes we could implement besides the strict filtering mode):

• drop all cabal edits, i.e. keep only the first .cabal entry within the filtering criteria (i.e. simulate Hackage without any cabal edits)
• pick up the latest cabal edit for packages not pruned by the filter-cutoff (even if those fall past the cutoff)
• "round up" to patch-level or minor releases. I.e. the filter-cutoff would result in a virtual upper version bound. E.g. if version 2.1.1.2 was within the filter-cutoff range, then, all versions < 2.2 would be be included as well into the snapshot

PS2: additional addressing mode: besides index-serial-no & posix-timestamp: cutoff relative to package releases, i.e.: --index-snapshot foo-1.2.3 would denote the hackage-index state at the exact point at which foo-1.2.3 was first uploaded and entered the index.

[dcoutts]

I see, so the 01-index.cache is shared for all projects, and contains the info needed to do the filtering? So does that have enough info for a revision number too?

And so we need a separate 01-index.cache because the info provided by hackage-security only provides efficient access to the latest revision of everything. Producing all revisions requires walking through the whole 00-index.tar (which is presumably what you do to make that new cache).

[hvr]

@dcoutts

I see, so the 01-index.cache is shared for all projects,

yes, that way it only needs to be constructed once at cabal update time, and is available for projects as well as one-off cabal install-invocations... cabal list could also benefit from having access to this new metadata.

and contains the info needed to do the filtering? So does that have enough info for a revision number too?

the revision number is implicitly contained if we can assume that the first .cabal entry for a given key is x-revision 0, and each subsequent masking entry with the same key increments the x-rev by one. Just as the "serial-no" is implicitly contained as well (it's basically the line-number of the cache-file).

And so we need a separate 01-index.cache because the info provided by hackage-security only provides efficient access to the latest revision of everything.

yes

Producing all revisions requires walking through the whole 00-index.tar (which is presumably what you do to make that new cache).

yes, and especially that part of the code has a big TODO/FIXME attached as it's a bit ugly :-)

[ezyang]

Closing for now.

[hvr]

I'm back working on this...

[hvr]

continuing in #3604 since GitHub doesn't allow me to change the target branch for an existing PR