catchpoints: more support for EnableCatchpointsWithOnlineAccounts

[cce]

Summary

Follow-on to #6177.

When writing catchpoint files, the catchpointFileWriter currently does not have any access to the consensus parameters, and so does not know if EnableCatchpointsWithOnlineAccounts is set. This means catchpoints files may contain chunks with OnlineAccountRecordV6 and OnlineRoundParamsRecordV6 even when EnableCatchpointsWithOnlineAccounts is not set. However these objects are ignored when calculating the label — the catchpoint label hash calculation is conditioned on EnableCatchpointsWithOnlineAccounts.

This adds an argument to makeCatchpointFileWriter so that catchpointFileWriter knows what the current consensus version is.

This also adds support to catchpointdump for analyzing and dumping the onlineaccount and onlineroundparams records in catchpoint files, and calculating the labels.

This also addresses a corner case when the state proof recoverability system (from #4803) tells the onlineaccounts tracker to retain more than 320 rounds of history (set by votersTracker.lowestRound()) and used here:

    maxOnlineLookback := basics.Round(ao.maxBalLookback())
    dcc.onlineAccountsForgetBefore = (dcc.newBase() + 1).SubSaturate(maxOnlineLookback)
    if dcc.lowestRound > 0 && dcc.lowestRound < dcc.onlineAccountsForgetBefore {
        // extend history as needed
        dcc.onlineAccountsForgetBefore = dcc.lowestRound
    }

In this case, catchpoint files will contain more than the expected 320 rows, and lead to catchpoint label hash mismatch if catchpoint-generating nodes have differing opinions of when the last state proof was verified. In practice, this can really only occur when a node is catching up quickly (after being stopped and restarted, or starting from 0) and flushing large batches of rounds — it might not have verified the most recent state proof when it hits the catchpoint first stage snapshot round.

Test Plan

Updated TestExactAccountChunk to exercise the new consensus params argument, can try updating other similar tests.

Needs a new test where dcc.lowestRound is set to something older than (dcc.newBase()+1).SubSaturate(320), and verifies the excludeBefore argument works.

[codecov]

Codecov Report

Attention: Patch coverage is 17.94872% with 128 lines in your changes missing coverage. Please review.

Project coverage is 51.87%. Comparing base (a1137a2) to head (69be46b).
Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
cmd/catchpointdump/file.go	0.00%	56 Missing ⚠️
ledger/store/trackerdb/sqlitedriver/kvsIter.go	0.00%	36 Missing ⚠️
cmd/catchpointdump/database.go	0.00%	6 Missing ⚠️
cmd/catchpointdump/net.go	0.00%	6 Missing ⚠️
ledger/store/trackerdb/sqlitedriver/accountsV2.go	0.00%	6 Missing ⚠️
ledger/catchpointfilewriter.go	73.33%	2 Missing and 2 partials ⚠️
ledger/catchpointtracker.go	80.00%	3 Missing and 1 partial ⚠️
...edger/store/trackerdb/sqlitedriver/sqlitedriver.go	0.00%	4 Missing ⚠️
ledger/catchupaccessor.go	33.33%	2 Missing ⚠️
ledger/store/trackerdb/dualdriver/dualdriver.go	0.00%	2 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6214      +/-   ##
==========================================
- Coverage   51.91%   51.87%   -0.05%     
==========================================
  Files         643      643              
  Lines       86234    86349     +115     
==========================================
+ Hits        44769    44794      +25     
- Misses      38599    38688      +89     
- Partials     2866     2867       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gmalouf]

Shared some questions