scala-ci.typesafe.com/artifactory is full? PR validation jobs are giving 413 errors

[SethTisue]

so e.g. at https://scala-ci.typesafe.com/job/scala-2.12.x-validate-main/4276/console:

[error] (partest / publish) java.io.IOException: PUT operation to URL https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots;build.timestamp=1603922141869/org/scala-lang/scala-partest/2.12.13-bin-bfc824a-SNAPSHOT/scala-partest-2.12.13-bin-bfc824a-SNAPSHOT.pom failed with status code 413: Request Entity Too Large; Response Body: {
[error]   "errors" : [ {
[error]     "status" : 413,
[error]     "message" : "Datastore disk usage is too high. Contact your Artifactory administrator to add additional storage space or change the disk quota limits."
[error]   } ]
[error] }

[SethTisue]

iirc, last time it happened we dealt with this by zeroing out https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/ — or did we not zero it out entirely, but just deleted the oldest builds? I can't remember

the PR validation snapshots do have some value even after a PR is merged, since they enable per-commit bisecting of regressions. (in contrast to the mergelies at https://scala-ci.typesafe.com/artifactory/scala-integration/ , which enable coarser-grained bisecting: per PR. the mergelies, we intend to retain indefinitely)

[SethTisue]

aha, I found the ticket from the last time this happened: #636

[SethTisue]

recent article (October 1, 2020) with advice: https://jfrog.com/knowledge-base/artifactory-cleanup-best-practices/

[SethTisue]

I went through all the remote repositories and set them to expire cached artifacts after 720 hours (30 days), then I ran "Cleanup Unused Cached Artifacts", but this is unlikely to buy us more than a small amount of time, as https://scala-ci.typesafe.com/artifactory/webapp/#/admin/advanced/storage_summary shows that the lion's share of storage is going to the PR validation snapshots and mergelies

[lrytz]

I'll run my script (#636 (comment)) to delete from pr-validation-snapshots what's older than 2019.

[lrytz]

Seems we already deleted what's older than 2019, and half of 2019 too. So I'm going for everything artifacts for non-merged commits older than 2020.

[lrytz]

🍿

[lrytz]

This didn't help enough, we're still at 85%.

I noticed

• 28 gigs in scala-release-temp. I think we used this at some point for temporary builds, eg for benchmarking, to avoid putting them in scala-integration. scala/scala@6ff389166e. We can probably clean that up.
• Artifactory's internal database takes 67 gigs (du -h /var/opt/jfrog/artifactory/data/derby). There's a function to compress that (https://www.jfrog.com/confluence/display/JFROG/Regular+Maintenance+Operations#RegularMaintenanceOperations-Storage), after trying that a few times with an error message it eventually worked, but it didn't help. Still 67 gigs.
• I can delete more PR validation builds, that repo is at 110 gigs
• scala-integration is at 207 gigs and we probably don't ever want to remove from there

[lrytz]

@SethTisue let me know what you think. IMO we can also bump the EBS volume size.

[SethTisue]

I've never used scala-release-temp or seen it used, so I have no objection to zeroing that one out.

IMO we can also bump the EBS volume size

Can that be done without a lot of rebuilding effort?

[lrytz]

Resizing the EBS is simple, then i'll give the resize2fs command a try