Using Rustic with large, immutable files

[s-leroux]

Hi,

I knew about this project through the Restic project discussion forum. The one killer feature for me is support for cold storage.

I am now considering Rustic as a Debian Linux file server backup solution.

The server hosts our users’ home directory and a few TB of raw AVCHD footage. Both require external backup, ideally on Amazon S3, Amazon Glacier, Google Cloud, OVH Cloud, or any other reliable and cost-effective data storage provider.

• The users’ home directory contains the usual file types (office documents, git repositories, source code files, photos, videos). The total size is in the 1 TB range, and as a crude estimate, 1% of that is updated monthly. We need to maintain a snapshot history of a few months.
• The footage directory tree is different: it is mostly a write-once, read-many directory, approaching now 10 TB. Bearing exceptional cases, footage stored there is never updated and rarely deleted. In fact, due to bit rot on the local disks, we will likely need to retain the initial file’s content in the snapshots indefinitely.

I plan to set up an automatic weekly backup of the users' data. But monthly backups and occasional manual backups are sufficient for the footage directory.

Questions:

1. Would Rustic be a suitable solution given our use case? You understand that I want to minimize storage costs, so cold storage support is a premium for us.
2. May I run two instances of Rustic concurrently, targeting the same S3/S3 Glacier bucket, or do you suggest creating a separate pair of (hot, cold) buckets for the users' data and footage backups?

Regards,
🤝 Sylvain

[kapitainsky]

re 1 I recommend this read:

https://archive.ph/jGA4r and https://archive.ph/9ZUTQ

re 2 - yes. rustic supports concurrent operations. So it is really up to you how you want your data arranged. Either approach will work

[aawsome]

About the second question: If you use a single repository, you'll benefit from deduplication between the data and a more simple setup. If you use two repositories, you'll benefit from smaller repository sizes (which lead to a bit faster operations and less memory usage).
Note that rustic supports configuration profiles https://github.com/rustic-rs/rustic/tree/main/config which can help in both cases.

[s-leroux]

re 1 I recommend this read:

https://archive.ph/jGA4r and https://archive.ph/9ZUTQ

Thanks @kapitainsky .
Indeed, I used these resources, along with the official documentation, to set up rustic on S3 with cold storage.

About the second question: If you use a single repository, you'll benefit from deduplication between the data and a more simple setup.

I was exactly about to ask for that clarification, Alex. So, not only may I have concurrent instances of rustic targeting the same repository (same bucket and same path on S3), but I also benefit from data deduplication between them. Deduplication is probably not of great help for our AVCHD data, but for the user's home directories, it might matter.

Regarding the configuration profiles, I already used them based on the instructions provided in the resources quoted by Kapitansky above. I must admit I didn't study in detail all the available config options though.

An interesting, yet theoretical, follow-up would consider config sharing the same cold repository but different hot repositories (or the opposite). Would that work? Does that have any practical usage?