Local lakefs auto attach existing s3 repos?

[pyretttt]

I am using simplified version of compose file from samples repository. As I observed when containers are relaunched, lakefs can't automatically attach existing s3 repositories. It refuses to create repo (with lakectl repo create), because it already exists in s3.

In documentation I found refs dump, but I find it tough to use, because it requires manual launching or dumping with intervals.

So Is there some option to list repositories, that exist on s3, so that lakefs can automatically attach them? Something like lakectl repo attach s3://data lakefs://data

[pyretttt]

Summon @arielshaqed for help

[pyretttt]

Just took a look at create bare it creates repository, and s3 data is preserved, but looks like it can't hook up refs, because admin panel shows ref not found

[pyretttt]

Okay I got it needed to add volumes such as
- /lakefs/metadata:/metadata
- /lakefs/lakefs:/lakefs

And probably these two:
- LAKEFS_DATABASE_LOCAL_PATH=/metadata
- LAKEFS_DATABASE_LOCAL_SYNC_WRITES=true

[arielshaqed]

Hi @pyretttt ,
Sorry I missed this. We usually track and answer questions on our Slack channels more than we manage to do for GitHub discussions.
I think your resolution makes sense. You do need your KV to survive restarts.
However do note that a local KV is probably a poor fit for any production usage. I cannot emphasize too much how important it is to keep both KV and blockstore on reliable storage. Unless you have a pretty serious NFS holding it, neither local storage nor local KV will be reliable! And local storage will never scale out. Disks do stop working.

If you already have your storage on S3, you should seriously consider going all AWS and putting your KV on DynamoDB.