Error migrating repo-avatars to minio

[Infinoid]

Description

I am testing the process of moving from local storage to AWS S3. I did this on a test machine:

for DATATYPE in attachments lfs avatars repo-avatars repo-archive packages actions_log actions_artifacts; do
    echo DATATYPE=$DATATYPE;
    gitea migrate-storage \
        --type=$DATATYPE \
        --minio-base-path=$DATATYPE \
        --storage=minio \
        --minio-endpoint=s3.us-east-1.amazonaws.com \
        --minio-access-key-id=xxxx \
        --minio-secret-access-key=yyyy \
        --minio-bucket=my-gitea \
        --minio-location=us-east-1 \
        --minio-use-ssl=true
done

This mostly worked. I am ignoring the "unsupported storage" errors for things that do not support minio... that's fair.

The problem is repo-avatars. Normal avatars migrated fine (and so did attachments and lfs), but repo-avatars gave me this error:

2024/01/08 09:03:35 ...les/storage/minio.go:86:NewMinioStorage() [I] Creating Minio storage at s3.us-east-1.amazonaws.com:my-gitea with base path repo-avatars
Command error: Put "https://my-gitea.s3.dualstack.us-east-1.amazonaws.com/repo-avatars": read /data/gitea/repo-avatars: is a directory

Yes, it's a directory, but that shouldn't be a problem. Right?

Config:

[picture]
ENABLE_FEDERATED_AVATAR = false
AVATAR_UPLOAD_PATH            = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars

The error message refers to the current storage path on the local disk, so here's what those folders look like:

gitea:/$ ls -ld /data/gitea/avatars /data/gitea/repo-avatars
drwxr-xr-x    3 git      git           4096 Nov 28 22:49 /data/gitea/avatars
drwxr-xr-x    3 git      git           4096 Apr 28  2023 /data/gitea/repo-avatars

Gitea Version

1.21.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/Infinoid/7bd96c126ad1245b013940de6fac118d

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

using official docker image with tag 1.21

Database

SQLite

[Infinoid]

I looked around to see what is different between user avatars and repo avatars. One difference is that the repository.avatar column contains empty strings, rather than NULLs.

The user.avatar column does not contain empty strings, because gitea has generated random geometrical images for each user.

mysql> select avatar from user where avatar is not null limit 3;
+----------------------------------+
| avatar                           |
+----------------------------------+
| 252d991d0b37525e1e80dbeeec1df56b |
| 6e78e0c1a67452d7f95b8429d3d091fb |
| f08b5ff8605831c052f976009f50702b |
+----------------------------------+
3 rows in set (0.00 sec)

mysql> select avatar from repository where avatar is not null limit 3;
+------------------------------------+
| avatar                             |
+------------------------------------+
| 5-e333e5f0e87d28cd771a480129b928c3 |
|                                    |
|                                    |
+------------------------------------+
3 rows in set (0.00 sec)

So there are empty avatar strings in the repository table but not the user table. I think this may explain why I was able to migrate storage for user avatars, but not repository avatars.

I think maybe the code is looking for non-NULL avatars, then concatenating the strings "/data/gitea/repo-avatars" + "/" + "", then opening the result, expecting it to be a file?

[lunny]

But migrating storage will not read or update database. It will only copy files from one place to another.
It will read database.

[Infinoid]

But migrating storage will not read or update database.

Hmm... are you sure? I think this db.Iterate call is where it gets confused.

[github-actions]

Automatically locked because of our CONTRIBUTING guidelines