PATCH(JSON) corrupts long UTF-8 Japanese text for issues and issue comments on v1.25.5

[codemafia0000]

Summary

Updating an existing issue or issue comment with PATCH + application/json can corrupt long UTF-8 Japanese text on Gitea 1.25.5.

I reproduced this on:

• PATCH /api/v1/repos/{owner}/{repo}/issues/{index}
• PATCH /api/v1/repos/{owner}/{repo}/issues/comments/{id}

What works

• POST with application/json
• PATCH with application/x-www-form-urlencoded

What breaks

• PATCH with application/json

Reproduction

1. Create an issue or an issue comment.
2. Prepare a long Japanese body, for example repeating 日本語更新テスト many times.
3. Update the existing issue or issue comment with PATCH and Content-Type: application/json.
4. Read it back from the API or DB.

Expected

The updated body should preserve the UTF-8 Japanese text exactly.

Actual

Some characters are corrupted and become replacement characters like ��� / U+FFFD.

Additional notes

• I could reproduce this on a local Gitea 1.25.5 instance.
• I could not reproduce corruption when sending the same update as application/x-www-form-urlencoded.
• I also could not reproduce corruption when creating a new comment with POST + application/json.

Gitea Version

1.25.5

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.35.1

Operating System

macOS 14.5

How are you running Gitea?

Running a local Gitea instance from a command-line binary on macOS 14.5.
Gitea version: 1.25.5
Git version: 2.35.1
The issue was reproduced against a local instance, not on demo.gitea.com.

Database

SQLite

[wxiaoguang]

It's likely related to your reverse proxy or firewall (WAF)

Can you provide reproducible setup with detailed steps? For example:

1. use the docker compose file, docker compose up
2. create a XXX user and YYY repo
3. use the curl .... command to send the request
4. check the result ....

[codemafia0000]

Thanks for the suggestion. I investigated further, and in my local setup I can reproduce this without any reverse proxy or WAF.

My local Gitea instance is running directly as the gitea web binary and listening on 127.0.0.1:39123 with:

• HTTP_ADDR = 127.0.0.1
• HTTP_PORT = 39123
• ROOT_URL = http://localhost:39123/

So this is direct loopback HTTP, without nginx/traefik/caddy in front.

I also re-tested with a minimal curl reproduction against http://127.0.0.1:39123/api/v1/..., and I can reproduce the corruption there as well.

Reproduction

set -euo pipefail

TOKEN='YOUR_TOKEN_HERE'
API='http://127.0.0.1:39123/api/v1'
OWNER='admin'
REPO='YOUR_EXISTING_REPO'

response="$(
  python3 - <<'PY' | curl -sS \
    -H "Authorization: token $TOKEN" \
    -H 'Content-Type: application/json' \
    --data-binary @- \
    "$API/repos/$OWNER/$REPO/issues"
import json
body = '日本語更新テスト ' * 320
print(json.dumps({'title': 'utf8 repro', 'body': body}, ensure_ascii=False))
PY
)"

python3 - <<'PY' "$response"
import json, sys
data = json.loads(sys.argv[1])
body = data["body"]
print("contains_replacement=" + ("yes" if "\ufffd" in body else "no"))
print("body_chars=" + str(len(body)))
if "\ufffd" in body:
    i = body.index("\ufffd")
    print("snippet=" + body[max(0, i-20):i+40])
PY

Output from my local run

issue_number=2
contains_replacement=yes
body_chars=2884
snippet=ト 日本語更新テスト 日本語更新テスト ���本語更新テスト 日本語更新テスト 日本語更新テスト 日本語更新テスト 日本

The returned JSON body itself already contains corrupted text such as:

���本語更新テスト
日本語更新テス���

So this does not seem to be only a UI rendering issue.

Also, I need to correct my earlier report: in my local 1.25.5 setup this is not limited to PATCH. I can reproduce it with long Japanese application/json request bodies more generally.
I also checked the SQLite database directly, and the corruption is already present in the stored issue.content value for the reproduced issue rows. So this does not seem to be only an API response or UI rendering problem.

The key conditions seem to be:

direct request to local Gitea, without reverse proxy
Content-Type: application/json
long enough UTF-8 Japanese text
similar-length ASCII text did not reproduce in my tests

Environment

• Gitea version: 1.25.5
• Git version: 2.35.1
• OS: macOS 14.5
• Database: SQLite3
• Running method: local gitea web binary
• Direct bind: 127.0.0.1:39123
• No reverse proxy / WAF in front of Gitea

[wxiaoguang]

Framework bug .........

func TestGoccy(t *testing.T) {
	type Struct struct {
		Title string `json:"title"`
		Body  string `json:"body"`
	}

	ln, err := net.Listen("tcp", "127.0.0.1:3000")
	require.NoError(t, err)

	t.Log("Listen on", ln.Addr())
	wg := sync.WaitGroup{}
	wg.Go(func() {
		_ = http.Serve(ln, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			var out *Struct
			require.NoError(t, goccy_json.NewDecoder(r.Body).Decode(&out))
			require.NoError(t, err)
			require.NotContains(t, out.Body, "\uFFFD") // BUG! Invalid UTF-8 character
		}))
	})
	time.Sleep(100 * time.Millisecond)
	body := strings.Repeat("日本語更新テスト ", 320)
	input := `{"title": "utf8 repro", "body": "$BODY"}`
	input = strings.ReplaceAll(input, "$BODY", body)
	_, err = http.Post("http://127.0.0.1:3000", "application/json", bytes.NewBufferString(input))
	wg.Wait()
}

[wxiaoguang]

Created upstream issue goccy/go-json#574

[wxiaoguang]

And our side: Fix corrupted JSON caused by goccy library #37214

[codemafia0000]

Thanks for investigating this and reducing it to a minimal repro.

I appreciate you opening the upstream issue and preparing the Gitea-side fix.