Application Aware Backups truncate database dumps 100M+

[ahyattdev]

Description

  annotations:
    k8up.io/backupcommand: bash -c 'set -euo pipefail; PGDATABASE="app" pg_dump --clean > /run/tmp/backup.sql; cat /run/tmp/backup.sql'
    k8up.io/file-extension: .sql

I first send the write to a file to ensure that the truncation isn't being done before k8up can see the output. The file at /run/tmp/backup.sql is complete, ending like this:

--
-- PostgreSQL database dump complete
--

The complete DB dump is 343MB on disk.

But the actually backed up file is incomplete at 340MB.

This suggests a bug when handling the final chunks of stdout from an Application Aware Backup.

The pre backup pod in K8up has no errors or indicators that these commands failed.

Additional Context

No response

Logs

[2026-03-07  08:33:14]	2026-03-07T13:33:14Z	INFO	k8up	Starting k8up…	{"version": "2.13.1", "date": "2025-10-03T07:31:20Z", "commit": "56ca1a38786778474be6260d7c90f65e40a3eff8", "go_os": "linux", "go_arch": "amd64", "go_version": "go1.23.12", "uid": 8888, "gid": 8888}
[2026-03-07  08:33:14]	2026-03-07T13:33:14Z	INFO	k8up.restic	initializing
[2026-03-07  08:33:14]	2026-03-07T13:33:14Z	INFO	k8up.restic	setting up a signal handler
[2026-03-07  08:33:14]	2026-03-07T13:33:14Z	INFO	k8up.restic	sleeping until init	{"duration": "3s"}
[2026-03-07  08:33:17]	2026-03-07T13:33:17Z	INFO	k8up.restic.restic.RepoInit.command	restic command	{"path": "/usr/local/bin/restic", "args": ["init"]}
[2026-03-07  08:33:17]	2026-03-07T13:33:17Z	INFO	k8up.restic.restic.RepoInit.command	Defining RESTIC_PROGRESS_FPS	{"frequency": 0.016666666666666666}
[2026-03-07  08:33:17]	2026-03-07T13:33:17Z	INFO	k8up.restic.restic.unlock	unlocking repository	{"all": false}
[2026-03-07  08:33:17]	2026-03-07T13:33:17Z	INFO	k8up.restic.restic.unlock.command	restic command	{"path": "/usr/local/bin/restic", "args": ["unlock"]}
[2026-03-07  08:33:17]	2026-03-07T13:33:17Z	INFO	k8up.restic.restic.unlock.command	Defining RESTIC_PROGRESS_FPS	{"frequency": 0.016666666666666666}
[2026-03-07  08:33:21]	2026-03-07T13:33:21Z	INFO	k8up.restic.restic.snapshots	getting list of snapshots
[2026-03-07  08:33:21]	2026-03-07T13:33:21Z	INFO	k8up.restic.restic.snapshots.command	restic command	{"path": "/usr/local/bin/restic", "args": ["snapshots", "--json"]}
[2026-03-07  08:33:21]	2026-03-07T13:33:21Z	INFO	k8up.restic.restic.snapshots.command	Defining RESTIC_PROGRESS_FPS	{"frequency": 0.016666666666666666}
[2026-03-07  08:43:56]	2026-03-07T13:43:56Z	INFO	k8up.restic.k8sClient	listing all pods	{"annotation": "k8up.io/backupcommand", "namespace": "rss"}
[2026-03-07  08:43:56]	2026-03-07T13:43:56Z	INFO	k8up.restic.k8sClient	adding to backup list	{"namespace": "rss", "pod": "rss-database-1"}
[2026-03-07  08:43:56]	2026-03-07T13:43:56Z	INFO	k8up.restic.k8sExec	executing command	{"command": "bash, -c, set -euo pipefail; PGDATABASE=\"app\" pg_dump --clean > /run/tmp/backup.sql; cat /run/tmp/backup.sql", "namespace": "rss", "pod": "rss-database-1"}
[2026-03-07  08:43:56]	2026-03-07T13:43:56Z	INFO	k8up.restic.restic.stdinBackup	starting stdin backup	{"filename": "/rss-postgres", "extension": ".sql"}
[2026-03-07  08:43:56]	2026-03-07T13:43:56Z	INFO	k8up.restic.restic.stdinBackup.command	restic command	{"path": "/usr/local/bin/restic", "args": ["backup", "--stdin-filename", "/rss-postgres.sql", "--host", "rss", "--json", "--stdin"]}
[2026-03-07  08:43:56]	2026-03-07T13:43:56Z	INFO	k8up.restic.restic.stdinBackup.command	Defining RESTIC_PROGRESS_FPS	{"frequency": 0.016666666666666666}
[2026-03-07  08:48:51]	2026-03-07T13:48:51Z	INFO	k8up.restic.restic.stdinBackup.progress	backup finished	{"new files": 1, "changed files": 0, "errors": 0}
[2026-03-07  08:48:51]	2026-03-07T13:48:51Z	INFO	k8up.restic.restic.stdinBackup.progress	stats	{"time": 290.991614567, "bytes added": 244525209, "bytes processed": 355663872}
[2026-03-07  08:48:51]	2026-03-07T13:48:51Z	INFO	k8up.restic.restic.MountCollector	stats mount dir doesn't exist, skipping stats	{"dir": "/data"}
[2026-03-07  08:48:51]	2026-03-07T13:48:51Z	INFO	k8up.restic.statsHandler.promStats	sending prometheus stats	{"url": "http://prometheus-prometheus-pushgateway.prometheus.svc.cluster.local:9091"}
[2026-03-07  08:48:52]	2026-03-07T13:48:52Z	INFO	k8up.restic	backups of annotated jobs have finished successfully
[2026-03-07  08:48:52]	2026-03-07T13:48:52Z	INFO	k8up.restic.restic.backup	starting backup
[2026-03-07  08:48:52]	2026-03-07T13:48:52Z	INFO	k8up.restic.restic.backup	backupdir does not exist, skipping. Sending snapshot list	{"dirname": "/data"}
[2026-03-07  08:48:52]	2026-03-07T13:48:52Z	INFO	k8up.restic.restic.snapshots	getting list of snapshots
[2026-03-07  08:48:52]	2026-03-07T13:48:52Z	INFO	k8up.restic.restic.snapshots.command	restic command	{"path": "/usr/local/bin/restic", "args": ["snapshots", "--json"]}
[2026-03-07  08:48:52]	2026-03-07T13:48:52Z	INFO	k8up.restic.restic.snapshots.command	Defining RESTIC_PROGRESS_FPS	{"frequency": 0.016666666666666666}

Expected Behavior

The Application Aware Backup is contains 100% of the data outputted by the backup command.

Steps To Reproduce

No response

Version of K8up

v4.8.6

Version of Kubernetes

v1.33.5+rke2r1

Distribution of Kubernetes

RKE2

[toabi]

Interesting. I'm currently researching why my postgresql dumps in custom format are incomplete. They are much smaller e.g. 4-5 MiB size. But sometimes just 0.5-1MiB is missing at the end. Until now I tought it's pg_dump failing and erroring. Maybe I'll also run it with sql format to look at the ending.

Update: I modified my script so it would fail if pg_dump is incomplete:

  backupCommand: >
    sh -ceu '
      tmp="$(mktemp ./pgdump.XXXXXX)"
      pg_dump \
        --format=plain \
        --no-owner \
        --no-acl \
        --dbname="$DB_CONNECTION_STRING" \
        --file="$tmp"
      if ! grep -q "^-- PostgreSQL database dump complete$" "$tmp"; then
        echo "ERROR: PostgreSQL dump did not complete successfully" >&2
        rm -f "$tmp"
        exit 1
      fi
      cat "$tmp"
    '

This does not fail BUT the resulting backups in the restic repository are still truncated.

[toabi]

Could it be that if restic runs into some connection issue when reading the stdin from the pod it actually raises a exit-code 3 and then that one gets ignored according to https://github.com/k8up-io/k8up/blob/v2.13.1/restic/cli/command.go#L121-L126 (just one theory)

[Kidswiss]

Interesting finding. We've never run into this yet.

I wonder, does it only happen with pg_dump? Maybe there's something in the bitstream that trips the stdout streaming for some reason?

@toabi you should be able to catch your exit code 3 theory via push gateway metrics. K8up will send more detailed information about a backup job there.

[toabi]

I setup a pushgateway and configured it in the schedule. But the pushed prometheus stats all looks happy. At least no errors.

But the pg dump which makes it to restic but is truncated… with the plain format it's quickly visible at least.

One thing I noticed though now: the push actually overwrites the other stats already pushed. I have two PBPs per app usually. One to create a file dump from s3 and another to create a database dump. In pushgateway I see it for a short while for my .pgsql file but then it get's overwritten with the .tar file stats. In prometheus I then only see the tar file. But I guess that's just another k8up bug related to that feature.

[toabi]

If somebody stumbles upon this issue too… what I now did because I don't want to fully dump my k8up setup:

I created a CronJob which dumps database and s3 bucket contents to a PVC and then in the main container creates a Backup CR instance to actually run the backup of the PVC.

Check and Prune are still done with the k8up schedule. This seems to work fine. Only missing feature is the nice @daily-random but the pro side is to have an easy way to trigger manual backups via create job --from.

[Kidswiss]

Do you guys maybe see something in the apiserver logs? Maybe the connection for the stdout streaming gets interrupted for some reason.

[rocketeerbkw]

TL;DR
Using a backupcommand that dumps the database to a file on disk first seems to fix it eg, dump > file.sql && cat file.sql

We are also experiencing this issue with MariaDB. Same as OP, there is no indication in the backup pod logs that anything is failing.

The k8up.io/backupcommand that was resulting in truncated dump files was:
/bin/sh -c "mysqldump --max-allowed-packet=500M --events --routines --quick --add-locks --no-autocommit --single-transaction --all-databases"

We saw this on databases that were gigabytes as well as 10s of megabytes, so final file size didn't seem to matter. Anecdotally, it seemed to fail in the middle of table dumps that had hundreds-of-thousands or millions of rows.

We never had this issue with k8upv1 (backup.appuio.ch), it only started happening when we switched to k8upv2 (backup.k8up.io).

Adjusting max-allowed-packets and other parts of the dump command never resulted in a good dump. We were able to fix it (so far) by changing the backup command so that it first writes to disk before printing to stdout:
/bin/sh -c "mysqldump --max-allowed-packet=500M --events --routines --quick --add-locks --no-autocommit --single-transaction --all-databases > /var/lib/mysql/backup/k8up-backup.sql && cat /var/lib/mysql/backup/k8up-backup.sql"

Also anecdotal, but we have different systems that template out a k8up backup. For the majority of our namespaces, we were already dumping to a file before sending to stdout, and it appears that these are not truncated VS the namespaces where we dumped directly to stdout got truncated.

[arska]

Fix submitted in #1169 — the pipe writer is now closed before signaling done, ensuring all data reaches restic's stdin. The fix also preserves the hard-fail behavior on stream errors.