Improve performance of generic repo

[elprans]

The APT and RPM repos work on a locally-synced copy of their S3 prefixes
(because the repo tooling expects repos to be in a local fs), but the
generic repo currently just reads/writes things to S3 directly. This is
problematic for atomicity and performance reasons.

Fix this by doing the S3 sync for the generic repo as well. While at
it, lower log levels on some things to reduce noise.

This was tested locally (running process_incoming.py on artifacts
produced with local gel-pkg runs).

[Copilot]

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

server/process_incoming.py:749

• Verify that the 'tf.extract' function supports the 'filter' parameter; otherwise, this change could lead to extraction issues or compatibility problems.

tf.extract(member, staging_dir, filter="data")

server/process_incoming.py:383

• It may be safer to verify that the metadata file exists before attempting to open it, to avoid potential runtime errors if the expected file is missing.

with open(f"{metadata_file}.metadata.json") as f:

[Copilot]

Consider adding exception handling around 'filename.unlink()' to ensure that a failure during file deletion does not crash the application unexpectedly.

Suggested change

	filename.unlink()
	try:
	filename.unlink()
	except FileNotFoundError:
	logger.warning("File not found during deletion: %s", filename)
	except PermissionError:
	logger.error("Permission denied during deletion: %s", filename)
	except Exception as e:
	logger.error("Unexpected error during deletion of %s: %s", filename, e)