better handle output

[skorzennik]

The way our users use SPAdes creates millions of temporary(?) files, which causes problems on some filesystems (like GPFS) that have build-in limits on # files per directory. Deleting these temp files when teh analysis is done is also time and resources consuming. This seems like a design flaw to me.

1- are you considering a way to consolidate the s/w to avoid creating such a huge number of files and/or deleting the ones not needed any longer as the analysis proceeds?

2- is there a way to tell SPAdes to clean up this mees when done?

Thanks,
Sylvain Korzennik, Ph.D. HPC analyst, Smithsonian Institution.

[asl]

SPAdes does not create millions of temporary files. Usually this is something like 16*number of threads used. All temporary files are deleted after last use. Certainly, if the SPAdes process is killed somehow, then there is no way to remove these files afterwards and it is users responsibility to cleanup.

Will you please clarify which files were not removed, what is the environment / command line options ideally along with spades.log file from the corresponding run?

[skorzennik]

Hi Anton, We ran a test case and the number of temporary files was <2000 until the "Mismatch correction started" phase. It went up to around 5.7 million files during that phase. The --careful option adds mismatch correction. Attached is the spades.log file. You will see that the test used System information: SPAdes version: 4.2.0 Python version: 3.14.3 OS: Linux-4.xx.x-xxxxxxxx.el8_10.x86_64-x86_64-with-glibc2.28 and the very large number of files seems to be associated w/ the use of the --careful option. Cheers, Sylvain --

…

On Thu, Feb 26, 2026 at 4:26 PM Anton Korobeynikov ***@***.***> wrote: *asl* left a comment (ablab/spades#1560) <#1560 (comment)> SPAdes does not create millions of temporary files. Usually this is something like 16*number of threads used. All temporary files are deleted after last use. Certainly, if the SPAdes process is killed somehow, then there is no way to remove these files afterwards and it is users responsibility to cleanup afterwards. Will you please clarify which files were not removed, what is the environment / command line options ideally along with spades.log file from the corresponding run? — Reply to this email directly, view it on GitHub <#1560 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC5QY2VKGZ3T5JFNQ33ZOGD4N5QHHAVCNFSM6AAAAACWBBDXF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSNRZGMZDEOBQGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[asl]

@skorzennik Looks like no spades.log was attached.

[skorzennik]

The log file was attached in the email reply, but looks like github ignored it - I've uploaded it, using github.

spades.log

[mkweskin]

Hi, I work with Sylvain and support biologists who use our HPC.

Looking at the temp directory during the 'Mismatch correction' phase of this --careful assembly, the cause of the large number of files is because a separate fasta files is created for each of 200k+ contigs during this phase. I didn't track how this then increases to ~5.7 million files, presumably there are intermediate output files being produced for each contig.

For some additional context, spades is often used by biologists at our institution for the assembly of low-coverage short-read eukaryotic genomic sequence data. The goal is assembly of the mitochondrial (or chloroplast) genome, with no intention to recover a full genome assembly. Another use is with target-enriched sequencing for the recovery of known loci using the Phyluce pipeline.

[asl]

Looking at the temp directory during the 'Mismatch correction' phase of this --careful assembly, the cause of the large number of files is because a separate fasta files is created for each of 200k+ contigs during this phase. I didn't track how this then increases to ~5.7 million files, presumably there are intermediate output files being produced for each contig.

Indeed, mismatch corrector that is run in --careful mode splits the input contig file into lots of intermediate files, essentially one per contig. However, the real problem here is why there are 200k+ contigs? As far as I can see, assembly is done in single cell mode. Single cell mode aimed for MDA-amplified single cell bacterial data. As a result, given the expected genome size, it is highly unlikely that assembly will end in 200k+ contigs.

Was the data MDA-amplified? Judging from the log, the data in question is SRR17967979. Was it MDA-amplified single-cell data? Highly unlikely. As a result, I would strongly suggest not to assemble it in single-cell mode, as this mode is specifically tailored over various sequencing artifacts that are specific to MDA. It might be over-agressive for any other data resulting into suboptimal assembly. On the other hand, in single-cell mode SPAdes cannot rely on various coverage heuristics, therefore it could miss lots of simplification opportunities on "normal" data.

To summarize, I would strongly recommend not to use single-cell mode for this kind of data as it simply not appropriate. My understanding is that mitochondrial genome part would likely be of much higher coverage compared to the host "contamination" (is it so?), therefore I would probably suggest trying isolate mode maybe even with additional coverage threshold to remove host data (note that automatic threshold determination might fail here). If the coverage is enough, then you'd likely skip the built-in read error correction and you really do not need mismatch correction (--careful). Your mileage may vary though.