GC is not trying hard enough

[rojer]

For reasons discussed in #135 and also as the last step of in-place encryption, i am performing full GC of the filesystem and would like to guarantee that all the deleted pages have been erased.
For some reason, first pass of GC on a well-littered filesystem returns after only a few relocations.
Here is the output of this loop:

spiffs_vfs_gc_all    GC result -10001 del pages 319 -> 270
spiffs_vfs_gc_all    GC result -10001 del pages 270 -> 222
spiffs_vfs_gc_all    GC result -10001 del pages 222 -> 174
spiffs_vfs_gc_all    GC result -10001 del pages 174 -> 128
spiffs_vfs_gc_all    GC result -10001 del pages 128 -> 80
spiffs_vfs_gc_all    GC result -10001 del pages 80 -> 53
spiffs_vfs_gc_all    GC result -10001 del pages 53 -> 39
spiffs_vfs_gc_all    GC result -10001 del pages 39 -> 35
spiffs_vfs_gc_all    GC result -10001 del pages 35 -> 30
spiffs_vfs_gc_all    GC result -10001 del pages 30 -> 30

Here is a more detailed output, with SPIFFS_vis before each GC invocation.

Looking at the final state, i'm wondering: (1) why did it take so long to reach it and (2) why does it still have deleted pages?

[pellepl]

Regarding our first question there's a cap limiting how many blocks a gc can tidy up in one loop, SPIFFS_GC_MAX_RUNS. This is to ensure that the gc will not try forever, and to roughly meet a given realtime requirement (though realtime meaning in this case can be questioned)

The second question is due to that data pages in a block being gc:d may be referred from another metadata page in another block. Once those pages have been moved, the metadata page must be deleted and rewritten, thus adding up to more garbage after the cleaning.

Once I get time to rewrite spiffs to address blocks indirectly, this update of metadata will not be needed (though at the cost of say 16 bits of ram per block comprised by the filesystem). This will be a major rewrite, but will solve a lot of issues regarding almost full filesystem and will also theoretically make it more robust against intermittent power losses. Oh, and the fs check will be rewritten in this overhaul also, by introducing a more precise fixup and a log page.

[sensslen]

We're currently also encountering a similar problem. We're writing a File of size 15522 bytes to a Flash of 8MB size. In a Test we just reqrote the File in a loop and after around 500 tries the write fails due to the Garbage Collector using too many iterations.

Is there a measure to calculate what number SPIFFS_GC_MAX_RUNS should be?

The File size is defined by a external Library we're using.

[pellepl]

Hi, please see this for a more thorough explanation of SPIFFS_GC_MAX_RUNS. Hope this helps.

[sensslen]

Hi, thanks for the link to the doc.

Unfortunately I already checked there and increasing the number of SPIFFS_GC_MAX_RUNS jus helps partly.

With the Value set to 5, I'm able to rewrite my File for 503 times. With the value set to 100 it fails after 510 writes, which seems strange, as 16kb is way less than our 8MB flash we're using.

[pellepl]

Ok, hmmm, I'll try to reproduce it here. Den 12 maj 2017 11:23 skrev "sensslen" <notifications@github.com>:

…

Hi, thanks for the link to the doc. Unfortunately I already checked there and increasing the number of SPIFFS_GC_MAX_RUNS jus helps partly. With the Value set to 5, I'm able to rewrite my File for 503 times. With the value set to 100 it fails after 510 writes, which seems strange, as 16kb is way less than our 8MB flash we're using. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#139 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE3NVw_xP0hW2TRdIMgvj5jZfB8q7rQJks5r5CUEgaJpZM4MzeFe> .

[mariankr]

Maybe this problem is related to my problem with seek.

TEST(seek_bug) {
  fs_reset_specific(0, 0, 8*1024*1024, 4096, 4096, 256);
  int tries = 0;
  do
  {
    spiffs_file fd = SPIFFS_open(FS, "EVENT", SPIFFS_CREAT | SPIFFS_RDWR, 0);
    if (SPIFFS_lseek(FS, fd, 0, SEEK_END) < 0)
    {
      printf("try number: %i\n", tries);
      printf("seek errno %i\n", SPIFFS_errno(FS));
      return SPIFFS_errno(FS);
    }
    static unsigned char buf[1] = {}; // data doesn't matter
    if (SPIFFS_write(FS, fd, &buf, sizeof(buf)) < 0) printf("errno %i\n", SPIFFS_errno(FS));
    SPIFFS_close(FS, fd);
  } while (tries++ < 1000000);
  SPIFFS_vis(FS);
  return TEST_RES_OK;
} TEST_END

after 26355 try i have seek error -10002

[sensslen]

I'm very sorry, but it is a false alarm! I forgot to move the file position back to zero, so what I did was to extend the File.... Obviously this leads to a full file system pretty quickly.