Improve Allocator interface so client can avoid needless copying

[BarabasGitHub]

In my opinion the current Allocator interface is suboptimal. Mainly because it does more than just allocate memory, it will also copy data for you. I think it's important to get something like this right, as it will be used pretty much everywhere. I'm giving my opinion because I'd like to see it done right, especially in Zig, which seems to get so many things right already.

Currently the Allocator interface is as follows (I left out some documentation to keep it shorter):

    /// Realloc is used to modify the size or alignment of an existing allocation,
    /// as well as to provide the allocator with an opportunity to move an allocation
    /// to a better location.
    reallocFn: fn (
        self: *Allocator,
        old_mem: []u8,
        old_alignment: u29,
        new_byte_count: usize,
        new_alignment: u29,
    ) Error![]u8,

    /// This function deallocates memory. It must succeed.
    shrinkFn: fn (
        self: *Allocator,
        old_mem: []u8,
        old_alignment: u29,
        new_byte_count: usize,
        new_alignment: u29,
    ) []u8,

I have issues with the semantics of reallocFn (shrinkFn is fine). It is essentially the same as the C library realloc function in that it can both modify (grow/shrink) the existing memory area, or allocate a totally new area and copy the data to there. The last part is where I think this is suboptimal. There are many cases where copying 'the data' is not what the caller wants.

1. Consider inserting an element in the middle of an array. In this case copying 'the data' includes all elements while actually only the elements up until the insertion point need to be copied, the rest needs to be moved up one element. Using the current reallocFn semantics, all elements after the insertion point will end up being copied twice.
2. Similar to 1, but worse is when a matrix is stored in one memory block row-by-row and a column needs to be appended (or inserted). The current reallocFn will copy all elements, while actually the rows need to be copied separately to slightly different locations. In addition to having to move the rows, the resulting code will be more complicated, because now the new memory comes already filled with the rows and the old memory block is no longer available. This means that the elements in the rows have to be copied backwards (starting from the end).
3. Consider an array which has capacity for 100 elements, but only has a few in use. Appending 100 new elements will require reallocation and reallocFn will happily copy all the 100 elements to the new memory, while actually only a few need to be copied and the rest is undefined data.
4. Though probably less likely, but if there is a data structure which refers to other nodes but stored in one block of memory the copying/moving done by reallocFn is completely useless and will actually destroy the data-structure.

What I would like to see is in the basis an allocate and free, with on top a realloc which only tries to modify the existing memory block and fails if that's not possible. In no case should it try to be smart and move data around, that's better left to the user, who has more knowledge and can move the data to it's correct position.

Please let me know what you think.

[ikskuh]

I think you have a misunderstanding on what realloc is actually doing.

The data copy is only necessary if the allocator cannot expand the current memory object. Then it is forced to relocate that memory object (move it to a completly different address, not sharing a single byte of memory with the previous location) where there is enough space to store the requested data size.

This also means that you are required to copy all bytes from the previous location to the new one as the new location has no defined memory content.

Your description of points 1. … 4. are all insert operations that operate on a fixed memory block. If you actually need to resize the data and the allocator is required to relocate the memory block, you alway have to copy all bytes, otherwise you result with invalid data.

[BarabasGitHub]

No, I understand that perfectly and I believe that's not the job of the allocator to do that for the reasons I stated above. There are many situation where it's not just a blind copy and you actually want to have the data in the new memory positioned slightly differently.

The data copy is only necessary if the allocator cannot expand the current memory object. Then it is forced to relocate that memory object (move it to a completly different address, not sharing a single byte of memory with the previous location) where there is enough space to store the requested data size.

No. I am saying it should fail in that case, because copying the data can be expensive and blindly copying everything might not actually be what you want.

Your description of points 1. … 4. are all insert operations that operate on a fixed memory block.

3 and 4 aren't. But yes if you're just appending one by one to a list/array then it just so happens that it's doing the right thing. In all the other cases probably not.

If you actually need to resize the data and the allocator is required to relocate the memory block, you alway have to copy all bytes, otherwise you result with invalid data.

Copy all the bytes, yes probably. But not necessarily in the exact same layout. And sometimes not even all the bytes, because some are unused or maybe some need to change.

[mikdusan]

What I would like to see is in the basis an allocate and free, with on top a realloc which only tries to modify the existing memory block and fails if that's not possible. In no case should it try to be smart and move data around, that's better left to the user, who has more knowledge and can move the data to it's correct position.

Just to be clear, do you want a realloc that never changes "existing" ptr absolute value but gives the caller a blessing that says "yup, it's been resized now" or "no, can't resize it", presumably the latter only if larger is not possible given ptr is anchored. ?

[BarabasGitHub]

Yes, that's what I meant.

presumably the latter only if larger is not possible given ptr is anchored. ?

Not entirely sure what you mean here, but I think yes. If it can't change the allocated size (and maybe smaller is always possible?) it should tell the caller "no can't resize it".

[marler8997]

I ran into this exact same situation in my alternative standard library for D (https://github.com/dragon-lang/mar). It's pretty easy to solve though. Just have the underlying reallocFn function not perform any copy. The caller will know whether the memory was moved based on the return value. This small change allows both the current behavior and for other caller's to implement their own "copy on move" mechanisms.

To implement the current behavior is as simple as creating a function that takes an allocator, calls realloc, then performs the copy if it sees the address was changed. This follows the "separation of concerns" principle where the realloc is concerned with reserving space, and another function can be concerned with preserving data, and both can be swapped out independently for different behavior.

[ikskuh]

Just have the underlying reallocFn function not perform any copy.

This will not work as a successful realloc means that the original memory will be released for future allocations and accessing it is undefined behaviour. So checking the pointers/slices afterwards will not work, as previous pointer/slice is not valid anymore.

The moving must happen before the allocator releases the old memory, but after allocating the new memory. This could probably be solved by passing an optional function to realloc that will perform a specialized "resize" operation

[marler8997]

The moving must happen before the allocator releases the old memory, but after allocating the new memory. This could probably be solved by passing an optional function to realloc that will perform a specialized "resize" operation

Oh that's right. I'm starting to remember it wasn't quite that simple :)

Looking at my code now what I did was have the underlying function implement a reallocInPlace that either works or returns an error. This way, the reallocInPlace functionality isn't coupled to a particular way of copying data from the old array, as there are many possible ways a caller could want to copy data from the old array. Maybe they only need a small section of the original array, or multiple disjoint sections. This allows the caller to use reallocate in place when possible, but use any mechanism they like to preserve data when it can't be extended in place. With this API, the current behavior and other variations on it can be implemented with simple functions that try reallocInPlace, then handle the default alloc/copy behavior if it fails.

[gereeter]

See also Rust's Alloc trait with grow_in_place and shrink_in_place methods (discussion: 1, 2) and jemalloc's xallocx function. jemalloc and TCMalloc both provide a nallocx function that can be used for the simplest in-place reallocation cases.

(Side note: jemalloc and TCMalloc also both provide a sized deallocation function sdallocx, a completely separate optimization inaccessible from Zig's current allocator interface.)

---

Personally, I'm a big fan of having in-place reallocation functions.

From an allocator implementation standpoint, the only case I've heard of for implementing realloc as anything more than reallocInPlace + memcpy on failure is to be able to use MREMAP_MAYMOVE on large allocations. Some allocators (e.g. jemalloc) try to avoid MREMAP_MAYMOVE anyway for fragmentation reasons.

From a user perspective, I'd argue that realloc (allowing moves) is just an implementation detail of dynamic arrays and useful for little else. If you want to resize something more complex like a hash table or a ring bugger, you could copy over to the new allocation and then shuffle things around, but it would be more efficient to do the shuffling as part of the copy; you'd only want to do the shuffling in-place if the reallocation itself was in-place.

[BarabasGitHub]

Cool, thanks for the links with examples. I did know about those.