Resize function does not allow use of full intended memory space

[bmarwritescode]

Description

The resize_to_fit method may expand the memory if needed. The desired addressable space is the u32 range. Given this, it could be possible that new_size is any value in the u32 range. According to the documentation for resize(), the maximum new length is isize::MAX which for 32-bit machines it would be half of the u32 range.

Impact

This will make it so that large portions of memory are unusable on certain architectures.

Recommendation

Replace this logic with resizing logic that will work for all addresses in the expected memory address range.

AuditHub Issue Link

https://app.audithub.dev/app/organizations/161/projects/521/project-viewer?version=1201&_t=1768255975985&issueId=685

[asterite]

It seems that the issue is not resize_to_fit that panics in this case. When trying to create a vec with size isize::MAX a panic also happens (on a 64-bit architecture):

fn main() {
    let vec = vec![0; isize::MAX as usize];
    println!("{}", vec.len());
}

$ cargo run
thread 'main' panicked at foo/src/main.rs:2:15:
capacity overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

In fact with large sizes, less than isize::MAX, a panic might also happen. And, if not, the memory allocation might be rejected by the OS.

So it seems there's no way to have that many values in memory at all. I'm not sure how can this be worked-around.

[vezenovm]

When trying to create a vec with size isize::MAX a panic also happens (on a 64-bit architecture)

This looks to be a limitation in the Rust allocator. In the comments of Rust's memory Layout for Layout::from_size_align we can see how the size a memory region's layout must be <= isize::MAX. This PR looks to be where this check started being enforced in Rust's allocator. In the PR we can see how this change affected the Vec tests.

We can also reproduce this error with a unit test here:

#[test]
#[should_panic(expected = "capacity overflow")]
fn resize_to_fit_panics_at_capacity_overflow() {
    let mut memory = Memory::<FieldElement>::default();
    // Attempting to resize to usize::MAX elements will always trigger capacity overflow
    // since usize::MAX > isize::MAX
    memory.resize_to_fit(usize::MAX);
}

We should have a capacity for memory similar to Rust's so that we can gracefully error rather than panic. We probably want to use i32 so that we are deterministic on 32-bit and 64-bit architectures.

To support the full u32 address space we would probably need chunked memory of some kind which is definitely a post-1.0 task.