Skip to content

Memory management #1337

@kddnewton

Description

@kddnewton

At the moment, we allocate memory as we need it, making a single malloc call for every node in the tree, as well as whichever fields on those nodes need memory. When we're done with the tree, we recursively visit the tree and deallocate all at once. This is a really good candidate for arena allocation. But more than that, it would be good enough just to centralize our memory management functions, which we need to do anyway. Below are a couple of tasks that are related to memory that would improve our general memory story.

  • Centralize our memory management functions. Currently we call the stdlib malloc/calloc/realloc/free functions whenever we need memory. (We purposefully avoid alloca because it's not C standard.) Those functions calls are scattered throughout the codebase. For all subsequent bullets, we need them to go through centralized functions. These functions should all accept a yp_parser_t *, but it can be marked as unused for now (YP_ATTRIBUTE_UNUSED). For this first task, they should all simply call the stdlib function. For example, a yp_malloc function would call malloc with the same arguments.
  • Provide the ability to pass memory function pointers to a yp_parser_t such that the yp_ version of the memory functions call the function pointers if they are provided. We need this for CRuby, which provides xmalloc which will attempt a GC if a call to malloc fails. This should follow the same pattern we have for the encoding_changed callback, which is that it provides a function that accepts a function pointer.
  • Request a certain number of pages at a time, estimated by the size of the input string. This would be our own arena allocation. yp_malloc would then change to call malloc/xmalloc only when the current arena has run out of allocatable memory. We can be very naive with this and make it simply a bump allocator since we so rarely deallocate nodes while parsing.

The first two of these tasks are necessary for our integration into CRuby. The last one is a nice-to-have enhancement, and is not necessary for v1.0.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions