Design of structs in LPython

[certik]

We want to map our structs into ordinary CPython. For now we use dataclasses, but #2275 shows that Python 3.11 changed a few things. So let's use this opportunity to design it in such a way that we just use the dataclass decorator directly from CPython, we do not wrap it in anyway.

Here are all the ways to use dataclasses that work in all Python versions including 3.11.

No Default Values

The first way is without initializing the members:

from dataclasses import dataclass
from numpy import array, ndarray

@dataclass
class X:
    a: int
    b: bool
    c: list[int]
    d: ndarray

def main0():
    x: X = X(123, True, [1, 2, 3], array([4, 5, 6]))
    print(x)
    assert x.a == 123
    assert x.b == True
    assert x.c[0] == 1
    assert x.d[1] == 5
    x.c[0] = 3
    x.d[0] = 3
    print(x)
    assert x.c[0] == 3
    assert x.d[0] == 3

main0()

The fields are mutable, and there is no problem at all, it works with lists and numpy arrays also.

Default Values

First of all, I don't think we need to provide default values. But if we do, here is how to do it:

from dataclasses import dataclass, field
from numpy import array, ndarray

@dataclass
class X:
    a: int = 123
    b: bool = True
    c: list[int] = field(default_factory=lambda: [1, 2, 3])
    d: ndarray = field(default_factory=lambda: array([4, 5, 6]))

def main0():
    x: X = X()
    print(x)
    assert x.a == 123
    assert x.b == True
    assert x.c[0] == 1
    assert x.d[1] == 5
    x.c[0] = 3
    x.d[0] = 3
    print(x)
    assert x.c[0] == 3
    assert x.d[0] == 3

main0()

This works for all Python versions including 3.11.

Arrays of Structs

This works in all Python versions.

from dataclasses import dataclass, field
from numpy import array, ndarray

@dataclass
class X:
    a: int = 123
    b: bool = True
    c: list[int] = field(default_factory=lambda: [1, 2, 3])
    d: ndarray = field(default_factory=lambda: array([4, 5, 6]))

def main0():
    x: ndarray = array([X(), X(), X()])
    print(x)
    x[0].a = 10
    x[0].c[1] = 10
    x[0].d[1] = 10

    x[2].a = 11
    x[2].c[1] = 11
    x[2].d[1] = 11
    print(x)

main0()

Using i32[3] Syntax

This seems to work for all Python versions.

from dataclasses import dataclass
from numpy import array, ndarray
from lpython import i32

@dataclass
class X:
    a: int
    b: bool
    c: list[int]
    d: i32[3]

def main0():
    x: X = X(123, True, [1, 2, 3], array([4, 5, 6]))
    print(x)
    assert x.a == 123
    assert x.b == True
    assert x.c[0] == 1
    assert x.d[1] == 5
    x.c[0] = 3
    x.d[0] = 3
    print(x)
    assert x.c[0] == 3
    assert x.d[0] == 3

main0()

[rebcabin]

Wow! Excellent!

[certik]

The way this should be implemented in lpython.py is to just do from dataclasses import dataclass, field inside lpython.py and that's it. No custom wrappers. Later on, we will support from dataclasses import dataclass, field directly as an option.

We can do this step by step.

[ubaidsk]

x: ndarray = array([X(), X(), X()])

I have the following doubt with respect to the above approach of arrays of structs. Please clarify.

• The annotation ndarray does not inform about the type of the array. Please, could you share how we could get the type of the array (for LPython) when it is declared as a function parameter?

[certik]

This works for compile time size:

    x: X[3] = array([X(), X(), X()])

And this works for allocatable (runtime size):

    x: Allocatable[X[:]]
    n = 15 # Read from a file
    x = vectorize(X)([None]*n)
    print(x)

The array of structs is a low priority. The above seems to create a numpy array of type "object", so it won't interoperate with C anyway. For the use case where we have an array of structs in C and are trying to "map" it to a CPython structure, we'll need to figure out some other solution, here is an example how to do it using ctypes: https://stackoverflow.com/a/17102186/479532, but if there is no way to map this into a numpy array of dataclasses, then we should provide a different mechanism for that. Here is what chatgpt suggests: https://chat.openai.com/share/4d07bfe3-72fa-40ff-b476-d26d33cdacee, apparently there is a way to use numpy.

[ubaidsk]

My concern currently is that the type annotation X[:] is not supported in CPython by default as a function parameter declaration. The emulation function dataclass in lpython.py helps us support this annotation.

Here #2275 (comment) you shared that we should not subscript X directly, but rather have a NumPy array of X. Please, could you elaborate your approach? It would be very helpful if you could share an example code/snippet with your approch (that works with CPython and LPython).

[certik]

We should not do X[:] ever (by overriding the __class_getitem__ in @dataclass). Instead, we should use NumPy. I shared above a link (https://chat.openai.com/share/4d07bfe3-72fa-40ff-b476-d26d33cdacee) how to do this, here is the relevant part:

# Creating a NumPy array that directly maps to the memory of the ctypes array
np_array = np.ctypeslib.as_array(test.STRUCT_ARRAY, shape=(num_of_structs,)).view(np.recarray)

# Accessing and modifying the NumPy array elements directly affects the underlying ctypes data
for i in range(num_of_structs):
    np_array[i].field_1 = 10
    np_array[i].field_2 = 20

# Printing the changes made in the ctypes array via the NumPy array
for i in range(num_of_structs):
    print(f"Element {i}: field_1 = {test.STRUCT_ARRAY[i].field_1}, field_2 = {test.STRUCT_ARRAY[i].field_2}")

I tested it and it works.

[certik]

So that we don't have to provide a custom dataclass wrapper, one option to declare X[:], is something like: Array[X, :]. So i32[:,:] becomes just syntactic sugar for Array[i32, :, :].

So I would do:

@dataclass
class X:
    a: int = 123
    b: bool = True
    c: list[int] = field(default_factory=lambda: [1, 2, 3])
    d: i32[3] = field(default_factory=lambda: array([4, 5, 6]))

and

@dataclass
class X:
    a: int
    b: bool
    c: list[int]
    d: i32[3]