Simplify lpython.py

[certik]

Right now the file is already quite small (less than 800 lines): https://github.com/lcompilers/lpython/blob/8df052e174adbd789fc7b03f5526e706195b3543/src/runtime/lpython/lpython.py, but every time it has some custom emulation layer, it almost always breaks.

In the past we have already removed two custom layers:

• We emulated UnsignedInteger and it was subtly broken in many cases. We removed it.
• goto: it was emulated and it broke in a new CPython version; we remove dit.

Now there is another breakage regarding pointers and packed structs using numpy arrays (#2267).

We should try to remove as many custom emulation classes as possible, and only use native CPython classes (like we now use "int" to represent unsigned integer). We might need to remove some support for C pointers or pointers in general, because CPython does not have it, and our emulation is too fragile.

Ideally the file should only contain some types, and various conversion functions that can be implemented very robustly and that do not return any "emulated/wrapped" types.

This will make the file much more robust and resilient against future CPython versions.

[rebcabin]

I think the only way to robustly support pointers is a "heavy" API interface, and it's probably not worth the effort. I think, design-wise, we should make beautiful array types and then discourage people from writing C-flavored Python.

[certik]

I think that lpython.py should only contain:

• new type annotations, such as these:

  lpython/src/runtime/lpython/lpython.py

  Lines 80 to 98 in 8df052e

  	i1 = Type("i1")
  	i8 = Type("i8")
  	i16 = Type("i16")
  	i32 = Type("i32")
  	i64 = Type("i64")
  	u8 = Type("u8")
  	u16 = Type("u16")
  	u32 = Type("u32")
  	u64 = Type("u64")
  	f32 = Type("f32")
  	f64 = Type("f64")
  	c32 = Type("c32")
  	c64 = Type("c64")
  	CPtr = Type("c_ptr")
  	Const = ConstType("Const")
  	Callable = Type("Callable")
  	Allocatable = Type("Allocatable")
  	Pointer = PointerType("Pointer")
  	S = Type("S")

• functions that implement some operation that is robustly and cleanly implemented using CPython and its own intrinsic types, an example is:

  lpython/src/runtime/lpython/lpython.py

  Lines 760 to 761 in 8df052e

  	def bitnot(x, bitsize):
  	return (~x) % (2 ** bitsize)

• decorators that are implemented in a clean and robust way, some examples are: @overload, @pythoncall, @ccall, @lpython.

If done right, the decorators would be used in low level code where you need to interface with CPython or C or need to use overloads (not very common). The custom functions like bitnot are very rare, most code would not use it. That leaves the type annotations --- we can later support some more default CPython annotations as well, perhaps the user can create them. Then people would not need import lpython at all. We would just compile regular CPython code that runs in CPython without any non-standard lpython.py library. Perhaps we can support numpy types like int32. For interfacing C, we could just support ctypes directly (not as nice syntax, but we could do it). And pythoncall is just a small decorator, and perhaps there is an even simpler way to do it, for example we can annotate a function (with an empty decorator) and interpret the insides as CPython. This would be optional, the lpython module provides a convenience way, but if you don't want to introduce this dependency into a larger project, we should support a "non-lpython module mode" as well.

What we do not want to do:

• depend on internals of ctypes and hoping they can "emulate" pointers, an example is

  lpython/src/runtime/lpython/lpython.py

  Lines 594 to 616 in 8df052e

  	def c_p_pointer(cptr, targettype, targetshape=None):
  	targettype_ptr = convert_type_to_ctype(targettype)
  	if isinstance(targettype, Array):
  	if targetshape is None:
  	raise ValueError("target shape must be "
  	"provided if target type is an array.")
  	# TODO: Add support for multi-dimensional shape of target variable
  	if py_is_dataclass(targettype._type):
  	return ctypes.cast(cptr.value, ctypes.py_object).value
  	newa = ctypes.cast(cptr, targettype_ptr)
  	return newa
  	else:
  	if py_is_dataclass(targettype):
  	if cptr.value is None:
  	return None
  	return ctypes.cast(cptr, ctypes.py_object).value
  	targettype_ptr = ctypes.POINTER(targettype_ptr)
  	newa = ctypes.cast(cptr, targettype_ptr)
  	if is_ctypes_Structure(targettype):
  	# return after wrapping newa inside PointerToStruct
  	return PointerToStruct(newa)
  	return newa

• Return wrapped entities, such as

  lpython/src/runtime/lpython/lpython.py

  Line 615 in 8df052e

  return PointerToStruct(newa)

  .

Instead, we need to design the subset of CPython in such a way that we can just use native CPython constructs. In particular, if it is an array, we should just provide some function that converts a CPtr to a numpy array, but we should NOT emulate pointer access or pointer arithmetic directly.

Probably we do not want to even support pointers directly like in here:

lpython/src/runtime/lpython/lpython.py

Line 509 in 8df052e

def pointer(x, type_=None):

, because it is returning a ctype "emulation", and that's fragile. Internally in ASR we can keep pointers, I think that's fine.

All we need to do is to provide some way to interact with C, but our current level of API seems "heavy". For example, take this:

lpython/integration_tests/structs_02.py

Lines 9 to 19 in 8df052e

	def f(a: CPtr) -> None:
	x: i32
	y: f32
	a1: A = A(3, f32(3.25))
	a2: Pointer[A]
	a2 = pointer(a1)
	print(a2, pointer(a1))
	x = a2.x
	y = a2.y
	assert x == 3
	assert f64(y) == 3.25

, currently we emulate this via ctypes. Rather, the implementation of pointer should be just:

def pointer(x):
    return x

Which in CPython b = pointer(a) increments the refcount of a and both a and b point to the same structure, and so it behaves identically to a pointer. LPython will ensure that the semantic subset that we do is 100% identical. Consequently, you can only take "pointers" of structs, lists, arrays, but not ints, floats.

However perhaps we don't even need a notion of pointers at all, we just need some light API to interact with C, and when C returns a pointer to int, we just need some way to call it, perhaps via a numpy array of ints, of size 1, or something like that, and LPython ensures that there is no overhead.