Interim Transpiler

[rmorshea]

I've been wishing I could use tag strings lately and so to satisfy that craving I thought it would be cool to create an import-time transpiler that would rewrite:

my_tag @ f"my {custom} string"
#      ^ or any other operator not typically used with strings

To be:

my_tag("my ", (lambda: custom, "custom", None, None), " string")

The syntax seems clever in a few different ways:

• it's valid Python
• syntax highlighters will highlight expressions in the string
• the transpiler will be straightforward to implement since the AST's JoinedStr already splits out the expressions

Something like this seems like a rather natural extension of @pauleveritt's work in viewdom.

Implementation Details

Some potential issues I've thought of and ways to solve them.

Static Typing

To deal with static type analyzers complaining about the unsupported @ operator, tag functions can be decorated with:

def tag(func: TagFunc[T]) -> Tag[T]:
    # convince MyPy that our new syntax is valid
    return cast(Tag, func)

class TagFunc(Protocol[T]):
    def __call__(self, *args: Thunk) -> T: ...

class Tag(Generic[T]):
    def __call__(self, *args: Thunk) -> T: ...
    def __matmul__(self, other: str) -> T: ...

An alternate syntax of my_tag(f"...") would not require this typing hack since tag functions must already accept *args: str | Thunk. The main problem here is that there are probably many times where some_function(f"...") would show up in a normal code-base. Thus it would be hard to determine whether any given instance of that syntax ought to be transpiled. Solving this would require users to mark which identifiers should be treated as tags - perhaps with a line like set_tags("my_tag"). This seems more inconvenient than having the tag author add the aforementioned decorator though.

Performance

To avoid transpiling every module, users would need to indicate which ones should be rewritten at import-time. This could be done by having the user import this library somewhere at the top of their module. At import time, the transpiler, before parsing the file, would then scan it for a line like import <this_library> or from <this_library> import ....

[gvanrossum]

Cool, and good that you've already thought through some alternatives in the design space. @ seems brilliant because, indeed, it's not something one would do with an f-string on the RHS.

How were you planning to trigger the transpiler? As an import hook, or as a codec?

[rmorshea]

My plan was to use an import hook, but I wasn't aware codecs could be used for this purpose, so that could be a good alternative as well.

[gvanrossum]

See https://peps.python.org/pep-0263/; you can register a codec with an arbitrary name. This would take the place of your "marker import". IIRC there are some issues with getting the codec registered when your package is installed though.

[rmorshea]

I think a codec could be better for a lot of reasons:

• I suspect that the codec only has to be re-run if a file has changed
• As you mention, we already need some way to mark files for transpilation, which the # coding=... comment would achieve
• Import hooks are pretty complicated to write and they could conflict with other hooks like the one PyTest uses.
• Python 3.9 comes with a handy ast.unparse function that would make it easy to write back the modified AST

IIRC there are some issues with getting the codec registered when your package is installed though.

Would the approach be to use a .pth file to call codecs.register(my_codec) before the user's code is imported?

[gvanrossum]

Yeah, if the .py file and the .pyc file match, the source is never read, so the codec isn't run. Pure win!

I suspect that there are some problems with asy.unparse though, since the AST doesn't preserve comments or whitespace. Notably the line numbers after the unparsing will differ, which will make tracebacks hugely confusing.

We used the codec trick at Dropbox for pyxl3, and IIRC the rewrite was done very differently, to ensure that the line numbers matched. I think the "parsing" was probably done with a regular expression. That should work here too.

[pauleveritt]

What would be the story for tracebacks and getting back to lines in source?

[gvanrossum]

The traceback code looks up the line number in the untranslated source (linecache.py just opens the file in text mode, no encoding parameter). So pyxl ensures that the translated line numbers match the original line numbers (but the column offsets don't). I think the translation Ryan proposes should be able to preserve line numbers as well.

[rmorshea]

Crazy idea, but what if...

my_tag @ f"my {super} {custom} string"

Became

my_tag((       super,  custom), raw=("super", "custom"), conv=(None, None), formatspec=(None, None), strings=("my ", " ", " string"))

Where the @tag decorator would zip and merge things as necessary to resolve differences between this new interface and the one defined for tag strings. Doing this might seem convoluted, but the neat thing about it is that you'd be able to move the expressions from the string around to match their column offsets as needed. This would even work for multi-line strings:

my_tag @ f"""
my
extra {super}
{custom}
string
"""

Would become:

my_tag((

       super,
 custom), raw=("super", "custom"), conv=(None, None), formatspec=(None, None), strings=("my ", " ", " string")

[rmorshea]

Shoot! The evaluation of the expressions would no longer be lazy.

[rmorshea]

The last way I can think of to preserve column offsets in tracebacks is by passing information about the location of expressions in the original source and using that to modify tracebacks which arise within the tag function itself.

my_tag("my ", (lambda: custom, "custom", None, None, 1, 16), " string")

Where the @tag decorator would modify my_tag by do something like:

def tag(func):

    def wrapper(*args, src_info=None):
        new_args = []
        for a in args:
            match a:
                case str():
                    new_args.append(a)
                case getvalue, src, conv, spec, *src_info:
                    getvalue = modify_tracebacks(getvalue, *src_info)
                    new_args.append((getvalue, src, conv, spec))
        return func(*args)
    
    return wrapper

def modify_tracebacks(getvalue, lineno=None, col_offset=None):
    if not (lineno and col_offset):
        return getvalue
    
    def wrapper():
        try:
            return getvalue()
        except Exception as error:
            # modify the traceback with the appropriate lineno and col_offset somehow
            error.with_traceback(...)

    return new_getvalue

If this worked, it'd mean that the transpiler wouldn't even need to worry about preserving line numbers in the areas of code it modified.

[gvanrossum]

IMO it's not worth worrying about column offsets for the initial prototype.

[pauleveritt]

@rmorshea If there's a way for me to join in with what you're doing and re-parent my stuff on your interim transpiler, let me know.

[rmorshea]

Will do. I might have time to create a repo for this tonight, but otherwise I won't be able to do much until next week.