Redoing variable declaration macro pragmas

[metagn]

This supersedes nim-lang/Nim#6696, which didn't get moved to this repo for some reason.

nim-lang/Nim#13508 implemented the following macro syntax for variables:

macro foo(varName, varType, varValue) = discard

let b {.foo.} = 4

This feature has a few shortcomings and seems unfinished. Before I get to the proposal I'll mention the few bugs that the current implementation has:

1. This feature cannot be used in let/var sections. (since they're sequential you can just break them up for this)
2. This feature cannot be used for const variables. (the PR just forgot to include const variables, it's in semVarOrLet but not in semConst)
3. Attaching types to the value argument in the template/macro gives weird errors. Try the snippet I posted with varValue: string, it says b is undefined or something. This is fine because even proc pragma macros can't use types.

Now to the design problems.

Problem 1: Additional arguments

let b {.foo: 3.} = 4 # invalid pragma: foo: "abc"

How do you even define foo for this? Where is the argument supposed to go in its signature?

Problem 2: var/let/const information

The current design gives no information if the variable being changed is var, let or const. The only arguments you get are the variable name (ident or accent), variable type and the value. You could pass information through the type I guess, the parser allows having something like var x: var T = 3, but this still requires work on the user and bloats code.

Problem 3: Incompatible with other pragmas

The current design does not allow using other pragmas. Not just other pragma macros, any pragmas.

# order doesn't matter
let b {.global, foo.} = 4 # invalid pragma: foo

This is disharmonious with proc pragma macros, they keep the other pragma regardless of the order, and if multiple macros are used the first one is evaluated.

I will add more problems if I find more, I think this many is enough.

Proposal

Instead of foo(varName, varType, varValue), we use this:

macro foo(decl) = discard

foo:
  let b = 3
# equivalent to
let b {.foo.} = 3

You can add arguments like how you would to proc pragma macros like:

macro foo(arg, decl) = discard

let b {.foo: 1.} = 3 # arg is 1, decl is `let b = 3`

We have the information of whether or not this is a let, var or const, so problem 2 is solved. It works slightly differently for sections:

let
  a = 1
  b = 2
  c {.foo.} = 3
  d = 4
  e = 5

# becomes

let
  a = 1
  b = 2
foo:
  let
    c = 3
let
  d = 4
  e = 5

This doesn't break any code since let/var/const sections are evaluated sequentially unlike type sections.

Here's an example of the usage of a user defined unpack macro in conjunction with this syntax:

let _ {.unpack: (a, (b, c)).} = (1, (2, 3))
echo a, b, c # 123

Flaws and backwards compatibility

If you use the current variable declaration pragma macros your code will be broken, but since it's such a fragile feature in regards to bugs I don't think we have many problems.

One flaw is that there's no way to supply type information here, unless we had some kind of VariableStmt like ForLoopStmt that had generic arguments that we could use here. I think that would be too complex though and no one really needs types here.

[Araq]

@timotheecour your feedback would be highly appreciated.

[timotheecour]

Thanks for the writeup! I'm aware of those flaws and the current implementation can indeed be improved; I wrote it because it solved real use cases, and in a way that can be improved without facing backward compatibility issues. It's also so recent that I don't think anyone would mind if we replace it with an improved design, with or without a deprecation cycle.

My original design intent was in fact very similar to the one you're describing (pretty much what I described in nim-lang/Nim#13830 where you replace type section by var section), but the reason I used a triplet (name,type,value) (deconstructed AST) instead of a single NimNode argument is to allow defining template pragmas, such as byaddr (nim-lang/Nim#13508) and evalonce (nim-lang/Nim#13750); which results in code that's both concise and doesn't add a (potentially cyclic) dependency on macros.nim.

Your proposal (as written) also has issues

• it doesnt' allow writing any non-trival var template pragmas, and requires importing macros.nim to deconstruct the AST for anything useful, which breaks some use cases because of cyclic dependencies. But it's fixable, see below.

• no way to supply type information

that'd be a non-starter IMO; but it's fixable, see below

here's my proposal for the "best of both worlds", 0 compromise, design

it's very similar to what you're describing and also to nim-lang/Nim#13830 where you replace type section with var section.
features:

• consistent with proc pragmas
• consistent with macro pragmas, but with the adjustment I gave here: macro pragmas in type section should apply to TypeSection, not TypeDef Nim#13830
• addresses all your concerns
• can use templates, not just macros (this last point requires a new nim feature, see below)

details:

# can be used with var/let/const
var a {.foo(arg1, arg2), foo2.}: T = bar # type and value are optional, usual nim rules
=>
foo(arg1, arg2):
  var a {.foo2. }: T = bar # gets expanded recursively if foo2 is also a var macro pragma

• var sections get split (we could re-merge if the AST output is a var section of same kind, but it doesn't matter much for var sections, unlike for types), as you described
• unlike what you wrote, type information can be supplied without any adjustment needed, see example above

can use templates, not just macros

we can define var template pragmas (thus eliminating need for the triplet destructured AST in current implementation) by introducing a new nim feature AstGet that is generally useful (I had a POC on a more limited version of this); it works similar to AstRepr in that it's interpreted before template substitution:

template byaddr(ast) =
  let tmp = addr(AstGet(ast[2]))
  template AstGet(ast[0]): untyped = tmp[]

# instead of the destructed form used in current implementation:
template byaddr(lhs, typ, rhs) = # simplified from https://github.com/nim-lang/Nim/pull/13508
  let tmp = addr(rhs)
  template AstGet(lhs): untyped = tmp[]

There are obviously many other use cases for AstGet, enabling templates in more cases where macro would be overkill

We can keep the 1 and 3 argument forms (no clash) and deprecate the 3 argument form once AstGet is implemented.

minor points

Attaching types to the value argument in the template/macro gives weird errors

not sure i understand this point; maybe post a full example? if you have macro foo(varName, varType, varValue) = discard and use let b {.foo.} = string; discard b, the b is undefined error is normal

[metagn]

unlike what you wrote, type information can be supplied without any adjustment needed, see example above

not sure i understand this point; maybe post a full example? if you have macro foo(varName, varType, varValue) = discard and use let b {.foo.} = string; discard b, the b is undefined error is normal

I should have worded more clearly. What I meant by "type information" was overloading based on the type of the RHS or the type annotation of the var/let statement.

template foo(varName, varType: untyped, varValue: int) =
  echo "int"
template foo(varName, varType: untyped, varValue: string) =
  echo "string"


let a {.foo.} = 4 # => int
let b {.foo.} = "abc" # => string

This is the limitation of the design I was proposing. There's no way to overload macros based on the type of the RHS or the type annotation itself. Your example code block (the first one) is exactly the same functionality as the initial proposal.

The "weird error message" is this:

template foo(varName, varType: untyped, varValue: string) =
  echo "works"


let a {.foo.} = 4 # => error: 'a' is undefined

I know this has a simple fix, along with all the other problems I listed (const, sections), but I'm just putting it out there.

[your proposal as written] doesnt' allow writing any non-trival var template pragmas, and requires importing macros.nim to deconstruct the AST for anything useful, which breaks some use cases because of cyclic dependencies. But it's fixable, see below.

I understand your point here but I don't see a use case where Nim can't handle it without a compiler change (with either the limited cyclic import support it does have or by reorganizing macros). Even ignoring that, proc macros have this same problem too. Your proposal for AstGet would serve in the case of this being a valid problem, and I think it could become a nice utility for templates if it evolves, but I also don't think we should just pretend it solves any problems that macros have over templates.

[timotheecour]

This is the limitation of the design I was proposing. There's no way to overload macros based on the type of the RHS or the type annotation itself

ya that's totally fine and consistent with proc macro pragmas; semantic phase hasn't happened at that stage; so that's not a limitation.

The "weird error message" is this:

ah ok, thanks for clarifying!

I understand your point here but I don't see a use case where Nim can't handle

the cyclic import problem is real. system imports/includes a large number of modules and import macros in most of these would not work (the limited cyclic import support would fall short; eg try it inside assertions.nim or dollars.nim), even when such modules would benefit from having macros. Liekwise for debugging; the venerable echo is not great for debugging for many reasons I'm not going into here, but writing your own which depends on macros.nim is currently not possible for use inside low level modules.

However, I've just found a clean solution to this:

• requires no compiler change
• allows writing non-trivial macros that don't depend on macros.nim, hence usable in low-level modules

# example user code / low-level library code that's usable in low-level modules
# all it needs is extract a few magics you need for your, typically those 2 cover a large number of use cases:
proc getAst(macroOrTemplate: untyped): NimNode {.magic: "ExpandToAst", noSideEffect.}
proc `[]`(n: NimNode, i: int): NimNode {.magic: "NChild", noSideEffect.}
macro byaddr2*(a): untyped =
  template impl(lhs, typ, rhs) =
    let tmp = addr(rhs)
    template lhs: untyped = tmp[]
  let a1 = a[0]
  getAst(impl(a1[0], a1[1], a1[2]))

# (other useful examples would be a `dbg` macro that would be usable in low level modules)

Given this works, I'm happy to drop support for the triplet (lhs,typ,rhs) and have it replaced it with the single AST node form.
I guess we've finally converged? (nim-lang/Nim#13830 still needs to be resolved though)

Now it's just a matter of implementing it...