[acorn] Missing `value` property in `Token` (TypeScript)

[e6nlaq]

import * as acorn from "acorn";

const token = acorn.tokenizer("const", { ecmaVersion: 6 });

console.log(token.getToken().value);

If you compile a file like the above with tsc, you will get the following error:

index.ts:5:30 - error TS2339: Property 'value' does not exist on type 'Token'.

5 console.log(token.getToken().value);
                               ~~~~~


Found 1 error in index.ts:5

Since the value property actually exists, the type definition is considered incorrect.

[marijnh]

There's such a thing as private properties—the types not describing every property in the code does not mean the types are incorrect.

I guess this one can be useful to external code in some situations. It is hidden because its type is a complete mess—what kind of value it holds depends on the token type, and even varies within a token type like Literal. Exporting it would mean documenting all that and, probably, giving it a type of any. What are you using the value for?

[namukang]

I can share my application's use of token.value.

My application supports users providing custom scripts that may include special variables that are automatically provided by the application. For example, the user can pass const url = $pageUrl; and due to $pageUrl starting with a $, my application detects that this is a special variable that should be defined before executing the user's script.

In order to detect all uses of special variables, I use acorn.tokenizer and check the token's value to see if it's a special variable:

function getScriptVariables(code: string): string[] {
  const tokens = [...acorn.tokenizer(code, { ecmaVersion: 2020 })];
  const variables = tokens.reduce<string[]>((variables, token) => {
    const isVariable =
      token.type.label === "name" &&
      typeof token.value === "string" &&
      isVariableName(token.value);
    return isVariable ? [...variables, token.value] : variables;
  }, []);
  return Array.from(new Set(variables));
}

I'd also like to have value defined on the Token type (even it's to any) to know that the property exists and to detect breaking changes in the future where the property may be removed.

If you think there's a better way to achieve what I'm looking to do without accessing token.value, feel free to let me know. Also, thanks for all your work on acorn and other projects!

[marijnh]

I would recommend doing a regular parse and walking the resulting tree to do this kind of thing. Tokenizing JavaScript without parsing it is problematic in general (there are valid scripts our tokenizer can't handle because it is unable to disambiguate things like regexp vs division operator in some circumstances).