Not yet using `tree-sitter.json`

[sogaiu]

I think at some point we should start using tree-sitter.json. The file is described here.

Also, there is some more info about eventual removal of the tree-sitter section from package.json here.

I'd like to get rid of package.json, but I think Cursorless may still depend on it, so that may not happen so soon.

---

(On a side note, the build-wasm command may be going away in 0.24; with recent releases you can already use build --wasm the replacement. My attempts at getting the older subcommand to stay around longer did not meet with success. Backward compatibility of the cli API does not rank that highly priority-wise AFAIU.)

[sogaiu]

At some point it looks like the tree-sitter cli's generate subcommand may result in the mutating of package.json and the creating of a tree-sitter.json file. AFAIU, this may happen if there isn't already a tree-sitter.json file.

Gave it a try here [1] and observed that some of the values that got into tree-sitter.json were wrong for our case (e.g. license, repository URL, and binding-generation):

{
  "grammars": [
    {
      "name": "clojure",
      "camelcase": "Clojure",
      "scope": "source.clojure",
      "path": ".",
      "file-types": [
        "bb",
        "clj",
        "cljc",
        "cljs"
      ]
    }
  ],
  "metadata": {
    "version": "0.0.13",
    "license": "MIT",
    "description": "Clojure grammar for tree-sitter",
    "links": {
      "repository": "https://github.com/tree-sitter/tree-sitter-clojure"
    }
  },
  "bindings": {
    "c": true,
    "go": true,
    "node": true,
    "python": true,
    "rust": true,
    "swift": true
  }
}

It may be that pre-emptively creating a tree-sitter.json will leave our package.json file alone.

Below is a draft of a possible tree-sitter.json:

{
  "grammars": [
    {
      "name": "clojure",
      "camelcase": "Clojure",
      "scope": "source.clojure",
      "path": ".",
      "file-types": [
        "bb",
        "clj",
        "cljc",
        "cljs"
      ]
    }
  ],
  "metadata": {
    "version": "0.0.13",
    "description": "Clojure grammar for tree-sitter"
  }
}

Unfortunately, it looks like version info is now duplicated across two files. We may be stuck with this unless / until Cursorless stops needing package.json.

---

On a side note, having both package.json and tree-sitter.json might be nicer because some users may be using older versions of the tree-sitter cli while others newer. It may be that having both files in the repository will serve a wider range of users (i.e. a broader range of versions of the cli can be used easily).

---

[1] Tested with:

tree-sitter 0.24.2 (0b167b0782a74e3702fb77cbef1486c6c01c24c8)

though it's possible an earlier version exhibits similar behavior.

[sogaiu]

The situation of grammars missing tree-sitter.json files was mentioned in a comment as part of a recent PR:

Ok, there's 3 out of 356 grammars with no tree-sitter.json file used in nvim-treesitter that must be generated from the grammar. The config file is sort of mandatory for this feature, as we use the version defined in this file. That means <1% of grammars are impacted, and I've gone ahead and put up PRs to migrate them 2 days ago now - if they don't bother to update, then that's on them, and they cannot use ABI 15 then (but nothing is broken with ABI 14).

[sogaiu]

I have a script to investigate tree-sitter.json use across locally collected grammar repositories. The default branches of most of the grammars listed here were fetched locally [1].

When I ran it today, the output ended with [2]:

Number with tree-sitter.json in root: 153
Number with tree-sitter.json in subdir: 2
Number without tree-sitter.json: 300
Number of repositories: 454

Via another script, here is some ABI use over the same collection [3]:

13: 75
14: 314
15: 39

To summarize:

• About twice as many grammar repositories (300) do NOT have tree-sitter.json compared with those that do (153).
• Although only 39 grammars appear to be using ABI 15, many more (> 150) have tree-sitter.json. At this time, the presence of tree-sitter.json isn't a good indicator of whether fetching a grammar's repository is going to get something that has been tested with ABI 15.

---

[1] The collection has been gradually added to so it includes older things that are no longer listed.

[2] I think the way to interpret 153 is that it is the number of repositories that have tree-sitter.json in the root directory. While 2 represents the number of repositories that have tree-sitter.json in a subdirectory. These two groups are not necessarily mutually exclusive, which may explain why 300 + 153 + 2 != 454 (in this case).

[3] I removed some output here for grammars that had ABI numbers 12 (in src/parser.c) and below -- also, not every grammar repository has src/parser.c committed so the totals won't add up to match the numbers above.

[sogaiu]

This is a reminder to update the what and why document if tree-sitter.json is added.

[george-gayton]

Selfishly I'd like to request the inclusion of tree-sitter.json.

My use case is that I'd like to add Clojure support to this cucumber service. The service requires the tree-sitter.json format to build - although I haven't spent much time into looking into alternatives.

That cucumber service currently supports go, java, javascript, php, python, ruby, rust and typescript - which all use tree-sitter.json with duplicated data in package.json.

[sogaiu]

I think at this point you are better off forking and adding it yourself.

It's unlikely that this change will be made soon.

[NoahTheDuke]

what is hurt or lost by adding it?

[george-gayton]

Is the problem with your approach here the duplicate information, in which case would a pre-commit hook handle this case?

[george-gayton]

For a use case that seems unable to proceed without official support: convo.

[sogaiu]

I don't follow the logic of that statement.

The link in the linked comment is not up-to-date, nor is this quoted content:

The bindings are not tested in any way and only really exist as a side-effect of running the tree-sitter cli's generate subcommand.

That commit is from around 2023-03-17. The bindings were removed in this commit from around 2024-01-28.

At the time of this writing, this is the latest, where it starts off:

The repository does not host any bindings (e.g. for Rust, Node, or other programming language).

They should be straight-forward to generate as long as one has a suitable tree-sitter cli and the grammar.js file mentioned above.

People are free to fork this repository and generate their own bindings.

I am not a node.js / npm community participant, nor do I plan on becoming one in the near future. I don't believe it makes sense to expect a person in that kind of situation to create and maintain bindings in a responsible manner.

Please see this comment in the issue of another grammar with a similar conversation.

[george-gayton]

Yeah this is understandable. I can see I've derailed this thread, apologies.

Thanks for the quick reply.

[wetneb]

I made a fork with tree-sitter.json and bindings, which are uploaded to crates.io and PyPI for now.

It doesn't store generated files such as parser.c inside the repository, so if your workflow relies on their existence, it's worth downloading the archive that contains generated files, published in each release: https://codeberg.org/grammar-orchard/tree-sitter-clojure-orchard/releases

If people are interested in using it in other ways, I'm open to adapting the publishing workflows of grammars stored in this Codeberg org to enable those uses. Co-maintainers are also welcome :)