Support adding languages using only queries

Author: pokey

.github/PULL_REQUEST_TEMPLATE/new_programming_language.md

@@ -11,6 +11,7 @@ Adds support for the `<language>` programming language
   - [ ] `"chuck arg"` with multiple arguments in list
   - [ ] `"chuck item"` with single argument in list
   - [ ] `"chuck item"` with multiple arguments in list
+- [ ] Added `@textFragment` captures. Usually you want to put these on comment and string nodes. This enables `"take round"` to work within comments and strings.
 - [ ] Added a test for `"change round"` inside a string, eg `"hello (there)"`
 - [ ] Supported` "type"` both for type annotations (eg `foo: string`) and declarations (eg `interface Foo {}`) (and added tests for this behaviour 😊)
 - [ ] Supported` "item"` both for map pairs and list entries (with tests of course)

docs/contributing/adding-a-new-language.md

@@ -16,20 +16,15 @@ First a few notes / tips:
 - We suggest opening a draft PR as soon as possible to get early feedback. Please use the new language PR template either by adding `?template=new_programming_language` to the end of the URL you used to open the PR, or just by copying and pasting from the [template](https://github.com/cursorless-dev/cursorless/blob/main/.github/PULL_REQUEST_TEMPLATE/new_programming_language.md?plain=1) to your PR body, if that's easier.
 - We suggest adding tests as early as possible, after each language feature you add. Recording tests is quick and painless using the test case recorder described below. We promise 😇
 
-Minimum changes that each language needs:
-
-- new file in `packages/cursorless-engine/src/languages/yourCoolNewLanguage.ts`. Take a look at [existing languages](../../packages/cursorless-engine/src/languages) as a base. At its core you're implementing your language's version of the `nodeMatchers` const, mapping scope types found in [`SimpleScopeTypeType`](../api/modules/common_src_types_command_partialtargetdescriptor_types/#simplescopetypetype) with matching expressions that align with the parse tree output.
-- new entry in [`getNodeMatcher.ts:languageMatchers`](../../packages/cursorless-engine/src/languages/getNodeMatcher.ts), importing your new file above
-- new entry in [`constants.ts`](../../packages/cursorless-engine/src/languages/constants.ts)
-- new text fragment extractor (default is likely fine) in [`getTextFragmentExtractor.ts:textFragmentExtractors`](../../packages/cursorless-engine/src/languages/getTextFragmentExtractor.ts)
+To add a new language, you just need to add a `.scm` file to the [`queries` directory](../../queries). The `.scm` query format is documented [here](https://tree-sitter.github.io/tree-sitter/using-parsers#query-syntax).
 
 The parse trees exposed by tree-sitter are often pretty close to what we're
 looking for, but we often need to look for specific patterns within the parse
-tree to get the scopes that the user expects. Fortunately, we have a
-domain-specific language that makes these definitions fairly compact.
+tree to get the scopes that the user expects. Fortunately, the tree-sitter query language makes these definitions fairly compact.
 
-- Check out the [docs](parse-tree-patterns.md) for the syntax tree pattern
-  matcher
+- Check out the [docs](https://tree-sitter.github.io/tree-sitter/using-parsers#query-syntax) for the query language.
+- Have a look at our custom query predicate operators in [`queryPredicateOperators.ts`](../../packages/cursorless-engine/src/languages/TreeSitterQuery/queryPredicateOperators.ts)
+- Look at the existing language definitions in the [`queries` directory](../../queries) for examples.
 - If you look in the debug console, you'll see debug output every time you move
   your cursor, which might be helpful.
 - You will likely want to look at `node-types.json` for your language, (eg [java](https://github.com/tree-sitter/tree-sitter-java/blob/master/src/node-types.json)). This file is generated from `grammar.js`, which might also be helpful to look at (eg [java](https://github.com/tree-sitter/tree-sitter-java/blob/master/grammar.js)).

docs/user/README.md

@@ -143,7 +143,7 @@ Note that if the mark is `"this"`, and you have multiple cursors, the modifiers
 
 ##### Syntactic scopes
 
-For programming languages where Cursorless has rich parse tree support, we support modifiers that expand to the nearest containing function, class, etc. See [the source code](../../packages/cursorless-engine/src/languages/constants.ts) for a list of supported languages. Below is a list of supported scope types, keeping in mind that this table can sometimes lag behind the actual list. Your cheatsheet (say `"cursorless cheatsheet"` with VSCode focused) will have the most up-to-date list.
+For programming languages where Cursorless has rich parse tree support, we support modifiers that expand to the nearest containing function, class, etc. See [the source code](../../queries) for a list of supported languages. Some languages are still supported using our legacy implementation; those will be listed in [here](../../packages/cursorless-engine/src/languages/LegacyLanguageId.ts). Below is a list of supported scope types, keeping in mind that this table can sometimes lag behind the actual list. Your cheatsheet (say `"cursorless cheatsheet"` with VSCode focused) will have the most up-to-date list.
 
 | Term           | Syntactic element                                   |
 | -------------- | --------------------------------------------------- |

packages/cursorless-engine/src/languages/LanguageDefinition.ts

@@ -8,7 +8,6 @@ import { ide } from "../singletons/ide.singleton";
 import { TreeSitter } from "../typings/TreeSitter";
 import { TreeSitterQuery } from "./TreeSitterQuery";
 import { TEXT_FRAGMENT_CAPTURE_NAME } from "./captureNames";
-import { LanguageId } from "./constants";
 
 /**
  * Represents a language definition for a single language, including the
@@ -35,7 +34,7 @@ export class LanguageDefinition {
    */
   static create(
     treeSitter: TreeSitter,
-    languageId: LanguageId,
+    languageId: string,
   ): LanguageDefinition | undefined {
     const queryPath = join(ide().assetsRoot, "queries", `${languageId}.scm`);
 

packages/cursorless-engine/src/languages/LanguageDefinitions.ts

@@ -1,6 +1,5 @@
 import { TreeSitter } from "..";
 import { LanguageDefinition } from "./LanguageDefinition";
-import { LanguageId } from "./constants";
 
 /**
  * Sentinel value to indicate that a language doesn't have
@@ -43,7 +42,7 @@ export class LanguageDefinitions {
 
     if (definition == null) {
       definition =
-        LanguageDefinition.create(this.treeSitter, languageId as LanguageId) ??
+        LanguageDefinition.create(this.treeSitter, languageId) ??
         LANGUAGE_UNDEFINED;
 
       this.languageDefinitions.set(languageId, definition);

packages/cursorless-engine/src/languages/LegacyLanguageId.ts

@@ -0,0 +1,28 @@
+/**
+ * The language IDs that we have full tree-sitter support for using our legacy
+ * modifiers.
+ */
+export type LegacyLanguageId =
+  | "c"
+  | "clojure"
+  | "cpp"
+  | "css"
+  | "csharp"
+  | "go"
+  | "html"
+  | "java"
+  | "javascript"
+  | "javascriptreact"
+  | "json"
+  | "jsonc"
+  | "latex"
+  | "markdown"
+  | "php"
+  | "python"
+  | "ruby"
+  | "scala"
+  | "scss"
+  | "rust"
+  | "typescript"
+  | "typescriptreact"
+  | "xml";

packages/cursorless-engine/src/languages/TreeSitterQuery/queryPredicateOperators.ts

@@ -6,8 +6,9 @@ import { HasSchema } from "./PredicateOperatorSchemaTypes";
 
 /**
  * A predicate operator that returns true if the node is not of the given type.
- * For example, `(not-type? string)` will match any node that is not a string.
- * It is acceptable to pass in multiple types, e.g. `(not-type? string comment)`.
+ * For example, `(not-type? @foo string)` will reject the match if the `@foo`
+ * capture is a `string` node. It is acceptable to pass in multiple types, e.g.
+ * `(not-type? @foo string comment)`.
  */
 class NotType extends QueryPredicateOperator<NotType> {
   name = "not-type?" as const;
@@ -19,9 +20,9 @@ class NotType extends QueryPredicateOperator<NotType> {
 
 /**
  * A predicate operator that returns true if the node's parent is not of the
- * given type. For example, `(not-parent-type? string)` will match any node that
- * is not a child of a string. It is acceptable to pass in multiple types, e.g.
- * `(not-parent-type? string comment)`.
+ * given type. For example, `(not-parent-type? @foo string)` will reject the
+ * match if the `@foo` capture is a child of a `string` node. It is acceptable
+ * to pass in multiple types, e.g. `(not-parent-type? @foo string comment)`.
  */
 class NotParentType extends QueryPredicateOperator<NotParentType> {
   name = "not-parent-type?" as const;
@@ -33,8 +34,8 @@ class NotParentType extends QueryPredicateOperator<NotParentType> {
 
 /**
  * A predicate operator that returns true if the node is the nth child of its
- * parent.  For example, `(is-nth-child? 0)` will match the first child of any
- * node.
+ * parent.  For example, `(is-nth-child? @foo 0)` will reject the match if the
+ * `@foo` capture is not the first child of its parent.
  */
 class IsNthChild extends QueryPredicateOperator<IsNthChild> {
   name = "is-nth-child?" as const;

packages/cursorless-engine/src/languages/constants.ts

@@ -9,7 +9,7 @@ import {
 import { notSupported } from "../util/nodeMatchers";
 import { selectionWithEditorFromRange } from "../util/selectionUtils";
 import clojure from "./clojure";
-import { SupportedLanguageId } from "./constants";
+import { LegacyLanguageId } from "./LegacyLanguageId";
 import cpp from "./cpp";
 import csharp from "./csharp";
 import go from "./go";
@@ -31,7 +31,7 @@ export function getNodeMatcher(
   scopeTypeType: SimpleScopeTypeType,
   includeSiblings: boolean,
 ): NodeMatcher {
-  const matchers = languageMatchers[languageId as SupportedLanguageId];
+  const matchers = languageMatchers[languageId as LegacyLanguageId];
 
   if (matchers == null) {
     throw new UnsupportedLanguageError(languageId);
@@ -51,7 +51,7 @@ export function getNodeMatcher(
 }
 
 const languageMatchers: Record<
-  SupportedLanguageId,
+  LegacyLanguageId,
   Record<SimpleScopeTypeType, NodeMatcher>
 > = {
   c: cpp,

packages/cursorless-engine/src/languages/getNodeMatcher.ts

@@ -3,7 +3,7 @@ import type { SyntaxNode } from "web-tree-sitter";
 import { SelectionWithEditor } from "../typings/Types";
 import { notSupported } from "../util/nodeMatchers";
 import { getNodeInternalRange, getNodeRange } from "../util/nodeSelectors";
-import { SupportedLanguageId } from "./constants";
+import { LegacyLanguageId } from "./LegacyLanguageId";
 import { getNodeMatcher } from "./getNodeMatcher";
 import { stringTextFragmentExtractor as htmlStringTextFragmentExtractor } from "./html";
 import { stringTextFragmentExtractor as jsonStringTextFragmentExtractor } from "./json";
@@ -18,7 +18,7 @@ export type TextFragmentExtractor = (
 ) => Range | null;
 
 function constructDefaultTextFragmentExtractor(
-  languageId: SupportedLanguageId,
+  languageId: LegacyLanguageId,
   stringTextFragmentExtractor?: TextFragmentExtractor,
 ): TextFragmentExtractor {
   const commentNodeMatcher = getNodeMatcher(languageId, "comment", false);
@@ -51,7 +51,7 @@ function constructDefaultTextFragmentExtractor(
 }
 
 function constructDefaultStringTextFragmentExtractor(
-  languageId: SupportedLanguageId,
+  languageId: LegacyLanguageId,
 ): TextFragmentExtractor {
   const stringNodeMatcher = getNodeMatcher(languageId, "string", false);
 
@@ -79,7 +79,7 @@ function constructDefaultStringTextFragmentExtractor(
  * @returns The range of the string text or null if the node is not a string
  */
 function constructHackedStringTextFragmentExtractor(
-  languageId: SupportedLanguageId,
+  languageId: LegacyLanguageId,
 ) {
   const stringNodeMatcher = getNodeMatcher(languageId, "string", false);
 
@@ -105,7 +105,7 @@ function constructHackedStringTextFragmentExtractor(
 export default function getTextFragmentExtractor(
   languageId: string,
 ): TextFragmentExtractor {
-  const extractor = textFragmentExtractors[languageId as SupportedLanguageId];
+  const extractor = textFragmentExtractors[languageId as LegacyLanguageId];
 
   if (extractor == null) {
     throw new UnsupportedLanguageError(languageId);
@@ -121,7 +121,7 @@ type FullDocumentTextFragmentExtractor = null;
 const fullDocumentTextFragmentExtractor = null;
 
 const textFragmentExtractors: Record<
-  SupportedLanguageId,
+  LegacyLanguageId,
   TextFragmentExtractor | FullDocumentTextFragmentExtractor
 > = {
   c: constructDefaultTextFragmentExtractor("c"),

packages/cursorless-engine/src/languages/getTextFragmentExtractor.ts

@@ -1,7 +1,6 @@
 import * as assert from "assert";
 import { flatten, range } from "lodash";
 import { tokenize } from ".";
-import { LanguageId } from "../languages/constants";
 import { unitTestSetup } from "../test/unitTestSetup";
 
 type TestCase = [string, string[]];
@@ -110,9 +109,7 @@ const shellScriptDialectTokenizerTests: LanguageTokenizerTests = {
   exclusionPredicate: (input: string) => !!input.match("-"),
 };
 
-const languageTokenizerTests: Partial<
-  Record<LanguageId, LanguageTokenizerTests>
-> = {
+const languageTokenizerTests: Record<string, LanguageTokenizerTests> = {
   css: cssDialectTokenizerTests,
   scss: cssDialectTokenizerTests,
   shellscript: shellScriptDialectTokenizerTests,