Search Provider extension API

[roblourens]

I looked into what it will take to run search in the extension host using extension APIs.

• Ripgrep search
  • Just need to port ripgrepTextSearch.ts and ripgrepFileSearch.ts
  • Currently it downloads the correct ripgrep binary at build time. A builtin extension in the core repo can still do that. A marketplace extension would have to ship all binaries and pick one at runtime.
  • Dependencies
    • Has lots of trivial dependencies from vs/base/common. It's ok to copy them, move to native node versions, or native ES versions
    • glob.ts, but we should get rid of that dependency. It's only used for sibling clauses and those should be handled on the frontend
  • API
    • Doesn't expose all features that builtin search needs
      • Whether to use gitignore file
      • Follow symlinks
      • Max filesize
      • Encoding (can be read from files.encoding, do we expect all
        extension authors to think of that?)
      • Result limit count (handled by frontend using CancellationToken?)
      • Searching outside of the workspace
      • Smart case (front end)
      • wordSeparators (front end)
      • Cache search (for file search)
      • Sort by score (for file search)
      • Multiroot search (change it to search per single folder)
    • Need to read RelativePattern, instead of the current IFolderQuery
  • Logging
    • Use extensionhost logger and output pane
  • Frontend changes - SearchService just need to talk to search provider API
• Non-rg search is harder
  • Need glob.ts, and all of its dependencies
  • Mime
  • Iconv and the stuff we have built around it
  • IPC stuff from core for spawning worker processes
  • All of the dependencies would have to be moved to a separate node module so they can be shared with search. We probably don't want to do that.

[roblourens]

• Add extension activation event for search
• SearchModel caches results per file uri so only one is shown per file. Make it merge IFileMatches
• For running in dev, download all ripgrep builds and pick the correct one for the platform. Publish a second vscode-ripgrep package to do this?
  • We can also just run 'yarn' from the host machine every time we set up
• Invoke the search provider once per folder in a multiroot workspace
• Handle some features on the frontend
  • Sibling clauses in exclude patterns
  • Result count limit
  • Smart case
  • wordSeparators (rg search didn't respect these before...)
• I think that we should address the search API now and change it to handle all the cases that it needs to for builtin search. What I'm working with currently:

export interface TextSearchQuery {
	pattern: string;
	isRegExp?: boolean;
	isCaseSensitive?: boolean;
	isWordMatch?: boolean;
}

export interface TextSearchOptions {
	folder: Uri;
	includes: GlobPattern[];
	excludes: GlobPattern[];
	maxFileSize?: number;
	disregardIgnoreFiles?: boolean;
	ignoreSymlinks?: boolean;
	encoding?: string;
}

export interface TextSearchResult {
	uri: Uri;
	range: Range;
	preview: { leading: string, matching: string, trailing: string };
}

• Make file search work
• Figure out how cacheKey and sortByScore fit into the extension API

[jrieken]

In the TextSearchOptions how does folder and includes work together? Is the former an implicit prefix for all include-patterns?

[roblourens]

Yes that's what I'm imagining, and I think includes/excludes maybe should actually just be strings of globs relative to folder, rather than GlobPatterns, which can be relative to some other absolute path, since I'm not sure how to interpret that along with folder.

Was the original intention to have the extension read workspaceFolders, or to include the folder in includes? I think it's easier for implementors to separate "folder to search in" and "glob patterns to apply" and to have all the relevant info on the query object.

This also doesn't handle searching in extra open files. I'm not sure how to express that.

folder?: Uri;
files?: Uri[];

where just one of those is set.

Or maybe they don't need to be differentiated.

searchTargets?: Uri|Uri[];

[roblourens]

export interface SearchProvider {
	provideFileSearchResults?(options: SearchOptions, progress: Progress<Uri>, token: CancellationToken): Thenable<void>;
	provideTextSearchResults?(query: TextSearchQuery, options: TextSearchOptions, progress: Progress<TextSearchResult>, token: CancellationToken): Thenable<void>;
}

export interface TextSearchQuery {
	pattern: string;
	isRegExp?: boolean;
	isCaseSensitive?: boolean;
	isWordMatch?: boolean;
}

export interface SearchOptions {
	folder: Uri;
	includes: string[]; // paths relative to folder
	excludes: string[];
	useIgnoreFiles?: boolean;
	followSymlinks?: boolean;
    previewOptions: TBD; // total length? # of context lines? leading and trailing # of chars?
}

export interface TextSearchOptions extends SearchOptions {
	maxFileSize?: number;
	encoding?: string;
}

export interface TextSearchResult {
	uri: Uri;
	range: Range;

    // For now, preview must be a single line of text
	preview: { text: string, match: Range };
}

• How to express searching files vs folders?
• Long lines
  • This API lets us handle long lines with one result more efficiently, but if there are many matches on a line, we will send lots of duplicate line texts
  • Could allow multiple matches on a line in one TextSearchResult
• File search - extHost will handle fuzzy matching with the search string, sorting, etc. The extension just returns all files in the folder.
• Need extension activation event for search