-
Notifications
You must be signed in to change notification settings - Fork 148
Node: add FT.CREATE command
#2501
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 24 commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
1a32394
Initial commit for FT.CREATE - no tests
acarbonetto 938e262
linter
acarbonetto 05a4cc1
Trying to fix tests
acarbonetto 71c973e
Add TLS to tests
acarbonetto d9a2909
Add FT modules test
acarbonetto a4a112d
Revert useTLS changes
acarbonetto 74442a7
Revert useTLS changes
acarbonetto e9fc446
Export options
acarbonetto fde3b6c
Add first ft.create test
acarbonetto 7af79e1
Fix imports
acarbonetto 9b550e2
Print attributes to args
acarbonetto 41db72e
fix attributes to args
acarbonetto 745da41
Fix tests
acarbonetto 2b57f3b
Fix tests
acarbonetto d861df6
Fix tests
acarbonetto 2b1ba26
Fix tests
acarbonetto c401d4f
Update more tests
acarbonetto e29be99
Update error tests
acarbonetto aeed5d2
Linter & fix order of options
acarbonetto 351d32a
Linter & fix order of options
acarbonetto 67437f8
Add example; linter
acarbonetto 7b5546e
Cleanup from self-review
acarbonetto 8cc465b
Cleanup from self-review
acarbonetto 6fdadc4
Fix merge conflicts
acarbonetto 9297eb6
Apply review comments
acarbonetto 5362967
Update for tests
acarbonetto bea6cf9
linter
acarbonetto File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| /** | ||
| * Copyright Valkey GLIDE Project Contributors - SPDX Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { Decoder, DecoderOption, GlideString } from "../BaseClient"; | ||
| import { GlideClient } from "../GlideClient"; | ||
| import { GlideClusterClient } from "../GlideClusterClient"; | ||
| import { Field, FtCreateOptions } from "./GlideFtOptions"; | ||
|
|
||
| /** Module for Vector Search commands */ | ||
| export class GlideFt { | ||
| /** | ||
| * Creates an index and initiates a backfill of that index. | ||
| * | ||
| * @param client The client to execute the command. | ||
| * @param indexName The index name for the index to be created. | ||
| * @param schema The fields of the index schema, specifying the fields and their types. | ||
| * @param options Optional arguments for the `FT.CREATE` command. | ||
acarbonetto marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| * | ||
| * @returns If the index is successfully created, returns "OK". | ||
| * | ||
| * @example | ||
| * ```typescript | ||
| * // Example usage of FT.CREATE to create a 6-dimensional JSON index using the HNSW algorithm | ||
| * await GlideFt.create(client, "json_idx1", [{ | ||
| * type: "VECTOR", | ||
| * name: "$.vec", | ||
| * alias: "VEC", | ||
| * attributes: { | ||
| * algorithm: "HNSW", | ||
| * type: "FLOAT32", | ||
| * dimension: 6, | ||
| * distanceMetric: "L2", | ||
| * numberOfEdges: 32, | ||
| * }, | ||
| * }], { | ||
| * dataType: "JSON", | ||
| * prefixes: ["json:"] | ||
| * }); | ||
| * ``` | ||
| */ | ||
| static async create( | ||
| client: GlideClient | GlideClusterClient, | ||
| indexName: GlideString, | ||
| schema: Field[], | ||
| options?: FtCreateOptions | DecoderOption, | ||
acarbonetto marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ): Promise<"OK" | null> { | ||
acarbonetto marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| const args: GlideString[] = ["FT.CREATE", indexName]; | ||
|
|
||
| if (options) { | ||
| if ("dataType" in options) { | ||
| args.push("ON", options.dataType); | ||
| } | ||
|
|
||
| if ("prefixes" in options && options.prefixes) { | ||
| args.push( | ||
| "PREFIX", | ||
| options.prefixes.length.toString(), | ||
| ...options.prefixes, | ||
| ); | ||
| } | ||
| } | ||
|
|
||
| args.push("SCHEMA"); | ||
|
|
||
| schema.forEach((f) => { | ||
| args.push(f.name); | ||
|
|
||
| if (f.alias) { | ||
| args.push("AS", f.alias); | ||
| } | ||
|
|
||
| args.push(f.type); | ||
|
|
||
| // TagField attributes | ||
| if (f.type === "TAG") { | ||
acarbonetto marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| if (f.separator) { | ||
| args.push("SEPARATOR", f.separator); | ||
| } | ||
|
|
||
| if (f.caseSensitive) { | ||
| args.push("CASESENSITIVE"); | ||
| } | ||
| } | ||
|
|
||
| if (f.type === "VECTOR" && f.attributes) { | ||
| args.push(f.attributes.algorithm); | ||
|
|
||
| const attributes: GlideString[] = []; | ||
|
|
||
| // all VectorFieldAttributes attributes | ||
| if (f.attributes.dimension) { | ||
| attributes.push("DIM", f.attributes.dimension.toString()); | ||
| } | ||
|
|
||
| if (f.attributes.distanceMetric) { | ||
| attributes.push( | ||
| "DISTANCE_METRIC", | ||
| f.attributes.distanceMetric.toString(), | ||
| ); | ||
| } | ||
|
|
||
| if (f.attributes.type) { | ||
| attributes.push("TYPE", f.attributes.type.toString()); | ||
| } | ||
|
|
||
| if (f.attributes.initialCap) { | ||
| attributes.push( | ||
| "INITIAL_CAP", | ||
| f.attributes.initialCap.toString(), | ||
| ); | ||
| } | ||
|
|
||
| // VectorFieldAttributesHnsw attributes | ||
| if ("m" in f.attributes && f.attributes.m) { | ||
| attributes.push("M", f.attributes.m.toString()); | ||
| } | ||
|
|
||
| if ( | ||
| "efContruction" in f.attributes && | ||
| f.attributes.efContruction | ||
| ) { | ||
| attributes.push( | ||
| "EF_CONSTRUCTION", | ||
| f.attributes.efContruction.toString(), | ||
| ); | ||
| } | ||
|
|
||
| if ("efRuntime" in f.attributes && f.attributes.efRuntime) { | ||
| attributes.push( | ||
| "EF_RUNTIME", | ||
| f.attributes.efRuntime.toString(), | ||
| ); | ||
| } | ||
|
|
||
| args.push(attributes.length.toString(), ...attributes); | ||
| } | ||
| }); | ||
|
|
||
| return _handleCustomCommand(client, args, { | ||
| decoder: Decoder.String, | ||
| }) as Promise<"OK" | null>; | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * @internal | ||
| */ | ||
| function _handleCustomCommand( | ||
| client: GlideClient | GlideClusterClient, | ||
| args: GlideString[], | ||
| decoderOption: DecoderOption, | ||
| ) { | ||
| return client instanceof GlideClient | ||
| ? (client as GlideClient).customCommand(args, decoderOption) | ||
| : (client as GlideClusterClient).customCommand(args, decoderOption); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| /** | ||
| * Copyright Valkey GLIDE Project Contributors - SPDX Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { GlideString } from "../BaseClient"; | ||
|
|
||
| interface BaseField { | ||
| /** The name of the field. */ | ||
| name: GlideString; | ||
| /** An alias for field. */ | ||
| alias?: GlideString; | ||
| } | ||
|
|
||
| /** | ||
| * If the field contains any blob of data. | ||
acarbonetto marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| */ | ||
| export type TextField = BaseField & { | ||
| /** Field identifier */ | ||
| type: "TEXT"; | ||
| }; | ||
|
|
||
| /** | ||
| * If the field contains a tag field. | ||
| */ | ||
| export type TagField = BaseField & { | ||
| /** Field identifier */ | ||
| type: "TAG"; | ||
| /** Specify how text in the attribute is split into individual tags. Must be a single character. */ | ||
| separator?: GlideString; | ||
| /** Preserve the original letter cases of tags. If set to False, characters are converted to lowercase by default. */ | ||
| caseSensitive?: boolean; | ||
| }; | ||
|
|
||
| /** | ||
| * If the field contains a number. | ||
| */ | ||
| export type NumericField = BaseField & { | ||
| /** Field identifier */ | ||
| type: "NUMERIC"; | ||
| }; | ||
|
|
||
| /** | ||
| * If the field is a vector field that supports vector search. | ||
| */ | ||
| export type VectorField = BaseField & { | ||
| /** Field identifier */ | ||
| type: "VECTOR"; | ||
Yury-Fridlyand marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| /** Additional attributes to be passed with the vector field after the algorithm name. */ | ||
| attributes: VectorFieldAttributesFlat | VectorFieldAttributesHnsw; | ||
| }; | ||
|
|
||
| /** | ||
| * Base class for defining vector field attributes to be used after the vector algorithm name. | ||
| */ | ||
| export interface VectorFieldAttributes { | ||
| /** Number of dimensions in the vector. Equivalent to DIM in the option. */ | ||
| dimension: number; | ||
| /** | ||
| * The distance metric used in vector type field. Can be one of [L2 | IP | COSINE]. | ||
| */ | ||
| distanceMetric: "L2" | "IP" | "COSINE"; | ||
Yury-Fridlyand marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| /** Vector type. The only supported type is FLOAT32. */ | ||
| type: "FLOAT32"; | ||
| /** | ||
| * Initial vector capacity in the index affecting memory allocation size of the index. Defaults to 1024. | ||
| */ | ||
| initialCap?: number; | ||
| } | ||
| export type VectorFieldAttributesFlat = VectorFieldAttributes & { | ||
| /** | ||
acarbonetto marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| * Vector field that supports vector search by FLAT (brute force) algorithm. | ||
| * The algorithm is a brute force linear processing of each vector in the index, yielding exact | ||
| * answers within the bounds of the precision of the distance computations. | ||
| */ | ||
| algorithm: "FLAT"; | ||
acarbonetto marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| }; | ||
| export type VectorFieldAttributesHnsw = VectorFieldAttributes & { | ||
| /** | ||
| * Vector field that supports vector search by HNSM (Hierarchical Navigable Small | ||
| * World) algorithm. | ||
| * The algorithm provides an approximation of the correct answer in exchange for substantially | ||
| * lower execution times. | ||
| */ | ||
| algorithm: "HNSW"; | ||
acarbonetto marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| /** | ||
| * Number of maximum allowed outgoing edges for each node in the graph in each layer. Default is 16, maximum is 512. | ||
| * Equivalent to the `m` attribute. | ||
| */ | ||
| numberOfEdges?: number; | ||
| /** | ||
| * Controls the number of vectors examined during index construction. Default value is 200, Maximum value is 4096. | ||
| * Equivalent to the `efContruction` attribute. | ||
| */ | ||
| vectorsExaminedOnConstruction?: number; | ||
| /** | ||
| * Controls the number of vectors examined during query operations. Default value is 10, Maximum value is 4096. | ||
| * Equivalent to the `efRuntime` attribute. | ||
| */ | ||
| vectorsExaminedOnRuntime?: number; | ||
| }; | ||
|
|
||
| export type Field = TextField | TagField | NumericField | VectorField; | ||
|
|
||
| /** | ||
| * Represents the input options to be used in the FT.CREATE command. | ||
| * All fields in this class are optional inputs for FT.CREATE. | ||
| */ | ||
| export interface FtCreateOptions { | ||
| /** The type of data to be indexed using FT.CREATE. */ | ||
| dataType: "JSON" | "HASH"; | ||
| /** The prefix of the key to be indexed. */ | ||
| prefixes?: GlideString[]; | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
small nit: with a negative lookahead removing the modules/pubsub/kill tests, would it be better to name this something like base rather than minimum?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like base. I originally called it 'fast'. The idea is that it skips the long tests, and could maybe be used for doing a sanity check on the client.