Skip to content

Commit 98f63a2

Browse files
acarbonettoBoazBD
authored andcommitted
Node: add FT.CREATE command (valkey-io#2501)
* Add FT.CREATE command for Node Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> --------- Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> Signed-off-by: BoazBD <boazbd@amazon.com>
1 parent 78a95b9 commit 98f63a2

File tree

8 files changed

+748
-210
lines changed

8 files changed

+748
-210
lines changed

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
* Java: Added `FT.SEARCH` ([#2439](https://github.com/valkey-io/valkey-glide/pull/2439))
1717
* Java: Added `FT.AGGREGATE` ([#2466](https://github.com/valkey-io/valkey-glide/pull/2466))
1818
* Java: Added `JSON.SET` and `JSON.GET` ([#2462](https://github.com/valkey-io/valkey-glide/pull/2462))
19+
* Node: Added `FT.CREATE` ([#2501](https://github.com/valkey-io/valkey-glide/pull/2501))
1920
* Java: Added `JSON.ARRINSERT` and `JSON.ARRLEN` ([#2476](https://github.com/valkey-io/valkey-glide/pull/2476))
2021
* Java: Added `JSON.DEL` and `JSON.FORGET` ([#2490](https://github.com/valkey-io/valkey-glide/pull/2490))
2122
* Java: Added `FT.ALIASADD`, `FT.ALIASDEL`, `FT.ALIASUPDATE` ([#2442](https://github.com/valkey-io/valkey-glide/pull/2442))

node/DEVELOPER.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,16 @@ To run the integration tests with existing servers, run the following command:
137137
138138
```bash
139139
npm run test -- --cluster-endpoints=localhost:7000 --standalone-endpoints=localhost:6379
140+
141+
# If those endpoints use TLS, add `--tls=true` (applies to both endpoints)
142+
npm run test -- --cluster-endpoints=localhost:7000 --standalone-endpoints=localhost:6379 --tls=true
143+
```
144+
145+
By default, the server_modules tests do not run using `npm run test`. After pointing to a server with JSON and VSS modules setup,
146+
run the following command:
147+
148+
```bash
149+
npm run test-modules
140150
```
141151
142152
### Submodules

node/index.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,6 @@ export * from "./src/GlideClient";
1010
export * from "./src/GlideClusterClient";
1111
export * from "./src/Logger";
1212
export * from "./src/server-modules/GlideJson";
13+
export * from "./src/server-modules/GlideFt";
14+
export * from "./src/server-modules/GlideFtOptions";
1315
export * from "./src/Transaction";

node/npm/glide/index.ts

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,14 @@ function initialize() {
118118
GlideClusterClient,
119119
GlideClientConfiguration,
120120
GlideJson,
121+
GlideFt,
122+
TextField,
123+
TagField,
124+
NumericField,
125+
VectorField,
126+
VectorFieldAttributesFlat,
127+
VectorFieldAttributesHnsw,
128+
FtCreateOptions,
121129
GlideRecord,
122130
GlideString,
123131
JsonGetOptions,
@@ -228,6 +236,14 @@ function initialize() {
228236
Decoder,
229237
DecoderOption,
230238
GeoAddOptions,
239+
GlideFt,
240+
TextField,
241+
TagField,
242+
NumericField,
243+
VectorField,
244+
VectorFieldAttributesFlat,
245+
VectorFieldAttributesHnsw,
246+
FtCreateOptions,
231247
GlideRecord,
232248
GlideJson,
233249
GlideString,

node/package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,8 @@
3232
"compile-protobuf-files": "cd src && pbjs -t static-module -o ProtobufMessage.js ../../glide-core/src/protobuf/*.proto && pbts -o ProtobufMessage.d.ts ProtobufMessage.js",
3333
"fix-protobuf-file": "replace 'this\\.encode\\(message, writer\\)\\.ldelim' 'this.encode(message, writer && writer.len ? writer.fork() : writer).ldelim' src/ProtobufMessage.js",
3434
"test": "npm run build-test-utils && jest --verbose --runInBand --testPathIgnorePatterns='ServerModules'",
35-
"test-modules": "npm run build-test-utils && jest --verbose --runInBand --testPathPattern='ServerModules'",
35+
"test-minimum": "npm run build-test-utils && jest --verbose --runInBand --testNamePattern='^(.(?!(GlideJson|GlideFt|pubsub|kill)))*$'",
36+
"test-modules": "npm run build-test-utils && jest --verbose --runInBand --testNamePattern='(GlideJson|GlideFt)'",
3637
"build-test-utils": "cd ../utils && npm i && npm run build",
3738
"lint:fix": "npm run install-linting && npx eslint -c ../eslint.config.mjs --fix && npm run prettier:format",
3839
"lint": "npm run install-linting && npx eslint -c ../eslint.config.mjs && npm run prettier:check:ci",

node/src/server-modules/GlideFt.ts

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
/**
2+
* Copyright Valkey GLIDE Project Contributors - SPDX Identifier: Apache-2.0
3+
*/
4+
5+
import { Decoder, DecoderOption, GlideString } from "../BaseClient";
6+
import { GlideClient } from "../GlideClient";
7+
import { GlideClusterClient } from "../GlideClusterClient";
8+
import { Field, FtCreateOptions } from "./GlideFtOptions";
9+
10+
/** Module for Vector Search commands */
11+
export class GlideFt {
12+
/**
13+
* Creates an index and initiates a backfill of that index.
14+
*
15+
* @param client The client to execute the command.
16+
* @param indexName The index name for the index to be created.
17+
* @param schema The fields of the index schema, specifying the fields and their types.
18+
* @param options Optional arguments for the `FT.CREATE` command. See {@link FtCreateOptions}.
19+
*
20+
* @returns If the index is successfully created, returns "OK".
21+
*
22+
* @example
23+
* ```typescript
24+
* // Example usage of FT.CREATE to create a 6-dimensional JSON index using the HNSW algorithm
25+
* await GlideFt.create(client, "json_idx1", [{
26+
* type: "VECTOR",
27+
* name: "$.vec",
28+
* alias: "VEC",
29+
* attributes: {
30+
* algorithm: "HNSW",
31+
* type: "FLOAT32",
32+
* dimension: 6,
33+
* distanceMetric: "L2",
34+
* numberOfEdges: 32,
35+
* },
36+
* }], {
37+
* dataType: "JSON",
38+
* prefixes: ["json:"]
39+
* });
40+
* ```
41+
*/
42+
static async create(
43+
client: GlideClient | GlideClusterClient,
44+
indexName: GlideString,
45+
schema: Field[],
46+
options?: FtCreateOptions,
47+
): Promise<"OK"> {
48+
const args: GlideString[] = ["FT.CREATE", indexName];
49+
50+
if (options) {
51+
if ("dataType" in options) {
52+
args.push("ON", options.dataType);
53+
}
54+
55+
if ("prefixes" in options && options.prefixes) {
56+
args.push(
57+
"PREFIX",
58+
options.prefixes.length.toString(),
59+
...options.prefixes,
60+
);
61+
}
62+
}
63+
64+
args.push("SCHEMA");
65+
66+
schema.forEach((f) => {
67+
args.push(f.name);
68+
69+
if (f.alias) {
70+
args.push("AS", f.alias);
71+
}
72+
73+
args.push(f.type);
74+
75+
switch (f.type) {
76+
case "TAG": {
77+
if (f.separator) {
78+
args.push("SEPARATOR", f.separator);
79+
}
80+
81+
if (f.caseSensitive) {
82+
args.push("CASESENSITIVE");
83+
}
84+
85+
break;
86+
}
87+
88+
case "VECTOR": {
89+
if (f.attributes) {
90+
args.push(f.attributes.algorithm);
91+
92+
const attributes: GlideString[] = [];
93+
94+
// all VectorFieldAttributes attributes
95+
if (f.attributes.dimension) {
96+
attributes.push(
97+
"DIM",
98+
f.attributes.dimension.toString(),
99+
);
100+
}
101+
102+
if (f.attributes.distanceMetric) {
103+
attributes.push(
104+
"DISTANCE_METRIC",
105+
f.attributes.distanceMetric.toString(),
106+
);
107+
}
108+
109+
if (f.attributes.type) {
110+
attributes.push(
111+
"TYPE",
112+
f.attributes.type.toString(),
113+
);
114+
}
115+
116+
if (f.attributes.initialCap) {
117+
attributes.push(
118+
"INITIAL_CAP",
119+
f.attributes.initialCap.toString(),
120+
);
121+
}
122+
123+
// VectorFieldAttributesHnsw attributes
124+
if ("m" in f.attributes && f.attributes.m) {
125+
attributes.push("M", f.attributes.m.toString());
126+
}
127+
128+
if (
129+
"efContruction" in f.attributes &&
130+
f.attributes.efContruction
131+
) {
132+
attributes.push(
133+
"EF_CONSTRUCTION",
134+
f.attributes.efContruction.toString(),
135+
);
136+
}
137+
138+
if (
139+
"efRuntime" in f.attributes &&
140+
f.attributes.efRuntime
141+
) {
142+
attributes.push(
143+
"EF_RUNTIME",
144+
f.attributes.efRuntime.toString(),
145+
);
146+
}
147+
148+
args.push(attributes.length.toString(), ...attributes);
149+
}
150+
151+
break;
152+
}
153+
154+
default:
155+
// no-op
156+
}
157+
});
158+
159+
return _handleCustomCommand(client, args, {
160+
decoder: Decoder.String,
161+
}) as Promise<"OK">;
162+
}
163+
}
164+
165+
/**
166+
* @internal
167+
*/
168+
function _handleCustomCommand(
169+
client: GlideClient | GlideClusterClient,
170+
args: GlideString[],
171+
decoderOption: DecoderOption,
172+
) {
173+
return client instanceof GlideClient
174+
? (client as GlideClient).customCommand(args, decoderOption)
175+
: (client as GlideClusterClient).customCommand(args, decoderOption);
176+
}
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
/**
2+
* Copyright Valkey GLIDE Project Contributors - SPDX Identifier: Apache-2.0
3+
*/
4+
5+
import { GlideString } from "../BaseClient";
6+
7+
interface BaseField {
8+
/** The name of the field. */
9+
name: GlideString;
10+
/** An alias for field. */
11+
alias?: GlideString;
12+
}
13+
14+
/**
15+
* Field contains any blob of data.
16+
*/
17+
export type TextField = BaseField & {
18+
/** Field identifier */
19+
type: "TEXT";
20+
};
21+
22+
/**
23+
* Tag fields are similar to full-text fields, but they interpret the text as a simple list of
24+
* tags delimited by a separator character.
25+
*
26+
* For HASH fields, separator default is a comma (`,`). For JSON fields, there is no default
27+
* separator; you must declare one explicitly if needed.
28+
*/
29+
export type TagField = BaseField & {
30+
/** Field identifier */
31+
type: "TAG";
32+
/** Specify how text in the attribute is split into individual tags. Must be a single character. */
33+
separator?: GlideString;
34+
/** Preserve the original letter cases of tags. If set to False, characters are converted to lowercase by default. */
35+
caseSensitive?: boolean;
36+
};
37+
38+
/**
39+
* Field contains a number.
40+
*/
41+
export type NumericField = BaseField & {
42+
/** Field identifier */
43+
type: "NUMERIC";
44+
};
45+
46+
/**
47+
* Superclass for vector field implementations, contains common logic.
48+
*/
49+
export type VectorField = BaseField & {
50+
/** Field identifier */
51+
type: "VECTOR";
52+
/** Additional attributes to be passed with the vector field after the algorithm name. */
53+
attributes: VectorFieldAttributesFlat | VectorFieldAttributesHnsw;
54+
};
55+
56+
/**
57+
* Base class for defining vector field attributes to be used after the vector algorithm name.
58+
*/
59+
export interface VectorFieldAttributes {
60+
/** Number of dimensions in the vector. Equivalent to DIM in the option. */
61+
dimension: number;
62+
/**
63+
* The distance metric used in vector type field. Can be one of [L2 | IP | COSINE].
64+
*/
65+
distanceMetric: "L2" | "IP" | "COSINE";
66+
/** Vector type. The only supported type is FLOAT32. */
67+
type: "FLOAT32";
68+
/**
69+
* Initial vector capacity in the index affecting memory allocation size of the index. Defaults to 1024.
70+
*/
71+
initialCap?: number;
72+
}
73+
74+
/**
75+
* Vector field that supports vector search by FLAT (brute force) algorithm.
76+
*
77+
* The algorithm is a brute force linear processing of each vector in the index, yielding exact
78+
* answers within the bounds of the precision of the distance computations.
79+
*/
80+
export type VectorFieldAttributesFlat = VectorFieldAttributes & {
81+
algorithm: "FLAT";
82+
};
83+
84+
/**
85+
* Vector field that supports vector search by HNSM (Hierarchical Navigable Small World) algorithm.
86+
*
87+
* The algorithm provides an approximation of the correct answer in exchange for substantially
88+
* lower execution times.
89+
*/
90+
export type VectorFieldAttributesHnsw = VectorFieldAttributes & {
91+
algorithm: "HNSW";
92+
/**
93+
* Number of maximum allowed outgoing edges for each node in the graph in each layer. Default is 16, maximum is 512.
94+
* Equivalent to the `m` attribute.
95+
*/
96+
numberOfEdges?: number;
97+
/**
98+
* Controls the number of vectors examined during index construction. Default value is 200, Maximum value is 4096.
99+
* Equivalent to the `efContruction` attribute.
100+
*/
101+
vectorsExaminedOnConstruction?: number;
102+
/**
103+
* Controls the number of vectors examined during query operations. Default value is 10, Maximum value is 4096.
104+
* Equivalent to the `efRuntime` attribute.
105+
*/
106+
vectorsExaminedOnRuntime?: number;
107+
};
108+
109+
export type Field = TextField | TagField | NumericField | VectorField;
110+
111+
/**
112+
* Represents the input options to be used in the FT.CREATE command.
113+
* All fields in this class are optional inputs for FT.CREATE.
114+
*/
115+
export interface FtCreateOptions {
116+
/** The type of data to be indexed using FT.CREATE. */
117+
dataType: "JSON" | "HASH";
118+
/** The prefix of the key to be indexed. */
119+
prefixes?: GlideString[];
120+
}

0 commit comments

Comments
 (0)