Skip to content

Commit 666a503

Browse files
rishabhmauryarishabhmaurya
committed
Update documentation
Signed-off-by: Rishabh Maurya <[email protected]>
1 parent ecda165 commit 666a503

File tree

5 files changed

+165
-44
lines changed

5 files changed

+165
-44
lines changed

‎plugins/arrow-flight-rpc/README.md

Lines changed: 48 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,61 @@
1-
# arrow-flight-rpc
1+
# Arrow Flight RPC Plugin
22

3-
Enable this transport with:
3+
The Arrow Flight RPC plugin provides streaming transport for node to node communication in OpenSearch using Apache Arrow Flight protocol. It integrates with the OpenSearch Security plugin to provide secure, authenticated streaming with TLS encryption.
44

5-
```
6-
setting 'aux.transport.types', '[arrow-flight-rpc]'
7-
setting 'aux.transport.arrow-flight-rpc.port', '9400-9500' //optional
8-
```
5+
## Installation and Setup
96

10-
## Testing
7+
### Development Mode (./gradlew run)
118

12-
### Unit Tests
9+
For development using gradle:
1310

11+
1. Enable feature flag in `opensearch.yml`:
12+
```yaml
13+
opensearch.experimental.feature.transport.stream.enabled: true
1414
```
15-
./gradlew run \
16-
-PinstalledPlugins="['arrow-flight-rpc']" \
17-
-Dtests.opensearch.aux.transport.types="[experimental-transport-arrow-flight-rpc]" \
18-
-Dtests.opensearch.opensearch.experimental.feature.arrow.streams.enabled=true
15+
16+
2. Run with plugin:
17+
```bash
18+
./gradlew run -PinstalledPlugins="['arrow-flight-rpc']"
1919
```
2020

21-
### Unit Tests
21+
### Manual Setup
2222

23-
```
24-
./gradlew :plugins:arrow-flight-rpc:test
25-
```
23+
For manual configuration and deployment:
2624

27-
### Integration Tests
25+
1. Enable feature flag in `opensearch.yml`:
26+
```yaml
27+
opensearch.experimental.feature.transport.stream.enabled: true
28+
```
2829
30+
2. Add system properties and JVM options:
2931
```
30-
./gradlew :plugins:arrow-flight-rpc:internalClusterTest
32+
-Dio.netty.allocator.numDirectArenas=1
33+
-Dio.netty.noUnsafe=false
34+
-Dio.netty.tryUnsafe=true
35+
-Dio.netty.tryReflectionSetAccessible=true
36+
--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED
3137
```
38+
39+
3. Install and run the plugin manually
40+
41+
## Documentation
42+
43+
For detailed usage and architecture information, see the [docs](docs/) folder:
44+
45+
- [Architecture Guide](docs/architecture.md) - Stream transport architecture and design
46+
- [Server-side Streaming Guide](docs/server-side-streaming-guide.md) - How to implement server-side streaming
47+
- [Transport Client Streaming Flow](docs/transport-client-streaming-flow.md) - Client-side streaming implementation
48+
- [Flight Client Channel Flow](docs/flight-client-channel-flow.md) - Client channel flow details
49+
- [Metrics](docs/metrics.md) - Monitoring and performance metrics
50+
- [Error Handling](docs/error-handling.md) - Error handling patterns
51+
- [Security Integration](docs/security-integration.md) - Security plugin integration and TLS setup
52+
- [Chaos Testing](docs/chaos.md) - Chaos testing setup and usage
53+
- [Netty4 vs Flight Comparison](docs/netty4-vs-flight-comparison.md) - Transport classes comparison cheat sheet
54+
55+
## Examples
56+
57+
See the [stream-transport-example](../examples/stream-transport-example/) plugin for a complete example of how to implement streaming transport actions.
58+
59+
## Limitations
60+
61+
- **REST Client Support**: Arrow Flight streaming is not available for REST API clients. It only works for node-to-node transport within the OpenSearch cluster.

‎plugins/arrow-flight-rpc/build.gradle

Lines changed: 0 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -95,32 +95,6 @@ internalClusterTest {
9595
systemProperty 'io.netty.tryUnsafe', 'true'
9696
systemProperty 'io.netty.tryReflectionSetAccessible', 'true'
9797
jvmArgs += ["--add-opens", "java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED"]
98-
99-
// Enable chaos testing via bytecode injection
100-
doFirst {
101-
def agentJar = createChaosAgent()
102-
jvmArgs "-javaagent:${agentJar}"
103-
}
104-
}
105-
106-
// Task to create chaos agent JAR
107-
def createChaosAgent() {
108-
def agentJar = file("${buildDir}/chaos-agent.jar")
109-
110-
if (!agentJar.exists()) {
111-
def manifestFile = file("${buildDir}/MANIFEST.MF")
112-
manifestFile.text = '''Manifest-Version: 1.0
113-
Premain-Class: org.opensearch.arrow.flight.chaos.ChaosAgent
114-
Agent-Class: org.opensearch.arrow.flight.chaos.ChaosAgent
115-
Can-Redefine-Classes: true
116-
Can-Retransform-Classes: true
117-
'''
118-
ant.jar(destfile: agentJar, manifest: manifestFile) {
119-
fileset(dir: sourceSets.internalClusterTest.output.classesDirs.first(), includes: 'org/opensearch/arrow/flight/chaos/ChaosAgent*.class')
120-
}
121-
}
122-
123-
return agentJar.absolutePath
12498
}
12599

126100
spotless {

‎plugins/arrow-flight-rpc/docs/chaos.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Chaos Testing
2+
3+
The Arrow Flight RPC plugin includes chaos testing capabilities to simulate network failures and test resilience.
4+
5+
## Enabling Chaos Testing
6+
7+
Chaos testing is disabled by default. To enable it, modify the `build.gradle` file:
8+
9+
### 1. Add Chaos Agent to internalClusterTest
10+
11+
Add this to the `internalClusterTest` task:
12+
13+
```gradle
14+
internalClusterTest {
15+
// Enable chaos testing via bytecode injection
16+
doFirst {
17+
def agentJar = createChaosAgent()
18+
jvmArgs "-javaagent:${agentJar}"
19+
}
20+
}
21+
```
22+
23+
### 2. Add Chaos Agent Creation Task
24+
25+
Add this task to create the chaos agent JAR:
26+
27+
```gradle
28+
// Task to create chaos agent JAR
29+
def createChaosAgent() {
30+
def agentJar = file("${buildDir}/chaos-agent.jar")
31+
32+
if (!agentJar.exists()) {
33+
def manifestFile = file("${buildDir}/MANIFEST.MF")
34+
manifestFile.text = '''Manifest-Version: 1.0
35+
Premain-Class: org.opensearch.arrow.flight.chaos.ChaosAgent
36+
Agent-Class: org.opensearch.arrow.flight.chaos.ChaosAgent
37+
Can-Redefine-Classes: true
38+
Can-Retransform-Classes: true
39+
'''
40+
ant.jar(destfile: agentJar, manifest: manifestFile) {
41+
fileset(dir: sourceSets.internalClusterTest.output.classesDirs.first(), includes: 'org/opensearch/arrow/flight/chaos/ChaosAgent*.class')
42+
}
43+
}
44+
45+
return agentJar.absolutePath
46+
}
47+
```
48+
49+
## Running Chaos Tests
50+
51+
Once enabled, run the chaos tests with:
52+
53+
```bash
54+
./gradlew :plugins:arrow-flight-rpc:internalClusterTest --tests="*Chaos*"
55+
```
56+
57+
## What Chaos Testing Does
58+
59+
The chaos testing framework:
60+
- Injects bytecode to simulate network failures
61+
- Tests client-side resilience to connection drops
62+
- Validates proper error handling and recovery
63+
- Ensures graceful degradation under adverse conditions

‎plugins/arrow-flight-rpc/docs/metrics.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,22 @@ This returns metrics for all nodes. To get metrics for a specific node:
1616
GET /_flight/stats/{node_id}
1717
```
1818

19+
## Monitoring Streaming Tasks
20+
21+
Streaming transport tasks can be monitored using the existing Tasks API:
22+
23+
```bash
24+
curl "localhost:9200/_cat/tasks?v"
25+
```
26+
27+
Streaming tasks are identified by the `stream-transport` type:
28+
29+
```
30+
action task_id parent_task_id type start_time timestamp running_time ip node
31+
indices:data/read/search TVk0SciMQtSwplV6rQwyMA:2165 - transport 1754082449785 21:07:29 169.5ms 127.0.0.1 node-1
32+
indices:data/read/search[phase/query] TVk0SciMQtSwplV6rQwyMA:2166 TVk0SciMQtSwplV6rQwyMA:2165 stream-transport 1754082449786 21:07:29 168.4ms 127.0.0.1 node-1
33+
```
34+
1935
## Metrics Structure
2036

2137
Metrics are organized into the following categories:

‎plugins/arrow-flight-rpc/docs/security-integration.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Security Plugin Integration
2+
3+
The Arrow Flight RPC plugin integrates with the OpenSearch Security plugin to provide secure streaming transport with TLS encryption.
4+
5+
## Configuration
6+
7+
Add these settings to `opensearch.yml`:
8+
9+
```yaml
10+
# Enable streaming transport
11+
opensearch.experimental.feature.transport.stream.enabled: true
12+
13+
# Use secure Flight as default transport
14+
transport.stream.type.default: FLIGHT-SECURE
15+
16+
# Enable Flight TLS
17+
flight.ssl.enable: true
18+
```
19+
20+
## Security Plugin Setup
21+
22+
Install and configure the security plugin:
23+
24+
```bash
25+
# Install security plugin
26+
bin/opensearch-plugin install opensearch-security
27+
28+
# Setup demo configuration
29+
plugins/opensearch-security/tools/install_demo_configuration.sh
30+
```
31+
32+
## Role-Based Access Control
33+
34+
The Flight transport supports all security plugin features:
35+
- Index-level permissions
36+
- Document-level security (DLS)
37+
- Field-level security (FLS)
38+
- Action-level permissions

0 commit comments

Comments
 (0)