Revert "Improve User Guide (#954)"

This reverts commit e52abb3.

Author: matthewmturner

docs/user-guide/src/SUMMARY.md

@@ -22,17 +22,16 @@
 - [Introduction](introduction.md)
 - [Example Usage](example-usage.md)
 - [Use as a Library](library.md)
-- [DataFusion CLI](cli.md)
 - [SQL Reference](sql/introduction.md)
 
   - [SELECT](sql/select.md)
   - [DDL](sql/ddl.md)
+    - [CREATE EXTERNAL TABLE](sql/ddl.md)
   - [Datafusion Specific Functions](sql/datafusion-functions.md)
 
-- [Ballista Distributed Compute](distributed/introduction.md)
-  - [Start a Ballista Cluster](distributed/deployment.md)
-    - [Cargo Install](distributed/cargo-install.md)
-    - [Docker](distributed/docker.md)
+- [Distributed](distributed/introduction.md)
+  - [Create a Ballista Cluster](distributed/deployment.md)
+    - [Docker](distributed/standalone.md)
     - [Docker Compose](distributed/docker-compose.md)
     - [Kubernetes](distributed/kubernetes.md)
     - [Raspberry Pi](distributed/raspberrypi.md)

docs/user-guide/src/cli.md

@@ -19,81 +19,5 @@
 
 ## Ballista Rust Client
 
-Ballista usage is very similar to DataFusion. Tha main difference is that the starting point is a `BallistaContext`
-instead of the DataFusion `ExecutionContext`. Ballista uses the same DataFrame API as DataFusion.
-
-The following code sample demonstrates how to create a `BallistaContext` to connect to a Ballista scheduler process.
-
-```rust
-let config = BallistaConfig::builder()
-    .set("ballista.shuffle.partitions", "4")
-    .build()?;
-
-// connect to Ballista scheduler
-let ctx = BallistaContext::remote("localhost", 50050, &config);
-```
-
-Here is a full example using the DataFrame API.
-
-```rust
-#[tokio::main]
-async fn main() -> Result<()> {
-    let config = BallistaConfig::builder()
-        .set("ballista.shuffle.partitions", "4")
-        .build()?;
-
-    // connect to Ballista scheduler
-    let ctx = BallistaContext::remote("localhost", 50050, &config);
-
-    let testdata = datafusion::arrow::util::test_util::parquet_test_data();
-
-    let filename = &format!("{}/alltypes_plain.parquet", testdata);
-
-    // define the query using the DataFrame trait
-    let df = ctx
-        .read_parquet(filename)?
-        .select_columns(&["id", "bool_col", "timestamp_col"])?
-        .filter(col("id").gt(lit(1)))?;
-
-    // print the results
-    df.show().await?;
-
-    Ok(())
-}
-```
-
-Here is a full example demonstrating SQL usage.
-
-```rust
-#[tokio::main]
-async fn main() -> Result<()> {
-    let config = BallistaConfig::builder()
-        .set("ballista.shuffle.partitions", "4")
-        .build()?;
-
-    // connect to Ballista scheduler
-    let ctx = BallistaContext::remote("localhost", 50050, &config);
-
-    let testdata = datafusion::arrow::util::test_util::arrow_test_data();
-
-    // register csv file with the execution context
-    ctx.register_csv(
-        "aggregate_test_100",
-        &format!("{}/csv/aggregate_test_100.csv", testdata),
-        CsvReadOptions::new(),
-    )?;
-
-    // execute the query
-    let df = ctx.sql(
-        "SELECT c1, MIN(c12), MAX(c12) \
-        FROM aggregate_test_100 \
-        WHERE c11 > 0.1 AND c11 < 0.9 \
-        GROUP BY c1",
-    )?;
-
-    // print the results
-    df.show().await?;
-
-    Ok(())
-}
-```
+The Rust client supports a `DataFrame` API as well as SQL. See the
+[TPC-H Benchmark Client](https://github.com/ballista-compute/ballista/tree/main/rust/benchmarks/tpch) for an example.

docs/user-guide/src/distributed/cargo-install.md

@@ -19,10 +19,8 @@
 
 # Deployment
 
-There are multiple ways that a Ballista cluster can be deployed.
+Ballista is packaged as Docker images. Refer to the following guides to create a Ballista cluster:
 
-- [Create a cluster using Cargo install](cargo-install.md)
-- [Create a cluster using Docker](docker.md)
+- [Create a cluster using Docker](standalone.md)
 - [Create a cluster using Docker Compose](docker-compose.md)
 - [Create a cluster using Kubernetes](kubernetes.md)
-- [Create a cluster on Raspberry Pi](raspberrypi.md)

docs/user-guide/src/distributed/client-rust.md

@@ -17,29 +17,11 @@
   under the License.
 -->
 
-# Starting a Ballista cluster using Docker Compose
+# Installing Ballista with Docker Compose
 
-Docker Compose is a convenient way to launch a cluster when testing locally.
-
-## Build Docker image
-
-There is no officially published Docker image so it is currently necessary to build the image from source instead.
-
-Run the following commands to clone the source repository and build the Docker image.
-
-```bash
-git clone [email protected]:apache/arrow-datafusion.git -b 5.1.0
-cd arrow-datafusion
-./dev/build-ballista-docker.sh
-```
-
-This will create an image with the tag `ballista:0.6.0`.
-
-## Start a cluster
-
-The following Docker Compose example demonstrates how to start a cluster using one scheduler process and one
-executor process, with the scheduler using etcd as a backing store. A data volume is mounted into each container
-so that Ballista can access the host file system.
+Docker Compose is a convenient way to launch a cluister when testing locally. The following Docker Compose example
+demonstrates how to start a cluster using a single process that acts as both a scheduler and an executor, with a data
+volume mounted into the container so that Ballista can access the host file system.
 
 ```yaml
 version: "2.2"
@@ -78,20 +60,4 @@ node cluster.
 docker-compose up
 ```
 
-This should show output similar to the following:
-
-```bash
-$ docker-compose up
-Creating network "ballista-benchmarks_default" with the default driver
-Creating ballista-benchmarks_etcd_1 ... done
-Creating ballista-benchmarks_ballista-scheduler_1 ... done
-Creating ballista-benchmarks_ballista-executor_1  ... done
-Attaching to ballista-benchmarks_etcd_1, ballista-benchmarks_ballista-scheduler_1, ballista-benchmarks_ballista-executor_1
-ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] Running with config:
-ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] work_dir: /tmp/.tmpLVx39c
-ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] concurrent_tasks: 4
-ballista-scheduler_1  | [2021-08-28T15:55:22Z INFO  ballista_scheduler] Ballista v0.6.0 Scheduler listening on 0.0.0.0:50050
-ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] Ballista v0.6.0 Rust Executor listening on 0.0.0.0:50051
-```
-
 The scheduler listens on port 50050 and this is the port that clients will need to connect to.

docs/user-guide/src/distributed/deployment.md

@@ -28,23 +28,25 @@ The foundational technologies in Ballista are:
 - [Apache Arrow](https://arrow.apache.org/) memory model and compute kernels for efficient processing of data.
 - [Apache Arrow Flight Protocol](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/) for efficient data transfer between processes.
 - [Google Protocol Buffers](https://developers.google.com/protocol-buffers) for serializing query plans.
-- [DataFusion](https://github.com/apache/arrow-datafusion/) for query execution.
+- [Docker](https://www.docker.com/) for packaging up executors along with user-defined code.
+
+## Architecture
+
+The following diagram highlights some of the integrations that will be possible with this unique architecture. Note that not all components shown here are available yet.
+
+![Ballista Architecture Diagram](img/ballista-architecture.png)
 
 ## How does this compare to Apache Spark?
 
 Although Ballista is largely inspired by Apache Spark, there are some key differences.
 
-- The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead
-  of GC pauses.
+- The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead of GC pauses.
 - Ballista is designed from the ground up to use columnar data, enabling a number of efficiencies such as vectorized
   processing (SIMD and GPU) and efficient compression. Although Spark does have some columnar support, it is still
   largely row-based today.
-- The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than
-  Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of
-  distributed compute.
-- The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors
-  in any programming language with minimal serialization overhead.
+- The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of distributed compute.
+- The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors in any programming language with minimal serialization overhead.
 
 ## Status
 
-Ballista is still in the early stages of development but is capable of executing complex analytical queries at scale.
+Ballista is at the proof-of-concept phase currently but is under active development by a growing community.