Skip to content

CHANGELOG.md

Latest commit

 

History

History
148 lines (128 loc) · 8.06 KB

CHANGELOG.md

File metadata and controls

148 lines (128 loc) · 8.06 KB
Raw

Changelog

All notable changes to pyspark-connect-web ("PySpark in JupyterLite") are documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unofficial personal project. Not affiliated with, sponsored by, or endorsed by the Apache Software Foundation. "Apache Spark", "Spark", and "PySpark" are trademarks of the ASF, used here only to describe interoperability. See the README disclaimer.

Distribution name on PyPI: pyspark-connect-web. Import/package name: pyspark_connect_web. Both are pure-Python, py3-none-any, and (by invariant) carry no grpcio dependency - .

Unreleased

Added

  • Nothing yet.

0.2.0 - 2026-06-12

Reliability + JupyterLite-kernel release. Fixes the intermittent in-browser spark.sql hang and adds verified end-to-end coverage of the real JupyterLite Pyodide kernel (not just the standalone harness).

Fixed

  • SAB bridge deadlock on spark.sql - an eager command runs two server-streaming RPCs back-to-back; the second's S_REQ_READY could race the abandoned first stream's S_IDLE, leaving the main thread parked forever (intermittent ~30-50% hang, always spark.sql). The bridge now treats S_REQ_READY as "exchange abandoned" and re-dispatches the pending request; worker STATE writes are generation-guarded so a stale stream cannot stomp a newer RPC. Reproduced + regression-covered in tests/js/bridge.test.mjs.

Added

  • Real JupyterLite-kernel e2e (tests/e2e/kernel.spec.ts): boots the actual lite kernel, installs in-kernel, binds spark, and runs range / spark.sql / groupBy through the kernel SAB bridge - the integration the standalone harness cannot exercise.

Changed

  • JupyterLite bumped to core 0.7.6 / pyodide-kernel 0.7.2 (module-worker kernel; the 0.6.1 classic worker was incompatible with recent Pyodide, so the lite kernel never booted). Pyodide is vendored same-origin at the exact version the kernel expects (0.29.3), shared by the kernel and the harness; exposeAppInBrowser lets the kernel bootstrap reach the app.
  • Browser deliverable confirmed working end-to-end on both paths (harness + kernel), 9 e2e checks green with no flakes.

0.1.0 - 2026-06-12

First tagged release of the PySpark in JupyterLite client: run the real PySpark Connect Python client inside a browser (JupyterLite/Pyodide), talking to a Spark Connect server over a grpc-web transport. Existing PySpark code runs unchanged - no reimplementation, no local JVM, no Python backend.

Added

  • grpc-web transport - a fetch-based grpc-web stub (length-prefixed framing, 0x80 trailer frame) that replaces only PySpark's Connect service stub. We patch, we do not fork. Implements ExecutePlan / ReattachExecute / ReleaseExecute (see Known limitations for the browser reattach constraint).
  • Slim client (pyspark-client) - the browser loads the pure-Python Spark Connect client (pyspark-client, no JVM/py4j), installed via micropip with deps=False (its grpcio/grpcio-status base deps are stubbed by the shim). Verified against real Spark 4.0.0 and 4.1.2 servers; pcw.install() guards pyspark>=4.0.
  • JupyterLite/Pyodide bridge - a Web Worker + Atomics/SharedArrayBuffer channel that makes Connect calls blocking, so .collect() / .toPandas() return synchronously and the public PySpark API stays unchanged. Page-side Worker wrapping wires the bridge into the JupyterLite Pyodide kernel without forking it; a COI service worker covers header-less hosts. Cross-origin isolation (COOP/COEP) is mandatory and asserted.
  • Arrow result decoding - IPC reassembly to pandas, including SPARK-53525 multi-chunk split-batch handling, byte/row-exact against a native Connect reference.
  • pcw.install() - idempotent, version-guarded to pyspark>=4.0; raises a clear error outside the range. Accepts sc://host:port/;transport=grpcweb plus http(s):// shorthand.
  • Real Spark-Connect-verified Python vertical - the full v0 read-path matrix (range, select/filter/groupBy/agg, toPandas, createDataFrame, spark.sql(...), 200k-row multi-response stream, mid-stream reattach) proven against a real in-process Spark Connect server in tests/integration/ with a pure-Python grpc-web<->gRPC bridge standing in for Envoy.
  • Deploy stack - dev docker compose (Spark 4.1.2 Connect + Envoy grpc-web proxy + COOP/COEP static host) and a hardened prod overlay (TLS, exact-origin CORS, bearer-token gate, size limits, health/readiness).
  • Pure-Python wheel - py3-none-any, dependencies = [], no grpcio; CI guards the no-grpcio invariant at source, wheel-metadata, and import time.
  • Packaging & release automation - python -m build sdist + wheel, PyPI publish via API token on a vX.Y.Z tag, JupyterLite site built as a release asset, GitHub Release notes sourced from this changelog (.github/workflows/release.yml).

Known limitations

  • In-browser mid-stream reattach recovery is unavailable. PySpark recovers a dropped operation by reading INVALID_HANDLE.OPERATION_NOT_FOUND from the gRPC trailer status via grpcio-status, which grpc-web cannot carry in the browser. A browser query whose connection drops mid-stream errors rather than resuming. Reattach recovery over real gRPC IS verified server-side (tests/integration/); the browser e2e marks this case as a known skip.
  • The package does not yet ship a PEP 561 py.typed marker (owned by the package source lanes; tracked in CONTRIBUTING.md / docs/packaging-release.md).
  • AddArtifacts is lowered to a single grpc-web unary() POST (grpc-web has no true client streaming); not exercised end-to-end yet.