blackd: a HTTP server for blackening (pytest-dev#460)

Author: zsol

.appveyor.yml

@@ -1,10 +1,10 @@
 install:
 - C:\Python36\python.exe -m pip install mypy
-- C:\Python36\python.exe -m pip install -e .
+- C:\Python36\python.exe -m pip install -e .[d]
 
 # Not a C# project
 build: off
 
 test_script:
 - C:\Python36\python.exe tests/test_black.py
-- C:\Python36\python.exe -m mypy black.py tests/test_black.py
+- C:\Python36\python.exe -m mypy black.py blackd.py tests/test_black.py

.travis.yml

@@ -4,12 +4,12 @@ language: python
 cache: pip
 install:
 - pip install coverage coveralls flake8 flake8-bugbear mypy
-- pip install -e .
+- pip install -e '.[d]'
 script:
 - coverage run tests/test_black.py
-- if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then mypy black.py tests/test_black.py; fi
+- if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then mypy black.py blackd.py tests/test_black.py; fi
 - if [[ $TRAVIS_PYTHON_VERSION == '3.7' ]]; then black --check --verbose .; fi
-- if [[ $TRAVIS_PYTHON_VERSION == '3.8-dev' ]]; then flake8 black.py tests/test_black.py; fi
+- if [[ $TRAVIS_PYTHON_VERSION == '3.8-dev' ]]; then flake8 black.py blackd.py tests/test_black.py; fi
 after_success:
 - coveralls
 notifications:

Pipfile

@@ -4,10 +4,12 @@ verify_ssl = true
 name = "pypi"
 
 [packages]
+aiohttp = ">=3.3.2"
 attrs = ">=17.4.0"
 click = ">=6.5"
 appdirs = "*"
 toml = ">=0.9.4"
+black = {editable = true, path = ".", extras = ["d"]}
 
 [dev-packages]
 pre-commit = "*"

Pipfile.lock

@@ -35,6 +35,7 @@ Try it out now using the [Black Playground](https://black.now.sh).
 **[Code style](#the-black-code-style)** |
 **[pyproject.toml](#pyprojecttoml)** |
 **[Editor integration](#editor-integration)** |
+**[blackd](#blackd)** |
 **[Version control integration](#version-control-integration)** |
 **[Ignoring unmodified files](#ignoring-unmodified-files)** |
 **[Testimonials](#testimonials)** |
@@ -745,6 +746,76 @@ affect your use case.
 
 This can be used for example with PyCharm's [File Watchers](https://www.jetbrains.com/help/pycharm/file-watchers.html).
 
+## blackd
+
+`blackd` is a small HTTP server that exposes *Black*'s functionality over
+a simple protocol. The main benefit of using it is to avoid paying the
+cost of starting up a new *Black* process every time you want to blacken
+a file.
+
+### Usage
+
+`blackd` is not packaged alongside *Black* by default because it has additional
+dependencies. You will need to do `pip install black[d]` to install it.
+
+You can start the server on the default port, binding only to the local interface
+by running `blackd`. You will see a single line mentioning the server's version,
+and the host and port it's listening on. `blackd` will then print an access log
+similar to most web servers on standard output, merged with any exception traces
+caused by invalid formatting requests.
+
+`blackd` provides even less options than *Black*. You can see them by running
+`blackd --help`:
+
+```text
+Usage: blackd [OPTIONS]
+
+Options:
+  --bind-host TEXT                Address to bind the server to.
+  --bind-port INTEGER             Port to listen on
+  --version                       Show the version and exit.
+  -h, --help                      Show this message and exit.
+```
+
+### Protocol
+
+`blackd` only accepts `POST` requests at the `/` path. The body of the request
+should contain the python source code to be formatted, encoded 
+according to the `charset` field in the `Content-Type` request header. If no
+`charset` is specified, `blackd` assumes `UTF-8`.
+
+There are a few HTTP headers that control how the source is formatted. These
+correspond to command line flags for *Black*. There is one exception to this:
+`X-Protocol-Version` which if present, should have the value `1`, otherwise the
+request is rejected with `HTTP 501` (Not Implemented). 
+
+The headers controlling how code is formatted are:
+
+ - `X-Line-Length`: corresponds to the `--line-length` command line flag.
+ - `X-Skip-String-Normalization`: corresponds to the `--skip-string-normalization`
+    command line flag. If present and its value is not the empty string, no string
+    normalization will be performed.
+ - `X-Fast-Or-Safe`: if set to `fast`, `blackd` will act as *Black* does when
+    passed the `--fast` command line flag.
+ - `X-Python-Variant`: if set to `pyi`, `blackd` will act as *Black* does when
+    passed the `--pyi` command line flag. Otherwise, its value must correspond to
+    a Python version. If this value represents at least Python 3.6, `blackd` will
+    act as *Black* does when passed the `--py36` command line flag.
+
+If any of these headers are set to invalid values, `blackd` returns a `HTTP 400`
+error response, mentioning the name of the problematic header in the message body.
+
+Apart from the above, `blackd` can produce the following response codes:
+
+ - `HTTP 204`: If the input is already well-formatted. The response body is
+	empty.
+ - `HTTP 200`: If formatting was needed on the input. The response body
+	contains the blackened Python code, and the `Content-Type` header is set
+	accordingly.
+ - `HTTP 400`: If the input contains a syntax error. Details of the error are
+	returned in the response body.
+ - `HTTP 500`: If there was any kind of error while trying to format the input.
+	The response body contains a textual representation of the error.
 
 ## Version control integration
 
@@ -850,8 +921,14 @@ More details can be found in [CONTRIBUTING](CONTRIBUTING.md).
 
 ### 18.8b0
 
+* added `blackd`, see [its documentation](#blackd) for more info (#349)
+
 * adjacent string literals are now correctly split into multiple lines (#463)
 
+* added `blackd`, see [its documentation](#blackd) for more info (#349)
+
+* code with `_` in numeric literals is recognized as Python 3.6+ (#461)
+
 * numeric literals are now formatted by *Black* (#452, #461, #464, #469):
 
   * numeric literals are normalized to include `_` separators on Python 3.6+ code

README.md

@@ -79,15 +79,15 @@
 
 
 class NothingChanged(UserWarning):
-    """Raised by :func:`format_file` when reformatted code is the same as source."""
+    """Raised when reformatted code is the same as source."""
 
 
 class CannotSplit(Exception):
-    """A readable split that fits the allotted line length is impossible.
+    """A readable split that fits the allotted line length is impossible."""
 
-    Raised by :func:`left_hand_split`, :func:`right_hand_split`, and
-    :func:`delimiter_split`.
-    """
+
+class InvalidInput(ValueError):
+    """Raised when input source code fails all parse attempts."""
 
 
 class WriteBack(Enum):
@@ -676,7 +676,7 @@ def lib2to3_parse(src_txt: str) -> Node:
                 faulty_line = lines[lineno - 1]
             except IndexError:
                 faulty_line = "<line number missing in source>"
-            exc = ValueError(f"Cannot parse: {lineno}:{column}: {faulty_line}")
+            exc = InvalidInput(f"Cannot parse: {lineno}:{column}: {faulty_line}")
     else:
         raise exc from None
 

black.py

@@ -0,0 +1,106 @@
+import asyncio
+from concurrent.futures import Executor, ProcessPoolExecutor
+from functools import partial
+import logging
+
+from aiohttp import web
+import black
+import click
+
+# This is used internally by tests to shut down the server prematurely
+_stop_signal = asyncio.Event()
+
+VERSION_HEADER = "X-Protocol-Version"
+LINE_LENGTH_HEADER = "X-Line-Length"
+PYTHON_VARIANT_HEADER = "X-Python-Variant"
+SKIP_STRING_NORMALIZATION_HEADER = "X-Skip-String-Normalization"
+FAST_OR_SAFE_HEADER = "X-Fast-Or-Safe"
+
+
+@click.command(context_settings={"help_option_names": ["-h", "--help"]})
+@click.option(
+    "--bind-host", type=str, help="Address to bind the server to.", default="localhost"
+)
+@click.option("--bind-port", type=int, help="Port to listen on", default=45484)
+@click.version_option(version=black.__version__)
+def main(bind_host: str, bind_port: int) -> None:
+    logging.basicConfig(level=logging.INFO)
+    app = make_app()
+    ver = black.__version__
+    black.out(f"blackd version {ver} listening on {bind_host} port {bind_port}")
+    web.run_app(app, host=bind_host, port=bind_port, handle_signals=True, print=None)
+
+
+def make_app() -> web.Application:
+    app = web.Application()
+    executor = ProcessPoolExecutor()
+    app.add_routes([web.post("/", partial(handle, executor=executor))])
+    return app
+
+
+async def handle(request: web.Request, executor: Executor) -> web.Response:
+    try:
+        if request.headers.get(VERSION_HEADER, "1") != "1":
+            return web.Response(
+                status=501, text="This server only supports protocol version 1"
+            )
+        try:
+            line_length = int(
+                request.headers.get(LINE_LENGTH_HEADER, black.DEFAULT_LINE_LENGTH)
+            )
+        except ValueError:
+            return web.Response(status=400, text="Invalid line length header value")
+        py36 = False
+        pyi = False
+        if PYTHON_VARIANT_HEADER in request.headers:
+            value = request.headers[PYTHON_VARIANT_HEADER]
+            if value == "pyi":
+                pyi = True
+            else:
+                try:
+                    major, *rest = value.split(".")
+                    if int(major) == 3 and len(rest) > 0:
+                        if int(rest[0]) >= 6:
+                            py36 = True
+                except ValueError:
+                    return web.Response(
+                        status=400, text=f"Invalid value for {PYTHON_VARIANT_HEADER}"
+                    )
+        skip_string_normalization = bool(
+            request.headers.get(SKIP_STRING_NORMALIZATION_HEADER, False)
+        )
+        fast = False
+        if request.headers.get(FAST_OR_SAFE_HEADER, "safe") == "fast":
+            fast = True
+        mode = black.FileMode.from_configuration(
+            py36=py36, pyi=pyi, skip_string_normalization=skip_string_normalization
+        )
+        req_bytes = await request.content.read()
+        charset = request.charset if request.charset is not None else "utf8"
+        req_str = req_bytes.decode(charset)
+        loop = asyncio.get_event_loop()
+        formatted_str = await loop.run_in_executor(
+            executor,
+            partial(
+                black.format_file_contents,
+                req_str,
+                line_length=line_length,
+                fast=fast,
+                mode=mode,
+            ),
+        )
+        return web.Response(
+            content_type=request.content_type, charset=charset, text=formatted_str
+        )
+    except black.NothingChanged:
+        return web.Response(status=204)
+    except black.InvalidInput as e:
+        return web.Response(status=400, text=str(e))
+    except Exception as e:
+        logging.exception("Exception during handling a request")
+        return web.Response(status=500, text=str(e))
+
+
+if __name__ == "__main__":
+    black.patch_click()
+    main()

blackd.py

@@ -0,0 +1 @@
+_build/generated/blackd.md

docs/blackd.md

@@ -52,6 +52,7 @@ Contents
    the_black_code_style
    pyproject_toml
    editor_integration
+   blackd
    version_control_integration
    ignoring_unmodified_files
    contributing

docs/index.rst

@@ -29,3 +29,6 @@ check_untyped_defs=True
 
 # No incremental mode
 cache_dir=/dev/null
+
+[mypy-aiohttp.*]
+follow_imports=skip