Skip to content
This repository was archived by the owner on Jul 3, 2023. It is now read-only.

API hardening TODOs #357

Open
lalithsuresh opened this issue Apr 9, 2023 · 1 comment
Open

API hardening TODOs #357

lalithsuresh opened this issue Apr 9, 2023 · 1 comment

Comments

@lalithsuresh
Copy link
Contributor

lalithsuresh commented Apr 9, 2023

Having attempted to interact with the REST API from the commandline (curl + websocat), here are some rough issues I've ran into that I'll try to coordinate with @gz to fix:

  • Currently, there is no way to ergonomically query objects in the API using information supplied by the user. It often requires users to create objects, then do list queries (read more state than needed), parse the results, and then do follow up queries. For example, when trying to use the HTTP input transport, the endpoint at the pipeline uses a generated name (connector-). To find that endpoint from a commandline, I need toGET /configs, use jq to search for the uuid of the entry that has config="<stream name>", use an out of band process to find the pipeline port, and then use the retrieved UUID and port to reach <pipeline url>:<port>/input_endpoint/connector-<uuid>.
  • The above problem comes down to revisiting object naming and identifiers, uniqueness requirements, search in the API, with the server side generating identifiers. This also affects idempotency expectations (e.g., when rerunning python/test.py, we create projects, pipelines etc with the same shape every time).
  • The server needs to generate UUIDs instead of an incrementing counter.
  • Component and schema evolution (e.g., adding new transport config entries requires rebuilding the python client, and adding new types of connectors requires an enum type to be available at the client).
  • API versioning.
@lalithsuresh lalithsuresh changed the title API hardening API hardening TODOs Apr 9, 2023
@gz
Copy link
Contributor

gz commented Apr 9, 2023

Some other things that came up:

  • Need to expose status [stopped, starting, running...] of a pipeline through the API (in PipelineDesr)
  • Drop pipeline_id/metadata but return just PipelineDescr+metadata
  • Should be able to retrieve a config of the running pipeline
  • Should return error codes (number) when throwing an error
  • project: rename to program
  • pipeline: rename status to stats
  • uuid and name are unique for everything, have /id to retrieve and /?name= for everything
  • drop ConnectorType enum (KafkaIn, KafkaOut)
  • pipeline stop/start/shutdown replace with POST action to <pipeline/id>

@gz gz mentioned this issue Apr 28, 2023
7 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants