Skip to content

Commit 97c0d07

Browse files
oddgrdbrokadchesedoakrantz01Ivan
authored
chore: merge main into prod for v0.12.0 (#730)
* feat: shuttle-serenity initial commit poc (#429) * feat: shuttle-serenity initial commit poc * remove shuttle-service * refactor: drop more shuttle_service stuff * refactor: drop default serenity framework * misc: add wasm32-wasi to nix shell * refactor: cargo sort * refactor: cargo fmt Co-authored-by: chesedo <[email protected]> * shuttle next wrapper POC (#431) * feat: runtime skeleton * feat: cleanup * feat: update cargo.lock, sort deps * fix: typo * refactor: create runtimes workspace (#432) * refactor: rename plugins to runtimes * refactor: drop serenity folder * refactor: rename runtime to shuttle-next * refactor: prepare shuttle-legacy runtime * feat: add --provisioner-address arg to both runtimes (#433) * refactor: fix up dependencies * feat: add --provisioner-address to both runtimes * feat: create a control plane interface (part 1) (#436) * feat: add proto for runtimes * refactor: legacy move main to lib * feat: impl Runtime server for legacy * feat: impl Runtime server for next * refactor: combine runtimes into one (#438) * feat: add --legacy flag * refactor: merge next and legacy runtimes * refactor: merge proto crates * refactor: rename to runtime * refactor: promote runtime * refactor: update README.md * refactor: cargo sort * feat(deployer): add support for building wasm projects (#437) * feat: move factory to runtime (#444) * feat: initial commit of moving factory to runtime * feat: replace deployer factory with dummy, clean commented code * feat: update legacy runtime readme * WIP feat: start runtime from deployer (#450) * WIP feat: start runtime from deployer * feat: attempt to start runtime as a child process * fix: add current dir * feat: clean up, update readme * feat: use cargo manifest dir for runtime path * refactor: clean up, fmt * refactor: clippy warnings for Eq Co-authored-by: chesedo <[email protected]> * feat: get logs from runtime (#459) * feat: get logs from runtime * refactor: trim down deployer * refactor: start runtime earlier * refactor: connect to client earlier * refactor: hook runtime logs to persistence * bug: associate deployment id with logs * refactor: cleanup * feat: make sure grpc connection stays open * feat(shuttle-next): first edition of axum-wasm router (#472) * feat(shuttle-next): POC of axum wasm router * feat: use std mutex * feat: serialize/deserialize http requests to rmp this serializes/deserializes http requests across the ffi to the axum-wasm router, using the rust messagepack dataformat. sending the body is still a wip * feat: serialize the full HTTP req/res to rmp * fix: comment typo * feat: start hyper with tonic and serve wasm router * feat: clone inner router arc * feat: send request body without serialization * docs: todo comment * feat: write response body * fix: serialize response parts * feat: deserialize parts directly from reader this also adds a new fd to separate streaming of body and parts * feat: add axum-wasm to runtime cli * refactor: remove unused method * refactor: typo Co-authored-by: Pieter <[email protected]> * refactor: comments, clean up wrappers * refactor: move axum-wasm utils to shuttle-common * refactor: fmt * refactor: clippy Co-authored-by: Pieter <[email protected]> * Feat(shuttle-next): stop runtime services (#481) * feat: add tonic endpoint to stop runtime services * refactor: use tracing * feat: implement stop method for serenity bot * feat: install protoc 21.9 in dockerfile * refactor: comment * feat: convert uuid bytes to string then uuid * docs: update readme * docs: update readme for serenity bot * feat: implement stop for axum-wasm and document it * ci: install newer protoc in CI * feat: remove axum-wasm feature in runtime for now we will come back to this later and likely split the runtime by features * docs: remove feature flags from readme * fix: clippy * fix: clippy * fix: clippy * feat: send kill signal without spawning task * feat: return error response on kill_tx err * feat(next): expand macro into axum routes (#488) * feat: app codegen model * refactor: qualify all namespaces * feat: low-level wasi export fn * refactor: restrict to supported axum methods * Feat: parse shuttle::endpoint macro (#490) * feat: parse params * feat: parse app * feat: add test for missing endpoint attribute * test: invalid args, params and syntax * feat: check for and test no params * feat: don't emit error for missing annotation this way users can create separate functions for their endpoint logic * fix: remove missing attribute error, not param * refactor: reword comment Co-authored-by: Pieter <[email protected]> * feat: duplicate param test, address review * feat: only strip endpoint attributes * feat: check for extra endpoint attributes * refactor: clearer error span for extra endpoints * feat: has_err variable to group param errors * refactor: remove optional hint Co-authored-by: Pieter <[email protected]> * docs: add TODO for multiple endpoint attributes Co-authored-by: Pieter <[email protected]> * feat: merge main into shuttle-next (#543) * feat(blog): add missing sqlx migration code to auth blog post (#408) * feat(blog): add missing sqlx migration code * fix: nit * misc: 0.7.0 (#407) * misc: drop patches * infra: pin postgres to version 14 * refactor: fix thruster example app name * infra: db_fqdn fix * tests: warp hello world * fix(cargo-shuttle): prevent crash when config owned by root (#409) * fix: use correct timeout start point (#410) * fix: use correct timeout start point * tweak health check frequency * fmt * feat(deployer): implement container memory limits (#411) * feat(deployer): implement container memory limits * test: fix warp integration test * bug: `transport error` when trying to connect to provisioner (#416) * bug: keep provisioner connections alive * refactor: reduce scope provisioner client is running * feat: gateway admin revive (#412) * feat: gateway admin command (revive) * fmt * clippy * refactor: revive deployers using GatewayService * tests: add ContextArgs * refactor: simplify passing around of fqdn * tests: update test archive * refactor: remove stray exec.rs file * refactor: unused is_error() Co-authored-by: chesedo <[email protected]> * bug: timeout curl health check on deployer (#415) * Article/beta article (#420) * updated docs url * beta article w/ damiens corrections * Feat(www): shuttle beta signup (#421) * feat(www): add beta signup form * feat: add formspree endpoint * feat: remove socials footer from beta page * feat: add dummy text above sign up form * feat: placeholder gif * feat: align dummy text left * feat: remove gif, add text * feat: update beta page text * feat: update the rest of the site * bug: Fix thruster postgres example (#414) * feat: shell completions (#343) * bug: package Secrets.toml (#422) * bug: package Secrets.toml * refactor: clippy suggestion * bug: big archives being cut off at 32 768 bytes (#423) * refactor: set chunked header * refactor: don't send stream across threads Doing this is causing us to loose everything after 32 768 bytes. Don't know the reason why 🤷 * refactor: fix health check * refactor: remove unused use * chore: v0.7.1 (#424) * chore: v0.7.1 * chore: resources v0.7.1 * docs: v0.7.1 * tests: v0.7.1 * tests: fix warp test * tests: fix warp test * tests: fix secrets e2e test * ci: remove hard coded DD env * misc: unintended shell changes * refactor: cargo check suggestion * refactor: provide better context for errors (#430) * refactor: do our own health checks on deployer containers (#427) * refactor: do our own health checks on deployer containers We are doing this because the docker health checks eats up 25% of a single CPU core and we are spinning up many of these containers. https://www.reddit.com/r/docker/comments/b68r53/healtchecks_add_high_cpu_load/ moby/moby#39102 * refactor: clippy suggestion * bug: clear build folder before extracting (#428) * refactor: make sure extract directory is created * tests: test build folder is cleared * bug: clear the build directory * added some images (#435) * feat(www): beta blog updates (#434) * feat: add captioned image component (#440) * feat: add captioned image component * feat: update images * feat: add header caption * fix: quotes in header * www: post small tweaks (#439) * a few tweaks to the article * feat: update gif Co-authored-by: oddgrd <[email protected]> * feat: replace cursed gif (#441) * Feat/update contributing (#426) * feat(docs): update contributing.md * feat(e2e): make e2e admin user unique * feat(e2e): clean up e2e test projects on test complete * feat(e2e): don't fail on superuser conflict * feat: add section about contributing to docs * feat: remove unwrap, add comment to drop impl * tests: update e2e test key Co-authored-by: chesedo <[email protected]> * chore: v0.7.2 (#442) * chore: v0.7.2 * chore: resources v0.7.2 * docs: v0.7.2 * misc: v0.7.2 * misc: script to deploy all examples * misc: targets for publishing crates * misc: remove old scripts * misc: instructions for next steps * misc: get Cargo.lock changes * misc: update .PHONY * misc: show usage * feat: link the tracing spans between services (#445) * ci: DD environments * refactor: try lowering broadcast channel for quicker feedback * refactor: propagate tracing across threads in deployer * refactor: propagate tracing between gateway and deployer * refactor: trace account name * refactor: associate project with each container * feat: trace and propagate proxy in gateway * feat: gateway record project * refactor: tracing use 'error' for errors * refactor: fix comment * refactor: drop recording unused field * Add docker-compose extra flags param in Makefile (#446) * Add docker-compose extra flags param * rename to avoid confusion * misc: restructure repo (#453) * misc: only run e2e on production * misc: move www to shuttle-hq/www repo * misc: move examples to shuttle-hq/examples repo * refactor: update links to examples repo * ci: remove checking fmt of examples * ci: checkout submodules * refactor: tf files have been moved to shuttle-hq/terraform-aws-shuttle * fix: wrap around common::ProjectName for parsing (#451) * fix: gateway state drifts, health checks and project recreation (#447) * misc: add more helpful flags to Makefile * misc: remove old migrator (#463) * feat: add account_tier column (#458) * feat: prefetch shuttle-service crates (#461) * feat: prefetch shuttle-service crates * refactor: add comment to prepare.sh files * Feat: revive via gateway endpoint (#460) * feat: revive via gateway api initial commit * feat: cleanup, document gateway testing * refactor: use post for revive endpoint Co-authored-by: Damien <[email protected]> * feat: send task to global queue Co-authored-by: Damien <[email protected]> * feat: create a new admin cli binary crate (#462) * feat: init shuttle-admin * feat: add args * feat: simple client * feat: read api_key * refactor: hook it all together * refactor: switch to post * fix: broken link (#467) * fix: add timeout to health checks (#468) * fix: add timeout to health checks * Parameterise the timeout * rollback on inadvertent circleci change * misc: log out a warning when a task has been running for a long time * longer timeouts * Address review * Fmt * feat(gateway): add custom domains table and routing (#465) * WIP feat: count recent start events before restart (#469) * feat: count recent start events before restart * feat: try_collect and handle error in start counter * fix: clippy * refactor: gateway clippy * feat: make deployer only answer its own project (#466) * feat: make deployer only answer its own project * bug: use correct project name * refactor: make backwards compatible * Fixed Links in Readme (#477) * refactor: base client error off response status code (#470) * refactor: base client error off response status code * Update common/src/models/error.rs Co-authored-by: Oddbjørn Grødem <[email protected]> * refactor: use tracing Co-authored-by: Oddbjørn Grødem <[email protected]> * feat: verify project exists before sending destroy task (#474) * feat: add a custom domains admin route (#473) Co-authored-by: Alex Krantz <[email protected]> * bug: deployer freezes (#478) * bug: reduce spawning to have deployer lock up less * refactor: don't return cargo logs * ci: green (#482) * tests: allow longer time for build * refactor: trim dependencies on persist * refactor: clippy suggestion * feat: TLS acceptor with SNI resolver (#471) * fix: custom domain routing (#484) * refactor: more metrics (#475) * feat: add more metric dimensions to deployer * feat: add more metric dimensions to gateway * refactor: common metrics code * refactor: forward account name * refactor: add backend feature to deployer * refactor: standardize naming * refactor: cargo sort * misc: configurable deployment tags (#486) * feat: gateway restores removed containers (#485) * fix: backend bumps and hot fixes (#487) * Feature/support actix web (#491) * Support actix-web: impl Service * Support actix-web: cli + boilerplate code * Support actix-web: add framework to doc comment * Support actix-web: add e2e test * Support actix-web: fmt * Support actix-web: sorting toml deps * Support actix-web: add actix-web to ci * Support actix-web: formatted boilerplate code for actix-web hello world example * Support actix-web: formatted boilerplate code for actix-web hello world example Support actix-web: update cargo lock zstd dep * Support actix-web: simplify example * Support actix-web: add test, change example dependency * fix: e2e test assert, reset example module Co-authored-by: maksim <[email protected]> Co-authored-by: oddgrd <[email protected]> * Doc: Fix command to prime database with docker-compose (#502) Signed-off-by: Federico Guerinoni <[email protected]> Signed-off-by: Federico Guerinoni <[email protected]> * Doc: Improve contributing documentation (#499) In the doc it mentions to go to subfolder of examples but it was not there :). Signed-off-by: Federico Guerinoni <[email protected]> Signed-off-by: Federico Guerinoni <[email protected]> * feat: static file support for a single folder (#501) * feat: static-folder resource * refactor: make local client work * refactor: make local deployer work * feat: storage_manager * refactor: comments * refactor: remove unwraps and expects * refactor: update tests * refactor: clippy suggestion * refactor: update version * refactor: update readme * refactor: update comment * refactor: change public to static * ci: add static-folder * refactor: code fixes * refactor: update tests * refactor: update description Co-authored-by: Damien <[email protected]> Co-authored-by: Damien <[email protected]> * ci: build binary (#483) * ci: build binary * ci: make tar archive * ci: make GH draft release with artifacts * ci: add aarch64 * ci: get version from tag * ci: rename to linux builds * ci: add windows * ci: add macos * ci: test all * ci: put all the artifacts under a command * ci: only on tagged * ci: restore old * ci: fix windows * ci: put archive in separate folder * ci: switch to tar * ci: restore old * refactor: better `cargo-binstall` support Co-authored-by: Isaiah Gamble <[email protected]> * refactor: binary name Co-authored-by: Isaiah Gamble <[email protected]> Co-authored-by: Isaiah Gamble <[email protected]> * feat: bump rust to 1.64, bump dependencies (#495) * feat: bump rust to 1.64, correct contributing.md bug * feat: bump service deps, change example branch * fix: pin chrono to 4.22 * chore: upgrade workspace dependencies * feat: remove aws-smithy-types ref: smithy-lang/smithy-rs#1760 * feat: upgrade axum to 0.6 in common * feat: upgrade to axum 0.6.0 in deployer * feat: upgrade gateway to axum 0.6.0 * feat: upgrde sqlx to 0.6.2 in gateway * feat: bump chrono to 0.4.23 * feat: replace deprecated chrono functions in deployer * feat: bump chrono in common * feat: ignore actix integration test * feat: implement new state extractor in gateway * feat: interactive project initialization (#498) * feat: bump pinned rust version to 1.65 (#504) * feat: bump pinned rust version to 1.65 lockfile update is from binary dist PR #483 * ci: install newer protoc * fix: special module name warning * fix: clippy * feat: install newer protoc in dockerfile as well * misc: env updates (#509) * ci: update toolchain for binaries * misc: update dev environment * misc: remove patches from Cargo.lock * feat: make the folder configurable (#508) * feat: make the folder configurable * Update resources/static-folder/README.md Co-authored-by: Damien <[email protected]> Co-authored-by: Damien <[email protected]> * feat: implement workspace inheritance (#506) * refactor: switch away from cargo package (#507) * feat: add spinner wait for `project new` and `project status --follow` (#503) Signed-off-by: Federico Guerinoni <[email protected]> Co-authored-by: oddgrd <[email protected]> * fix: capitalise correctly (#511) * fix: make nice (#512) * feat: find (soon to be) invalid project names (#479) * feat: find (soon to be) invalid project names * refactor: move logic to admin client * refactor: missed axum 0.6 update (#513) * feat: build tests in release profile, limit build workers (#514) * feat: build tests in release profile, limit build workers * fix: typo in manifests * feat: create `init` project from correct dir (#518) * misc: interactive init gif (#519) * Feat/set examples submodule to main (#520) * feat: revert examples submodule back to main * feat: un-ignore actix tests * chore: 0.8.0 (#521) * chore: bump examples (#522) * bug: hacking static folders (#524) * fix: actix integration with state (#523) * feat(gateway,deployer): add more tracing events (#500) * feat(deployer): add more tracing events * feat(gateway): add more tracing events * feat: canonicalize before trace (#531) * feat: 'clean' subcommand (#530) * feat: 'clean' subcommand * refactor: output cleaning is done * Feat/set cpu limit (#529) * feat: remove redundant actix worker limit * feat: remove redundant build job limit * feat: set cpu limit for deployer container * fix: limit actix worker and build jobs to 4 * feat: add panamax for mirroring crates.io (#525) * feat: build queue (#532) * feat: per-project parallelism (#533) * feat: temp validation of project name in gateway (#534) * fix(deployer): keep Cargo.lock between deployments (#517) * ci: remove build and push req on build binaries (#535) * refactor: don't crash when failing to release slot (#536) * refactor: release build slot parse type correctly (#538) * refactor: remove prefetch (#539) * feat: add cron job for syncing mirror (#537) * chore: bump cargo-shuttle to 0.8.1 (#540) * chore: bump cargo-shuttle to 0.8.1 * chore: cargo.lock * refactor: update manifests and lockfile implement workspace inheritance for runtime, and remove default-features from shuttle-service * refactor: fmt * fix: unused imports, add runtime to deploymentmanager * feat: move storagemanager to common * fix: storage manager imports * fix: codegen test with outdated expected error * fix: service tests after removing service default features * fix: deployer tests after removing service default features * fix: typo * ci: attempt to fix protoc error * fix: deployer runtime client connection * refactor: remove redundant feature flags * refactor: remove storage_manager from deployer crate Signed-off-by: Federico Guerinoni <[email protected]> Co-authored-by: Pieter <[email protected]> Co-authored-by: Alexander Krantz <[email protected]> Co-authored-by: Damien <[email protected]> Co-authored-by: Ivan <[email protected]> Co-authored-by: Peter Mertz <[email protected]> Co-authored-by: XyLyXyRR <[email protected]> Co-authored-by: Damien <[email protected]> Co-authored-by: Nereuxofficial <[email protected]> Co-authored-by: Maxim <[email protected]> Co-authored-by: maksim <[email protected]> Co-authored-by: Federico Guerinoni <[email protected]> Co-authored-by: Isaiah Gamble <[email protected]> Co-authored-by: Stijn ("stain") Seghers <[email protected]> * refactor: one store per request (#510) * refactor: read from file * refactor: make store for every request * refactor: remove the serenity runtime (#549) * Feat: shuttle next bump deps (#551) * fix: axum tests * feat: bump wasmtime crates * feat: return streaming body from wasm router (#558) * feat: stream body to and from router * fix: reader to stream hack didn't work in wasm I added a post endpoint and test, which proved that my hacky stream from BufReader didn't work on the wasm side * refactor: test string * feat: update codegen with axum-wasm changes * refactor: clean up * refactor: typo * feat: guard against large request bodies * feat: embed runtime into client and deployer (#559) * refactor: runtime startup code * feat: hook cli to runtime * feat: hook up logs * feat: custom port * feat: start a next project with the local runner * feat: embed executable * refactor: axum update * refactor: tonic version from workspace * refactor: kill runtime correctly * feat: DB resources for local runs * feat: static folders for local runs * feat: secrets for local runner * refactor: cleanup logs and errors * refactor: rebuild runtime if it changed * refactor: more comments * feat: minimal axum logs * refactor: unneeded default features * refactor: fix rerun-if for runtime * bug: codegen handle_request * refactor: restore db error * bug: expanded broken merge (#567) * feat: refactor router and router inner (#566) * refactor: clean up imports * refactor: remove RouterInner * feat: handle wasi errors * feat: handle_request error handling * docs: comments * refactor: make hyper dev-dep, better error message * fix: axum-wasm missed import and bracket * feat: hook in runtime logs (#568) * feat: simple wasm log to send over unix socket * feat: hook up logs subscription * refactor: extra comments * refactor: simpler interface * refactor: combine JsonVisitors * refactor: clippy suggestions * refactor: update tests * refactor: extra comment * refactor: increase channel for better performance * refactor: move off stderr * misc: merge main (#572) * Merge remote-tracking branch 'upstream/main' into shuttle-next * refactor: touch ups * fix: missing feature flag in common dep (#573) * feat: change log read to spawn blocking (#574) * feat: DX (#577) * feat: shuttle-next crate for better DX * refactor: trim dependencies * bug: fix next check * ci: add shuttle-runtime * refactor: activating missing features on dependencies * refactor: update to shuttle_next::endpoint * feat: get runtime binary from cargo install (#578) * feat: get runtime binary from cargo install * refactor: remove build.rs * refactor: determine environment with debug assertions * ci: comment out cargo-sort installation * fix: clippy * feat: use cargo home dir, install runtime in prepare.sh * fix: unused import * refactor: build from local version in prepare.sh * fix: local debug run installing from incorrect path * feat: canonicalize path to debug runtime * feat: set release runtime install branch to prod * refactor: move secondary protoc install to common stage * feat: deployer next (#575) * feat: propagate next runtime * feat: store is_next in DB * feat: runtime manager to allow deployer to start up both runtimes * feat: make sure tests run * refactor: better migration query * refactor: handle runtime errors better * feat: shutdown runtimes * bug: missing so * bug: stop services * bug: ffi and runtime manager not living long enough * bug: missing so error * refactor: run cleanups * refactor: clippy suggestions * misc: remove unneeded codegen feature (#584) * refactor: proto fixups (#585) * refactor: comment * refactor: restore binding to external address * Feat: cleanup fds and tmp (#586) * feat: remove redundant body fd and /tmp * feat: update codegen * refactor: remove tmp from workspace exclude * ci: next patch and wasi target * refactor: unwraps and mor (#587) * refactor: better error handling * refactor: trim mutexes * refactor: remove abstract factory * refactor: remove extension todo * feat: build our images with the release profile (#583) * feat: build our images with the release profile Currently our containerfile builds images with the default debug profile, which means when `get_runtime_executable` is called it will try to install from a checked out local version of `shuttle-runtime`. * feat: add build arg for cargo profile * refactor: update comment * fix: broken codegen/axum test * feat: install wasm32-wasi target in deploy.sh * feat: build deps in `CARGO_PROFILE` * feat: use closure to determine shuttle-runtime path * fix: path to runtime in docker deployer, clippy * fix: incorrect function for cargo home * refactor: always use cargo_home for path to runtime in deployer * Feature/eng 378 axum wasm multiple handlers per endpoint (#588) * refactor: clean up if let * feat: move method validation, add test for it * test: add test for chained endpoint handlers * feat: chain endpoints with the same address * feat: duplicate method error, handler chain bug fix chaining handlers with a full type path caused a bug where a `,` was inserted, I think the reason is that `path` types can't be followed by `.`. Importing all the routing methods in `app` works, but it is not ideal * refactor: remove some clones * refactor: remove unwrap Co-authored-by: Pieter <[email protected]> * feat: use BTreeMap for building endpoint chains * fix: only add namespace to first handler in chain * fix: skip rustfmt for quote in app_to_token test rustfmt will add a comma to the expected output, which will also be done in the expanded macro output, but not in the stringified output * refactor: revert btreemap change * docs: document why we sort the endpoint chains --------- Co-authored-by: Pieter <[email protected]> * refactor: find code to wrap legacy runtime (#675) * refactor: find code to wrap legacy runtime * misc: update README with test instructions * feat: remove box self from services, remove syncwrapper from axum service (#677) * feat: remove box self from all services * feat: remove syncwrapper from axum integration * Fix/move docker profiles to dev (#674) * refactor: move docker profiles to dev.yml * docs: update comment in makefile * feat: extract next runtime into separate binary (#679) * feat: extract next runtime into separate binary * fix: remove startrequestsservice name * Feature: remove tokio runtime from all resources (#680) * feat: remove runtime from resources * fix: compile errors, comments * fix: remove comment * feat: trim the service loader, unpin tokio (#681) * feat: trim the serviceloader, unpin tokio * fix: tokio sync is needed * feat: unpin tokio in runtime as well * fix: static folder tests need tokio rt * feat: rename loader, build_crate return bin path * fix: build_Crate invalid args * chore(typos): fix typos (#682) * chore(typos): fix typos * Fix indentation for poise init example (#687) * Feature/eng 477 make wasm dependencies optional (#688) * feat: make wasm runtime dependencies optional * feat: feature gate next lib, update readme * ci: rename loader feature in ci * refactor: hyper with server feature in next features * Feature: eng 465 update all the codegens (#686) * feat: initial commit of codegen rewrite * refactor: wrapper to loader * tests: update the unit tests * fix: don't unwrap return type * feat: setup tracing in loader codegen * fix: pass inn logger to loader * feat: mvp of shuttle-next local run * refactor: move logger to runtime * refactor: move storagemanager export to runtime * docs: add note about git tags in contrib (#691) * bug: idle static folder (#692) * misc: add more logs * bug: copy static folder * bug: extra subdirectory * chore: v0.11.3 (#695) * Feature: eng 483 trim and fix the tests in shuttle-service (#693) * tests: update build_crate tests * tests: remove loader tests * feat: cleanup service deps * feat: setup integration tests in runtime * feat: expected panic message for not_shuttle * refactor: simplify dummy provisioner * feat: re-export main and service from runtime * fix: Respect `Cargo.lock` when building containers (#700) Building the shuttle containers was failing during the `cargo chef cook` step because of a cyclic dependency introduced by an upstream crate. This did not happen when building outside of a container, because the versions pinned by `Cargo.lock` did not contain any cycles. See tkaitchuck/aHash#95 * ci: resolve CI errors in shuttle-next (#580) * test: compile wasm module in axum runtime test setup * ci: add next patch override to CI * ci: include wasm32-wasi target in rust install * fix: deployer tests where runtime fails to start * fix: incorrect provisioner address * feat: log service state changes in runtime * feat: don't send stop req on startup failure * refactor: unused imports * refactor: handling legacy panics * tests: deadlock less * refactor: fixups * refactor: clippy suggestions * tests: mock provisioner * refactor: restore capture from 'log' and colors * refactor: clippy suggestions * tests: longer wait * tests: don't panic while holding lock * tests: don't panic on stream closed * tests: don't filter out state logs * tests: bigger timeout * ci: remove duplicate patch * refactor: comments --------- Co-authored-by: chesedo <[email protected]> * Merge remote-tracking branch 'upstream/main' into shuttle-next * Feature/eng 486 update deployer with runtime changes (#696) * feat: update deployer to work with runtime changes * feat: pass next feature flag to runtime installs * fix: revert this change, the error is a rust-analyzer bug * tests: make deployer test resources bins * ci: add shuttle-runtime patch to CI * refactor: consistent naming * fix: redundant clone of runtime manager * feat: delete rocket bin from runtime * feat: make storagemanager a trait object * docs: add comment about why we use arc * refactor: use &str in storagemanager * feat: add runtime patch to deployer * fix: deployer logic bug, add debug logs * feat: extract service integrations into separate crates (#702) * feat: draft of extracting service integrations * refactor: revert codegen changes * refactor: depend on shuttle_runtime & service integration * feat: remove rocket bin, comment out version check * feat: impl from for integration wrapper * feat: cleanup shuttle-axum docs, remove version check * feat: actix-web service integration * feat: extract poem service integration * feat: extract poise service integration * feat: extract rocket service integration * feat: extract salvo service integration * feat: extract the serenity service integration * feat: extract the thruster service integration * feat: extract warp service integration * feat: extract the tower service integration * feat: delete persist from service * feat: extract tide service integration * feat: update cargo.lock * feat: make service integration inner pub * fix: merge fixes * refactor: rename integrations, remove comment * ci: run check-standalone on services * feat: update test resources * ci: refactor workspace-clippy job * fix: add tokio dev dep to services * fix: remaining services tests * [next] tests: CI go green (#704) * tests: thread local tracing * tests: bind panic in runtime * tests: post create project params * tests: fix cargo-shuttle * misc: std missing from tracing * tests: update main test for shuttle runtime * ci: merge workspace jobs * ci: move service to platform tests * ci: better names * refactor: trim dependencies Trim dependencies to minimize the build times of user projects. * refactor: catch error and panics in main * tests: trim cargo-shuttle tests Make sure there is only one main test for each major vector / feature. * refactor: missing serde derive * refactor: missing serde std * ci: balance machines * tests: give more time * ci: bump images more * ci: bump workspace image * ci: 2xlarge not available '/ * feat: emit error when shuttle::main is named main (#707) * Feature: eng 484 update init with codegen changes (#706) * feat: update init boilerplate with shuttle-next changes * tests: update shuttle_init_tests * tests: ignore broken integration tests * feat: refactor init to create bin * fix: bug when passing path to init * fix: codegen clippy (#709) * fix: codegen clippy * fix: clippy * misc: v0.12.0-rc1 (#708) * misc: v0.12.0-rc1 * misc: publish proto * refactor: remove relative from build.rs * refactor: add description to proto * refactor: make runtime publishable * refactor: make actix publishable * feat: make shuttle-axum publishable * feat: make poem publishable * feat: make poise publishable * feat: make shuttle-rocket publishable * refactor: make warp publishable * refactor: make tower publishable * refactor: make tide publishable * refactor: make thruster publishable * refactor: make serenity publishable * refactor: make salvo publishable --------- Co-authored-by: oddgrd <[email protected]> * [next] refactor: update runtime manager (#711) * refactor: allow multiple runtimes to run at the same time This is needed so that a new deployment can enter the loading state without killing currently running runtimes. * refactor: extra comment * Feature: eng 497 update or remove the docs in shuttle (#710) * docs: update service docs * docs: move service crate-docs to runtime * docs: update main readme * tests: fix init test * refactor: rename legacy runtime to alpha (#713) * refactor: rename legacy runtime to alpha * refactor: move next to services (#714) * refactor: move next to services * ci: correct next * refactor: align folder * [next] refactor: remove ids from runtime (#712) * refactor: remove deployment id * refactor: create proto LogItem directly * refactor: more tracing for local binary * misc: add kondo to env * refactor: clippy suggestion * tests: update * feat: embed protoc in shuttle-proto (#715) * feat: embed protoc in shuttle-proto * fix: path to shuttle-next in runtime next-resource * [next] bug: communicating resources (#716) * refactor: send claim to provisioner * feat: recording of provisioned resources This returns provisioned resources from runtime so that they can be recorded by deployer. Closes #ENG-252. * refactor: restore resources for idle containers waking up * tests: next tests using wrong folder * tests: swap rocket for the more stable poem * refactor: pass auth_uri to runtime * misc: update all patches * feat(local): don't install next-runtime from git (#718) * fix: static folder local run clearing file contents, add missing tests in cargo-shuttle (#717) * fix: static folder local run clearing file contents * fix: clippy * fix: cargo shuttle init tests not logging in * fix: bail if non-interactive and --new without api-key * fix: update codegen tests * [next] refactor: tracing (#719) * refactor: use tracing setup helper * refactor: better trace propagation * refactor: clippy suggestion * fix: windows local run path bug (#721) * refactor: pull out vendored protoc from shuttle-proto (#726) * feat: use grpcio to avoid needing protoc * refactor: remove vendored protoc, revert to tonic otlp * [next] bug: misc fixes (#725) * bug: thread local logging breaking serenity * refactor: patch all crates * refactor: extend tokens to 15 minutes * bug: next runtime not sending stop signal (#728) * bug: next runtime not sending stop signal * refactor: better unwrap * chore: v0.12.0 (#727) * chore: v0.12.0 * chore: bump examples --------- Signed-off-by: Federico Guerinoni <[email protected]> Co-authored-by: Damien <[email protected]> Co-authored-by: chesedo <[email protected]> Co-authored-by: Alexander Krantz <[email protected]> Co-authored-by: Ivan <[email protected]> Co-authored-by: Peter Mertz <[email protected]> Co-authored-by: XyLyXyRR <[email protected]> Co-authored-by: Damien <[email protected]> Co-authored-by: Nereuxofficial <[email protected]> Co-authored-by: Maxim <[email protected]> Co-authored-by: maksim <[email protected]> Co-authored-by: Federico Guerinoni <[email protected]> Co-authored-by: Isaiah Gamble <[email protected]> Co-authored-by: Stijn ("stain") Seghers <[email protected]> Co-authored-by: Sara Tavares <[email protected]> Co-authored-by: HexPandaa <[email protected]> Co-authored-by: Kieren Davies <[email protected]>
1 parent b1c8aba commit 97c0d07

File tree

210 files changed

+9263
-7021
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

210 files changed

+9263
-7021
lines changed

.circleci/config.yml

Lines changed: 47 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,12 @@ executors:
99
docker-rust:
1010
docker:
1111
- image: cimg/rust:1.65.0
12+
resource_class: small
1213
image-ubuntu:
1314
machine:
1415
image: ubuntu-2204:2022.04.1
1516
docker_layer_caching: true
17+
resource_class: xlarge
1618

1719
# sscache steps are from this guide
1820
# https://medium.com/@edouard.oger/rust-caching-on-circleci-using-sccache-c996344f0115
@@ -86,18 +88,33 @@ commands:
8688
cat\<< EOF > ~/.cargo/config.toml
8789
[patch.crates-io]
8890
shuttle-service = { path = "$PWD/service" }
91+
shuttle-runtime = { path = "$PWD/runtime" }
92+
8993
shuttle-aws-rds = { path = "$PWD/resources/aws-rds" }
9094
shuttle-persist = { path = "$PWD/resources/persist" }
9195
shuttle-shared-db = { path = "$PWD/resources/shared-db" }
9296
shuttle-secrets = { path = "$PWD/resources/secrets" }
9397
shuttle-static-folder = { path = "$PWD/resources/static-folder" }
98+
99+
shuttle-axum = { path = "$PWD/services/shuttle-axum" }
100+
shuttle-actix-web = { path = "$PWD/services/shuttle-actix-web" }
101+
shuttle-next = { path = "$PWD/services/shuttle-next" }
102+
shuttle-poem = { path = "$PWD/services/shuttle-poem" }
103+
shuttle-poise = { path = "$PWD/services/shuttle-poise" }
104+
shuttle-rocket = { path = "$PWD/services/shuttle-rocket" }
105+
shuttle-salvo = { path = "$PWD/services/shuttle-salvo" }
106+
shuttle-serenity = { path = "$PWD/services/shuttle-serenity" }
107+
shuttle-thruster = { path = "$PWD/services/shuttle-thruster" }
108+
shuttle-tide = { path = "$PWD/services/shuttle-tide" }
109+
shuttle-tower = { path = "$PWD/services/shuttle-tower" }
110+
shuttle-warp = { path = "$PWD/services/shuttle-warp" }
94111
EOF
95112
install-rust:
96113
steps:
97114
- run:
98115
name: Install Rust
99116
command: |
100-
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
117+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --target add wasm32-wasi
101118
sudo apt update && sudo apt install -y libssl1.1
102119
install-protoc:
103120
steps:
@@ -146,33 +163,22 @@ commands:
146163
- << parameters.target >>.env
147164

148165
jobs:
149-
workspace-fmt:
166+
workspace:
150167
executor: docker-rust
168+
resource_class: xlarge
151169
steps:
152170
- checkout
153171
- restore-cargo-cache
154172
- install-protoc
155173
- run: cargo fmt --all --check
156-
- run: cargo install cargo-sort
157174
# TODO: this is incompatible with workspace inheritance, uncomment when
158175
# https://github.com/DevinR528/cargo-sort/pull/29 is merged
176+
# - run: cargo install cargo-sort
159177
# - run: cargo sort --check --workspace
160-
- run: cargo check --workspace --all-targets
161-
- save-cargo-cache
162-
workspace-clippy:
163-
parameters:
164-
framework:
165-
description: "Framework to activate"
166-
type: string
167-
executor: docker-rust
168-
steps:
169-
- checkout
170-
- restore-cargo-cache
171-
- install-protoc
172178
- run: |
173179
cargo clippy --tests \
174180
--all-targets \
175-
--features="codegen,loader,<< parameters.framework >>" \
181+
--all-features \
176182
--no-deps -- \
177183
--D warnings \
178184
-A clippy::let-unit-value \
@@ -190,9 +196,9 @@ jobs:
190196
- install-protoc
191197
- apply-patches
192198
- run: cargo fmt --all --check --manifest-path << parameters.path >>/Cargo.toml
193-
- run: cargo install cargo-sort
194199
# TODO: this is incompatible with workspace inheritance, uncomment when
195200
# https://github.com/DevinR528/cargo-sort/pull/29 is merged
201+
# - run: cargo install cargo-sort
196202
# - run: cargo sort --check << parameters.path >>
197203
- run: |
198204
cargo clippy --tests \
@@ -205,20 +211,6 @@ jobs:
205211
-A clippy::format-push-string
206212
- run: cargo test --all-features --manifest-path << parameters.path >>/Cargo.toml -- --nocapture
207213
- save-cargo-cache
208-
service-test:
209-
# Using an image since tests will start a docker container
210-
executor: image-ubuntu
211-
steps:
212-
- install-rust
213-
- checkout
214-
- restore-cargo-cache
215-
- run:
216-
name: Run unit tests
217-
command: cargo test --package shuttle-service --features="codegen,loader" --lib -- --nocapture
218-
- run:
219-
name: Run integration tests
220-
command: cargo test --package shuttle-service --features="codegen,loader" --test '*' -- --nocapture
221-
- save-cargo-cache
222214
platform-test:
223215
parameters:
224216
crate:
@@ -228,6 +220,7 @@ jobs:
228220
executor: image-ubuntu
229221
steps:
230222
- install-rust
223+
- install-protoc
231224
- checkout
232225
- run: git submodule sync
233226
- run: git submodule update --init
@@ -244,7 +237,6 @@ jobs:
244237
(cargo test --package << parameters.crate >> --all-features --test '*' -- --list 2>&1 | grep -q "no test target matches pattern") && echo "nothing to test" || cargo test --package << parameters.crate >> --all-features --test '*' -- --nocapture
245238
- save-cargo-cache
246239
e2e-test:
247-
resource_class: xlarge
248240
executor: image-ubuntu
249241
steps:
250242
- install-rust
@@ -276,7 +268,6 @@ jobs:
276268
key: docker-buildx-{{ .Branch }}
277269
when: always
278270
build-and-push:
279-
resource_class: xlarge
280271
executor: image-ubuntu
281272
steps:
282273
- checkout
@@ -350,7 +341,7 @@ jobs:
350341
build-binaries-mac:
351342
macos:
352343
xcode: 12.5.1
353-
resource_class: medium
344+
resource_class: xlarge
354345
steps:
355346
- checkout
356347
- run:
@@ -388,28 +379,9 @@ jobs:
388379
workflows:
389380
ci:
390381
jobs:
391-
- workspace-fmt
392-
- workspace-clippy:
393-
name: workspace-clippy-<< matrix.framework >>
394-
requires:
395-
- workspace-fmt
396-
matrix:
397-
parameters:
398-
framework:
399-
[
400-
"web-actix-web",
401-
"web-axum",
402-
"web-rocket",
403-
"web-poem",
404-
"web-thruster",
405-
"web-tide",
406-
"web-tower",
407-
"web-warp",
408-
"web-salvo",
409-
"bot-serenity",
410-
"bot-poise",
411-
]
382+
- workspace
412383
- check-standalone:
384+
name: << matrix.path >>
413385
matrix:
414386
parameters:
415387
path:
@@ -418,27 +390,38 @@ workflows:
418390
- resources/secrets
419391
- resources/shared-db
420392
- resources/static-folder
421-
- service-test:
422-
requires:
423-
- workspace-clippy
393+
- services/shuttle-actix-web
394+
- services/shuttle-axum
395+
- services/shuttle-next
396+
- services/shuttle-poem
397+
- services/shuttle-poise
398+
- services/shuttle-rocket
399+
- services/shuttle-salvo
400+
- services/shuttle-serenity
401+
- services/shuttle-thruster
402+
- services/shuttle-tide
403+
- services/shuttle-tower
404+
- services/shuttle-warp
424405
- platform-test:
406+
name: << matrix.crate >>
425407
requires:
426-
- workspace-clippy
408+
- workspace
427409
matrix:
428410
parameters:
429411
crate:
430412
[
431413
"shuttle-auth",
432-
"shuttle-deployer",
433414
"cargo-shuttle",
434415
"shuttle-codegen",
435416
"shuttle-common",
417+
"shuttle-deployer",
436418
"shuttle-proto",
437419
"shuttle-provisioner",
420+
"shuttle-runtime",
421+
"shuttle-service",
438422
]
439423
- e2e-test:
440424
requires:
441-
- service-test
442425
- platform-test
443426
- check-standalone
444427
filters:
@@ -454,23 +437,23 @@ workflows:
454437
name: build-binaries-x86_64-gnu
455438
image: ubuntu-2204:2022.04.1
456439
target: x86_64-unknown-linux-gnu
457-
resource_class: medium
440+
resource_class: xlarge
458441
filters:
459442
branches:
460443
only: production
461444
- build-binaries-linux:
462445
name: build-binaries-x86_64-musl
463446
image: ubuntu-2204:2022.04.1
464447
target: x86_64-unknown-linux-musl
465-
resource_class: medium
448+
resource_class: xlarge
466449
filters:
467450
branches:
468451
only: production
469452
- build-binaries-linux:
470453
name: build-binaries-aarch64
471454
image: ubuntu-2004:202101-01
472455
target: aarch64-unknown-linux-musl
473-
resource_class: arm.medium
456+
resource_class: arm.xlarge
474457
filters:
475458
branches:
476459
only: production

CONTRIBUTING.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@ git clone [email protected]:shuttle-hq/shuttle.git
2424
cd shuttle
2525
```
2626

27+
> Note: We need the git tags for the local development workflow, but they may not be included when you clone the repository.
28+
To make sure you have them, run `git fetch upstream --tags`, where upstream is the name of the shuttle remote repository.
29+
2730
You should now be ready to setup a local environment to test code changes to core `shuttle` packages as follows:
2831

2932
From the root of the shuttle repo, build the required images with:
@@ -53,13 +56,26 @@ In order to test local changes to the library crates, you may want to add the be
5356
```toml
5457
[patch.crates-io]
5558
shuttle-service = { path = "[base]/shuttle/service" }
56-
shuttle-common = { path = "[base]/shuttle/common" }
57-
shuttle-proto = { path = "[base]/shuttle/proto" }
59+
shuttle-runtime = { path = "[base]/shuttle/runtime" }
60+
5861
shuttle-aws-rds = { path = "[base]/shuttle/resources/aws-rds" }
5962
shuttle-persist = { path = "[base]/shuttle/resources/persist" }
6063
shuttle-shared-db = { path = "[base]/shuttle/resources/shared-db" }
6164
shuttle-secrets = { path = "[base]/shuttle/resources/secrets" }
6265
shuttle-static-folder = { path = "[base]/shuttle/resources/static-folder" }
66+
67+
shuttle-axum = { path = "[base]/shuttle/services/shuttle-axum" }
68+
shuttle-actix-web = { path = "[base]/shuttle/services/shuttle-actix-web" }
69+
shuttle-next = { path = "[base]/shuttle/services/shuttle-next" }
70+
shuttle-poem = { path = "[base]/shuttle/services/shuttle-poem" }
71+
shuttle-poise = { path = "[base]/shuttle/services/shuttle-poise" }
72+
shuttle-rocket = { path = "[base]/shuttle/services/shuttle-rocket" }
73+
shuttle-salvo = { path = "[base]/shuttle/services/shuttle-salvo" }
74+
shuttle-serenity = { path = "[base]/shuttle/services/shuttle-serenity" }
75+
shuttle-thruster = { path = "[base]/shuttle/services/shuttle-thruster" }
76+
shuttle-tide = { path = "[base]/shuttle/services/shuttle-tide" }
77+
shuttle-tower = { path = "[base]/shuttle/services/shuttle-tower" }
78+
shuttle-warp = { path = "[base]/shuttle/services/shuttle-warp" }
6379
```
6480

6581
Before we can login to our local instance of shuttle, we need to create a user.
@@ -164,7 +180,7 @@ To run the unit tests for a specific crate, from the root of the repository run:
164180
cargo test --package <crate-name> --all-features --lib -- --nocapture
165181
```
166182

167-
To run the integration tests for a spesific crate (if it has any), from the root of the repository run:
183+
To run the integration tests for a specific crate (if it has any), from the root of the repository run:
168184

169185
```bash
170186
# replace <crate-name> with the name of the crate to test, e.g. `cargo-shuttle`

0 commit comments

Comments
 (0)