Skip to content

Commit 762552b

Browse files
XeXe
committed
add skypilot instructions
Signed-off-by: Xe Iaso <[email protected]>
1 parent 3b5b8ac commit 762552b

File tree

4 files changed

+136
-14
lines changed

4 files changed

+136
-14
lines changed

‎docs/blueprints/model-storage/fly-io.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ following environment variables in your deployments:
2929

3030
| Envvar name | Value |
3131
| ----------------------: | :----------------------------------------------------- |
32-
| `AWS_ACCESS_KEY_ID` | The access key ID from the runner keypair |
33-
| `AWS_SECRET_ACCESS_KEY` | The secret access key from the runner keypair |
32+
| `AWS_ACCESS_KEY_ID` | The access key ID from the workload keypair |
33+
| `AWS_SECRET_ACCESS_KEY` | The secret access key from the workload keypair |
3434
| `AWS_ENDPOINT_URL_S3` | `https://fly.storage.tigris.dev` |
3535
| `AWS_REGION` | `auto` |
3636
| `MODEL_PATH` | `ByteDance/SDXL-Lightning` |
@@ -43,16 +43,16 @@ Then create a GPU machine with an l40s GPU in it in Seattle:
4343
fly machine run \
4444
-a your-app-name-here \
4545
--name sdxl-lightning \
46-
-e AWS_ACCESS_KEY_ID=<runner-keypair-access-key-id> \
47-
-e AWS_SECRET_ACCESS_KEY=<runner-keypair-secret-access-key> \
46+
-e AWS_ACCESS_KEY_ID=<workload-keypair-access-key-id> \
47+
-e AWS_SECRET_ACCESS_KEY=<workload-keypair-secret-access-key> \
4848
-e AWS_ENDPOINT_URL_S3=https://fly.storage.tigris.dev \
4949
-e AWS_REGION=auto \
5050
-e MODEL_BUCKET_NAME=model-storage \
5151
-e PUBLIC_BUCKET_NAME=generated-images \
5252
-e MODEL_PATH=ByteDance/SDXL-Lightning \
5353
--vm-gpu-kind l40s \
5454
-r sea \
55-
ghcr.io/tigrisdata-community/runner/sdxl:latest \
55+
yasomimi/sdxl-tigris:latest:latest \
5656
-- python -m cog.server.http --host ::
5757
```
5858

‎docs/blueprints/model-storage/index.mdx‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ Tigris console and create a new access key. Don't assign any permissions to it.
119119
Copy the access key ID and secret access keys into either your notes or a
120120
password manager, you will not be able to see them again. These credentials will
121121
be used later to deploy your app in the cloud. This keypair will be referred to
122-
as the `runner-keypair` in this tutorial.
122+
as the `workload-keypair` in this tutorial.
123123

124124
Open `iam/model-runner.json` in your text editor. Change all references for
125125
`model-storage` and `generated-images` to the buckets you created earlier.
@@ -147,7 +147,7 @@ Attach it to the token you just created:
147147
```text
148148
aws iam attach-user-policy \
149149
--policy-arn arn:aws:iam::flyio_hunter2hunter2:policy/sdxl-runner \
150-
--user-name tid_runner_keypair_access_key_id
150+
--user-name tid_workload_keypair_access_key_id
151151
```
152152

153153
### Running inference
@@ -172,8 +172,8 @@ environment variables in your deployments:
172172

173173
| Envvar name | Value |
174174
| ----------------------: | :----------------------------------------------------- |
175-
| `AWS_ACCESS_KEY_ID` | The access key ID from the runner keypair |
176-
| `AWS_SECRET_ACCESS_KEY` | The secret access key from the runner keypair |
175+
| `AWS_ACCESS_KEY_ID` | The access key ID from the workload keypair |
176+
| `AWS_SECRET_ACCESS_KEY` | The secret access key from the workload keypair |
177177
| `AWS_ENDPOINT_URL_S3` | `https://fly.storage.tigris.dev` |
178178
| `AWS_REGION` | `auto` |
179179
| `MODEL_PATH` | `ByteDance/SDXL-Lightning` |
@@ -184,4 +184,5 @@ You can run this on any platform you want that has the right GPUs, however we
184184
have tutorials for a few platforms to try:
185185

186186
- [Fly.io](./fly-io)
187+
- [Skypilot](./skypilot)
187188
- [Vast.ai](./vast-ai)

‎docs/blueprints/model-storage/skypilot.md‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Using model weights in Tigris anywhere with Skypilot
2+
3+
:::note
4+
5+
The instructions on this page assume you have followed all of the steps in the
6+
main [Storing Model Weights in Tigris](/blueprints/model-storage) blueprint.
7+
8+
:::
9+
10+
[Skypilot](https://skypilot.readthedocs.io/en/latest/docs/index.html) is a tool
11+
that lets you route GPU compute to the cheapest possible locale based on your
12+
requirements. The same configuration lets you control AWS, Azure, Google Cloud,
13+
Oracle Cloud, Kubernetes, Runpod, Fluidstack, or more. For more information
14+
about Skypilot, check out
15+
[their documentation](https://skypilot.readthedocs.io/en/latest/docs/index.html).
16+
17+
To get started, you'll need to install Skypilot
18+
[following their directions](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
19+
Be sure to have [Conda](https://anaconda.org/anaconda/conda) installed.
20+
21+
You will need to configure your cloud of choice for this example. See
22+
[Skypilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#cloud-account-setup)
23+
on how to do this. We have tested this against a few clouds:
24+
25+
- [AWS](https://aws.amazon.com/)
26+
- [Lambda](https://lambdalabs.com/)
27+
- [Runpod](https://www.runpod.io/)
28+
29+
However the other providers should work fine.
30+
31+
## Customizing the `skypilot.yaml` file
32+
33+
Open `skypilot.yaml` in your favorite text editor. Customize the environment
34+
variables in the `envs:` key:
35+
36+
```yaml
37+
envs:
38+
# Tigris config
39+
AWS_ACCESS_KEY_ID: tid_AzureDiamond # workload access key ID
40+
AWS_SECRET_ACCESS_KEY: tsec_hunter2 # workload secret access key
41+
AWS_ENDPOINT_URL_S3: https://fly.storage.tigris.dev
42+
AWS_REGION: auto
43+
44+
# Bucket names
45+
MODEL_BUCKET_NAME: model-storage
46+
PUBLIC_BUCKET_NAME: generated-images
47+
48+
# Model to load
49+
MODEL_PATH: ByteDance/SDXL-Lightning
50+
```
51+
52+
| Envvar name | Value |
53+
| ----------------------: | :----------------------------------------------------- |
54+
| `AWS_ACCESS_KEY_ID` | The access key ID from the workload keypair |
55+
| `AWS_SECRET_ACCESS_KEY` | The secret access key from the workload keypair |
56+
| `AWS_ENDPOINT_URL_S3` | `https://fly.storage.tigris.dev` |
57+
| `AWS_REGION` | `auto` |
58+
| `MODEL_PATH` | `ByteDance/SDXL-Lightning` |
59+
| `MODEL_BUCKET_NAME` | `model-storage` (replace with your own bucket name) |
60+
| `PUBLIC_BUCKET_NAME` | `generated-images` (replace with your own bucket name) |
61+
62+
## Launching it in a cloud
63+
64+
Run `sky serve up` to start the image in a cloud:
65+
66+
```text
67+
sky serve up skypilot.yaml -n sdxl
68+
```
69+
70+
Wait a few minutes for everything to converge, and then you can use the endpoint
71+
URL to poke it:
72+
73+
```text
74+
⚙︎ Service registered.
75+
76+
Service name: sdxl
77+
Endpoint URL: 3.84.60.169:30001
78+
```
79+
80+
:::note
81+
82+
You can run `sky serve status` to find out if your endpoint is ready:
83+
84+
```text
85+
$ sky serve status
86+
<...>
87+
Service Replicas
88+
SERVICE_NAME ID VERSION ENDPOINT LAUNCHED RESOURCES STATUS REGION
89+
sdxl 1 1 http://69.30.85.69:22112 47 secs ago 1x RunPod({'RTXA4000': 1}) READY CA
90+
```
91+
92+
:::
93+
94+
Finally, run a test generation with this curl command:
95+
96+
```text
97+
curl "http://ip:port/predictions/$(uuidgen)" \
98+
-X PUT \
99+
-H "Content-Type: application/json" \
100+
--data-binary '{
101+
"input": {
102+
"prompt": "The space needle in Seattle, best quality, masterpiece",
103+
"aspect_ratio": "1:1",
104+
"guidance_scale": 3.5,
105+
"num_inference_steps": 4,
106+
"max_sequence_length": 512,
107+
"output_format": "png",
108+
"num_outputs": 1
109+
}
110+
}'
111+
```
112+
113+
If all goes well, you should get an image like this:
114+
115+
![The word 'success' in front of the Space Needle](./success.webp)
116+
117+
You can destroy the machine with this command:
118+
119+
```text
120+
sky serve down sdxl
121+
```

‎docs/blueprints/model-storage/vast-ai.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,8 @@ following environment variables in your deployments:
4747

4848
| Envvar name | Value |
4949
| ----------------------: | :----------------------------------------------------- |
50-
| `AWS_ACCESS_KEY_ID` | The access key ID from the runner keypair |
51-
| `AWS_SECRET_ACCESS_KEY` | The secret access key from the runner keypair |
50+
| `AWS_ACCESS_KEY_ID` | The access key ID from the workload keypair |
51+
| `AWS_SECRET_ACCESS_KEY` | The secret access key from the workload keypair |
5252
| `AWS_ENDPOINT_URL_S3` | `https://fly.storage.tigris.dev` |
5353
| `AWS_REGION` | `auto` |
5454
| `MODEL_PATH` | `ByteDance/SDXL-Lightning` |
@@ -59,16 +59,16 @@ Format all of your environment variables as you would in a `docker run` command.
5959
EG:
6060

6161
```text
62-
"-p 5000:5000 -e AWS_ACCESS_KEY_ID=<runner-keypair-access-key-id> -e AWS_SECRET_ACCESS_KEY=<runner-keypair-secret-access-key> -e AWS_ENDPOINT_URL_S3=https://fly.storage.tigris.dev -e AWS_REGION=auto -e MODEL_BUCKET_NAME=model-storage -e MODEL_PATH=ByteDance/SDXL-Lightning -e PUBLIC_BUCKET_NAME=generated-images"
62+
"-p 5000:5000 -e AWS_ACCESS_KEY_ID=<workload-keypair-access-key-id> -e AWS_SECRET_ACCESS_KEY=<workload-keypair-secret-access-key> -e AWS_ENDPOINT_URL_S3=https://fly.storage.tigris.dev -e AWS_REGION=auto -e MODEL_BUCKET_NAME=model-storage -e MODEL_PATH=ByteDance/SDXL-Lightning -e PUBLIC_BUCKET_NAME=generated-images"
6363
```
6464

6565
Then execute the launch command:
6666

6767
```text
6868
vastai create instance \
6969
<id-from-search> \
70-
--image ghcr.io/tigrisdata-community/runner/sdxl:latest \
71-
--env "-p 5000:5000 -e AWS_ACCESS_KEY_ID=<runner-keypair-access-key-id> -e AWS_SECRET_ACCESS_KEY=<runner-keypair-secret-access-key> -e AWS_ENDPOINT_URL_S3=https://fly.storage.tigris.dev -e AWS_REGION=auto -e MODEL_BUCKET_NAME=model-storage -e MODEL_PATH=ByteDance/SDXL-Lightning -e PUBLIC_BUCKET_NAME=generated-images" \
70+
--image yasomimi/sdxl-tigris:latest:latest \
71+
--env "-p 5000:5000 -e AWS_ACCESS_KEY_ID=<workload-keypair-access-key-id> -e AWS_SECRET_ACCESS_KEY=<workload-keypair-secret-access-key> -e AWS_ENDPOINT_URL_S3=https://fly.storage.tigris.dev -e AWS_REGION=auto -e MODEL_BUCKET_NAME=model-storage -e MODEL_PATH=ByteDance/SDXL-Lightning -e PUBLIC_BUCKET_NAME=generated-images" \
7272
--disk 48 \
7373
--onstart-cmd "python -m cog.server.http"
7474
```

0 commit comments

Comments
 (0)