Skip to content

Commit 6c3bf27

Browse files
authored
Update readme (#9)
update readme
1 parent de322f8 commit 6c3bf27

File tree

1 file changed

+48
-8
lines changed

1 file changed

+48
-8
lines changed

README.md

Lines changed: 48 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,57 @@
11
# Jetstream-PyTorch
22
JetStream Engine implementation in PyTorch
33

4+
# Outline
45

5-
# Install
6+
1. Ssh to Cloud TPU VM (using v5e-8 TPU VM)
7+
a. Create a Cloud TPU VM if you haven’t
8+
2. Download jetstream-pytorch github repo
9+
3. Clone repo and install dependencies
10+
4. Download and convert weights
11+
5. Run checkpoint converter (quantizer)
12+
6. Local run
13+
7. Run the server
14+
8. Run benchmarks
15+
9. Typical Errors
616

7-
### 1. Get the jetstream-pytorch code
17+
# Ssh to Cloud TPU VM (using v5e-8 TPU VM)
18+
19+
```bash
20+
gcloud compute config-ssh
21+
gcloud compute tpus tpu-vm ssh "your-tpu-vm" --project "your-project" --zone "your-project-zone"
22+
```
23+
## Create a Cloud TPU VM in a GCP project if you haven’t
24+
Follow step 1-9 in the following guide
25+
* https://cloud.google.com/tpu/docs/v5e-inference#prepare-a-project
26+
27+
# Clone repo and install dependencies
28+
29+
## Get the jetstream-pytorch code
830
```bash
931
git clone https://github.com/google/jetstream-pytorch.git
1032
```
1133

12-
1.1 (optional) Create a virtual env using `venv` or `conda` and activate it.
34+
(optional) Create a virtual env using `venv` or `conda` and activate it.
1335

14-
### 2. Run installation script:
36+
## 2. Run installation script:
1537

1638
```bash
1739
cd jetstream-pytorch
1840
source install_everything.sh
1941
```
42+
NOTE: the above script will export PYTHONPATH, so sourcing will make it to take effect in the current shell
2043

44+
# Download and convert weights
2145

22-
# Get weights
23-
24-
### First get official llama weights from meta-llama
46+
## Get official llama weights from meta-llama
2547

2648
Following instructions here: https://github.com/meta-llama/llama#download
2749

2850
After you have downloaded the weights, it will also download a `tokenizer.model` file that is
2951
the tokenizer that we will use.
3052

31-
### Run weight merger to convert (and )
53+
## Run weight safetensor convert
54+
3255
```bash
3356
export input_ckpt_dir=Original llama weights directory
3457
export output_ckpt_dir=The output directory
@@ -73,3 +96,20 @@ export dataset_path=ShareGPT_V3_unfiltered_cleaned_split.json
7396
python benchmarks/benchmark_serving.py --tokenizer $tokenizer_path --num-prompts 2000 --dataset-path $dataset_path --dataset sharegpt --save-request-outputs
7497
```
7598
Please look at `deps/JetStream/benchmarks/README.md` for more information.
99+
100+
101+
# Typical Errors
102+
103+
## Unexpected keyword argument 'device'
104+
105+
Fix:
106+
* Uninstall jax and jaxlib dependencies
107+
* Reinstall using `source install_everything.sh
108+
109+
## Out of memory
110+
111+
Fix:
112+
* Use smaller batch size
113+
* Use quantization
114+
115+

0 commit comments

Comments
 (0)