Deploying and scaling up with SkyPilot

vLLM

vLLM can be run and scaled to multiple service replicas on clouds and Kubernetes with SkyPilot, an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc, can be found in SkyPilot AI gallery.

Prerequisites

  • Go to the HuggingFace model page and request access to the model meta-llama/Meta-Llama-3-8B-Instruct.

  • Check that you have installed SkyPilot (docs).

  • Check that sky check shows clouds or Kubernetes are enabled.

  1. pip install skypilot-nightly
  2. sky check

Run on a single instance

See the vLLM SkyPilot YAML for serving, serving.yaml.

  1. resources:
  2.     accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
  3.     use_spot: True
  4.     disk_size: 512  # Ensure model checkpoints can fit.
  5.     disk_tier: best
  6.     ports: 8081  # Expose to internet traffic.
  8. envs:
  9.     MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
  10.     HF_TOKEN: <your-huggingface-token>  # Change to your own huggingface token, or use --env to pass.
  12. setup: |
  13.     conda create -n vllm python=3.10 -y
  14.     conda activate vllm
  16.     pip install vllm==0.4.0.post1
  17.     # Install Gradio for web UI.
  18.     pip install gradio openai
  19.     pip install flash-attn==2.5.7
  21. run: |
  22.     conda activate vllm
  23.     echo 'Starting vllm api server...'
  24.     python -u -m vllm.entrypoints.openai.api_server \
  25.         --port 8081 \
  26.         --model $MODEL_NAME \
  27.         --trust-remote-code \
  28.         --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
  29.         2>&1 | tee api_server.log &
  31.     echo 'Waiting for vllm api server to start...'
  32.     while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
  34.     echo 'Starting gradio server...'
  35.     git clone https://github.com/vllm-project/vllm.git || true
  36.     python vllm/examples/gradio_openai_chatbot_webserver.py \
  37.         -m $MODEL_NAME \
  38.         --port 8811 \
  39.         --model-url http://localhost:8081/v1 \
  40.         --stop-token-ids 128009,128001

Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, …):

  1. HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN

Check the output of the command. There will be a shareable gradio link (like the last line of the following). Open it in your browser to use the LLaMA model to do the text completion.

  1. (task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live

Optional: Serve the 70B model instead of the default 8B and use more GPU:

  1. HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct

Scale up to multiple replicas

SkyPilot can scale up the service to multiple service replicas with built-in autoscaling, load-balancing and fault-tolerance. You can do it by adding a services section to the YAML file.

  1. service:
  2.     replicas: 2
  3.     # An actual request for readiness probe.
  4.     readiness_probe:
  5.         path: /v1/chat/completions
  6.         post_data:
  7.         model: $MODEL_NAME
  8.         messages:
  9.             - role: user
  10.             content: Hello! What is your name?
  11.     max_tokens: 1

Click to see the full recipe YAML

  1. service:
  2.     replicas: 2
  3.     # An actual request for readiness probe.
  4.     readiness_probe:
  5.         path: /v1/chat/completions
  6.         post_data:
  7.         model: $MODEL_NAME
  8.         messages:
  9.             - role: user
  10.             content: Hello! What is your name?
  11.     max_tokens: 1
  13. resources:
  14.     accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
  15.     use_spot: True
  16.     disk_size: 512  # Ensure model checkpoints can fit.
  17.     disk_tier: best
  18.     ports: 8081  # Expose to internet traffic.
  20. envs:
  21.     MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
  22.     HF_TOKEN: <your-huggingface-token>  # Change to your own huggingface token, or use --env to pass.
  24. setup: |
  25.     conda create -n vllm python=3.10 -y
  26.     conda activate vllm
  28.     pip install vllm==0.4.0.post1
  29.     # Install Gradio for web UI.
  30.     pip install gradio openai
  31.     pip install flash-attn==2.5.7
  33. run: |
  34.     conda activate vllm
  35.     echo 'Starting vllm api server...'
  36.     python -u -m vllm.entrypoints.openai.api_server \
  37.         --port 8081 \
  38.         --model $MODEL_NAME \
  39.         --trust-remote-code \
  40.         --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
  41.         2>&1 | tee api_server.log &
  43.     echo 'Waiting for vllm api server to start...'
  44.     while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
  46.     echo 'Starting gradio server...'
  47.     git clone https://github.com/vllm-project/vllm.git || true
  48.     python vllm/examples/gradio_openai_chatbot_webserver.py \
  49.         -m $MODEL_NAME \
  50.         --port 8811 \
  51.         --model-url http://localhost:8081/v1 \
  52.         --stop-token-ids 128009,128001

Start the serving the Llama-3 8B model on multiple replicas:

  1. HF_TOKEN="your-huggingface-token" sky serve up -n vllm serving.yaml --env HF_TOKEN

Wait until the service is ready:

  1. watch -n10 sky serve status vllm

Example outputs:

  1. Services
  2. NAME  VERSION  UPTIME  STATUS  REPLICAS  ENDPOINT
  3. vllm  1        35s     READY   2/2       xx.yy.zz.100:30001
  5. Service Replicas
  6. SERVICE_NAME  ID  VERSION  IP            LAUNCHED     RESOURCES          STATUS  REGION
  7. vllm          1   1        xx.yy.zz.121  18 mins ago  1x GCP({'L4': 1})  READY   us-east4
  8. vllm          2   1        xx.yy.zz.245  18 mins ago  1x GCP({'L4': 1})  READY   us-east4

After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:

  1. ENDPOINT=$(sky serve status --endpoint 8081 vllm)
  2. curl -L http://$ENDPOINT/v1/chat/completions \
  3.     -H "Content-Type: application/json" \
  4.     -d '{
  5.         "model": "meta-llama/Meta-Llama-3-8B-Instruct",
  6.         "messages": [
  7.         {
  8.             "role": "system",
  9.             "content": "You are a helpful assistant."
  10.         },
  11.         {
  12.             "role": "user",
  13.             "content": "Who are you?"
  14.         }
  15.         ],
  16.         "stop_token_ids": [128009,  128001]
  17.     }'

To enable autoscaling, you could specify additional configs in services:

  1. services:
  2.     replica_policy:
  3.         min_replicas: 0
  4.         max_replicas: 3
  5.     target_qps_per_replica: 2

This will scale the service up to when the QPS exceeds 2 for each replica.

Optional: Connect a GUI to the endpoint

It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas.

Click to see the full GUI YAML

  1. envs:
  2.     MODEL_NAME: meta-llama/Meta-Llama-3-70B-Instruct
  3.     ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
  5. resources:
  6.     cpus: 2
  8. setup: |
  9.     conda activate vllm
  10.     if [ $? -ne 0 ]; then
  11.         conda create -n vllm python=3.10 -y
  12.         conda activate vllm
  13.     fi
  15.     # Install Gradio for web UI.
  16.     pip install gradio openai
  18. run: |
  19.     conda activate vllm
  20.     export PATH=$PATH:/sbin
  21.     WORKER_IP=$(hostname -I | cut -d' ' -f1)
  22.     CONTROLLER_PORT=21001
  23.     WORKER_PORT=21002
  25.     echo 'Starting gradio server...'
  26.     git clone https://github.com/vllm-project/vllm.git || true
  27.     python vllm/examples/gradio_openai_chatbot_webserver.py \
  28.         -m $MODEL_NAME \
  29.         --port 8811 \
  30.         --model-url http://$ENDPOINT/v1 \
  31.         --stop-token-ids 128009,128001 | tee ~/gradio.log
  1. Start the chat web UI:
  1. sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
  1. Then, we can access the GUI at the returned gradio link:
  1. | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live