Deploying with dstack

vLLM_plus_dstack

vLLM can be run on a cloud based GPU machine with dstack, an open-source framework for running LLMs on any cloud. This tutorial assumes that you have already configured credentials, gateway, and GPU quotas on your cloud environment.

To install dstack client, run:

  1. $ pip install "dstack[all]
  2. $ dstack server

Next, to configure your dstack project, run:

  1. $ mkdir -p vllm-dstack
  2. $ cd vllm-dstack
  3. $ dstack init

Next, to provision a VM instance with LLM of your choice(NousResearch/Llama-2-7b-chat-hf for this example), create the following serve.dstack.yml file for the dstack Service:

  1. type: service
  3. python: "3.11"
  4. env:
  5.     - MODEL=NousResearch/Llama-2-7b-chat-hf
  6. port: 8000
  7. resources:
  8.     gpu: 24GB
  9. commands:
  10.     - pip install vllm
  11.     - vllm serve $MODEL --port 8000
  12. model:
  13.     format: openai
  14.     type: chat
  15.     name: NousResearch/Llama-2-7b-chat-hf

Then, run the following CLI for provisioning:

  1. $ dstack run . -f serve.dstack.yml
  3. ⠸ Getting run plan...
  4.  Configuration  serve.dstack.yml
  5.  Project        deep-diver-main
  6.  User           deep-diver
  7.  Min resources  2..xCPU, 8GB.., 1xGPU (24GB)
  8.  Max price      -
  9.  Max duration   -
  10.  Spot policy    auto
  11.  Retry policy   no
  13.  #  BACKEND  REGION       INSTANCE       RESOURCES                               SPOT  PRICE
  14.  1  gcp   us-central1  g2-standard-4  4xCPU, 16GB, 1xL4 (24GB), 100GB (disk)  yes   $0.223804
  15.  2  gcp   us-east1     g2-standard-4  4xCPU, 16GB, 1xL4 (24GB), 100GB (disk)  yes   $0.223804
  16.  3  gcp   us-west1     g2-standard-4  4xCPU, 16GB, 1xL4 (24GB), 100GB (disk)  yes   $0.223804
  17.     ...
  18.  Shown 3 of 193 offers, $5.876 max
  20. Continue? [y/n]: y
  21. ⠙ Submitting run...
  22. ⠏ Launching spicy-treefrog-1 (pulling)
  23. spicy-treefrog-1 provisioning completed (running)
  24. Service is published at ...

After the provisioning, you can interact with the model by using the OpenAI SDK:

  1. from openai import OpenAI
  3. client = OpenAI(
  4.     base_url="https://gateway.<gateway domain>",
  5.     api_key="<YOUR-DSTACK-SERVER-ACCESS-TOKEN>"
  6. )
  8. completion = client.chat.completions.create(
  9.     model="NousResearch/Llama-2-7b-chat-hf",
  10.     messages=[
  11.         {
  12.             "role": "user",
  13.             "content": "Compose a poem that explains the concept of recursion in programming.",
  14.         }
  15.     ]
  16. )
  18. print(completion.choices[0].message.content)

Note

dstack automatically handles authentication on the gateway using dstack’s tokens. Meanwhile, if you don’t want to configure a gateway, you can provision dstack Task instead of Service. The Task is for development purpose only. If you want to know more about hands-on materials how to serve vLLM using dstack, check out this repository