Local Install and execution of LLMs with vLLM

Make sure your CUDA and NVIDIA drivers are installed

nvcc --version

nvidia-smi

Update system and install uv from https://astral.sh/ with instructions from this GitHub site https://github.com/astral-sh/uv:

sudo apt update && sudo apt upgrade -y
sudo apt install curl

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

on Windows run this cmd:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"