vLLM Invocation

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve --model /llm/models/Qwen/Qwen3.8-27B-FP8 --dtype=float16 --enforce-eager --port 8000 --host 0.0.0.0 --trust-remote-code --disable-sliding-window --gpu-memory-util=0.90 --max-num-batched-tokens=8192 --max-model-len=192608 --block-size 64 -tp=2 --speculative-config '{"method":"mtp","num_speculative_tokens":2}' --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --max-num-seqs=4 --kv-cache-dtype fp8 

Popular posts from this blog

Create a new repo

Create a new repository Generate a local git repo in the project working directory: git init . SSH to remote server create a bare repo using the project name: ssh tim@fileserv git init --bare /files/git/xxxxxxx exit Add the files to be tracked and commit: git add blah.c git commit -a -m "Initial commit." Add the remote repo and push: git remote add origin tim@fileserv:/files/git/xxxxxxx git push --set-upstream origin master
Read more

Change Fedora Silverblue automatic updates to check

Set the AutomaticUpdatePolicy to check in /etc/rpm-ostreed.conf: [Daemon] AutomaticUpdatePolicy=check Then reload the service: sudo rpm-ostree reload Enable the timer if not already active: sudo systemctl enable rpm-ostreed-automatic.timer --now This configures the system to check for updates automatically and display them in rpm-ostree status, but does not download or apply them.  You’ll see notifications about available updates, including CVEs, without any automatic downloading. 
Read more

Add VMs to Fedora Atomic

sudo rpm-ostree install libvirt libvirt-daemon-config-network libvirt-daemon-kvm qemu-kvm virt-manager virt-viewer systemctl reboot sudo systemctl enable --now libvirtd sudo grep -E '^libvirt:' /usr/lib/group | sudo tee -a /etc/group sudo usermod -a -G libvirt $USER sudo grep -E '^qemu:' /usr/lib/group | sudo tee -a /etc/group sudo usermod -aG qemu tim
Read more