RUEN
To articles
Free Setup

Linux commands for a GPU host: a searchable cheat sheet

Vast Sale editorial, 12 min

A Vast.ai host does not need to be a Linux administrator, but a dozen commands come up every week: check the cards, find out why the disk is full, see whether the daemon is alive, who took a port. This page collects the ones that matter: only what is actually useful on a GPU rental server, with a one- or two-line explanation each. Type a word into the search — “disk”, “temperature”, “port”, “vast” — and only the matching rows stay.

The commands are verified on Ubuntu 22.04 Server, which we use for hosts. Where administrator rights are needed, sudo is included. Commands marked “careful” can interrupt a rental or delete data: read the explanation before pressing Enter.
142 commands
Try:

Finding your way: files and directories

pwd
Shows which directory you are in.
ls -lah
Lists files with permissions, owner, size and hidden files.
cd /var/log
Changes directory. cd - goes back to the previous one, cd alone goes home.
cat file
Prints the whole file. For long files use less file (q to quit, / to search).
tail -n 50 file
Last 50 lines of a file. tail -f file follows it live, handy for logs.
grep -n "error" file
Finds lines containing text. -i ignores case, -r searches a whole directory.
find / -name "*.log" -size +100M
Finds files by name and size — for example, logs that grew too large.
cp -r src dst
Copies a file or a directory (-r includes contents).
mv old new
Moves or renames a file or a directory.
rm -rf folder
careful Deletes a directory and everything in it, no questions asked. Unrecoverable; a typo in the path is expensive.
mkdir -p a/b/c
Creates a directory together with its parents.
nano file
A simple editor: Ctrl+O saves, Ctrl+X exits.
chmod +x script.sh
Makes a file executable.
chown user:user file
Changes the owner of a file or a directory (-R recursively).
ln -s /data/models ~/models
A symbolic link: a directory or file becomes reachable by a second path, for example from a large drive.
history | grep nvidia
Finds a command you typed before. Ctrl+R searches history interactively.
sudo !!
Repeats the last command with sudo when you forgot to add it.

Server health: load, memory, processes

htop
A live view of processes, cores and memory. Install with sudo apt install htop; q to quit.
top
The same, available on any system. %CPU is the process load, RES its memory.
uptime
How long the server has been up and the load average over 1, 5 and 15 minutes. Numbers above the core count mean processes are queuing.
free -h
Memory and swap in readable units. Look at the available column, not free.
nproc
Number of CPU cores available to the system.
lscpu
CPU model, cores, threads, clocks, virtualization.
ps aux | grep python
Finds a process by name. First column is the user, second is the PID.
kill PID
Terminates a process gracefully. kill -9 PID forces it when it does not respond.
pkill -f name
Kills every process whose command line contains the word.
sudo dmesg -T | tail -n 50
Kernel messages with timestamps: GPU, disk, memory and network errors show up here.
sudo dmesg -T | grep -i -E "xid|nvrm"
Only NVIDIA driver errors. The first thing to check when a card drops or the self-test fails.
sensors
CPU and motherboard temperatures. Install with sudo apt install lm-sensors, then run sudo sensors-detect.
hostnamectl
Hostname, Ubuntu release and kernel version in one command.
uname -r
Kernel version. Needed to check which kernel the driver was built for after an update.
timedatectl
Time, timezone and sync status. A clock that drifts breaks TLS connections and confuses logs.

GPUs: NVIDIA

nvidia-smi
The host's main command: cards, driver version, temperature, power draw, utilization, memory in use and processes on each GPU.
watch -n 2 nvidia-smi
The same table refreshed every 2 seconds. Ctrl+C to quit.
nvidia-smi -L
Lists GPUs with model and UUID. A quick check that the system sees every card.
nvidia-smi --query-gpu=index,name,temperature.gpu,power.draw,utilization.gpu,memory.used --format=csv
Only the columns you need, as CSV — convenient for logging or sending to a Telegram bot.
nvidia-smi -q -d TEMPERATURE,POWER,CLOCK
A detailed report: throttle thresholds, power limits, current clocks.
nvidia-smi -q -d PERFORMANCE
Why a card drops its clocks: overheating, power cap, idle.
nvidia-smi -q -d ECC
ECC memory error counters on cards that have ECC. Non-zero values are a reason to pull the card from rental.
sudo nvidia-smi -pl 300
Caps the card's power draw in watts. Lowers heat and the electricity bill at the cost of some performance.
sudo nvidia-smi -pm 1
Enables persistence mode: the driver stays loaded and the cards respond faster.
sudo nvidia-smi -r -i 0
careful Resets GPU 0 without rebooting the server. Only when it has no processes and no rentals.
nvidia-smi topo -m
How the cards connect to each other and to the CPU: PCIe, NVLink. Matters for multi-GPU rentals.
lspci | grep -i nvidia
GPUs on the PCIe bus regardless of the driver. If a card is missing here, the problem is hardware or the riser.
sudo lspci -vv -s 01:00.0 | grep -i lnksta
PCIe link speed and width for a card (for example, 16GT/s, x16). Take the address from lspci.
dpkg -l | grep nvidia-driver
Which driver package is installed and its version.
sudo apt-mark hold nvidia-driver-570
Freezes the driver package (use your version). An automatic driver update under a running Vast daemon breaks rentals.
nvidia-smi --query-gpu=index,fan.speed --format=csv
Fan speed in percent for each card.

The Vast.ai host

sudo systemctl status vastai
Is the Vast daemon alive: active (running) and the last lines of its log.
sudo systemctl restart vastai
Restarts the daemon when it hangs or stops reporting to the panel. Running rentals are not stopped.
sudo journalctl -u vastai -f
The daemon log, live.
sudo tail -f /var/lib/vastai_kaalia/kaalia.log
The detailed agent log: checks, self-test, panel connectivity, container start errors.
cat /var/lib/vastai_kaalia/host_port_range
The port range the daemon assigns to rentals. The same range must be open in the firewall and on the router.
cat /var/lib/vastai_kaalia/host_ipaddr
The IP address the agent treats as the machine's public address.
docker ps
Client rentals currently running on the machine: the Vast daemon runs them as containers.
vastai show machines
Your machines via the CLI: id, status, reliability, prices. Requires pip install vastai and vastai set api-key <key>.
vastai self-test machine <id>
Self-test before listing: GPUs, network, Docker. Run it with no active clients; takes about 6 minutes.
vastai list machine <id> --price_gpu 0.25 --price_disk 0.003 --price_inetu 0.004 --price_inetd 0.004
Lists the machine for rent: price per GPU-hour, per GB of disk per month and per GB of traffic (the CLI uses gigabytes, the panel uses terabytes). The panel is simpler; the CLI suits scripts.
vastai unlist machine <id>
Takes the machine off the market: no new orders, current ones finish. Do this before maintenance.
vastai set min-bid <id> --price 0.10
The minimum price for interruptible rentals.
vastai show user
Balance, accumulated earnings and account details.
vastai --help
The full list of CLI commands; each has its own --help.

Docker

docker ps -a
All containers, including stopped ones. Without -a — only running.
docker stats --no-stream
How much CPU, memory and network each container is using right now.
docker logs --tail 100 <container>
The last 100 lines of a container's output. -f follows it live.
docker images
Downloaded images and their sizes.
docker system df
Disk space used by images, containers and volumes. The first check when the disk is full.
docker system prune
careful Removes stopped containers, unused networks and dangling images. On a Vast host, only with no active rentals: the daemon keeps images for fast starts.
sudo systemctl status docker
Is the Docker service running. Without it the Vast daemon cannot start a single rental.
docker info | grep -i -E "storage driver|root dir"
Where Docker keeps its data and which storage driver is in use.
docker exec -it <container> bash
Opens a shell inside a container. Only for your own containers — never enter clients' rentals.

Disks and space

df -h
Free space per partition. A full root or Docker partition is the most common cause of failed rentals.
sudo du -xh / --max-depth=2 2>/dev/null | sort -h | tail -n 20
The twenty largest directories on the system — quickly see what took the space.
du -sh /var/lib/docker /var/log
Size of specific directories; on a host these two grow fastest.
lsblk -f
Disks, partitions, filesystems and mount points.
sudo fdisk -l
Disks and partitions with sizes. Useful when a new SSD does not show up.
sudo nvme list
NVMe drive models and capacities (sudo apt install nvme-cli).
sudo smartctl -a /dev/nvme0n1
Drive health: wear percentage, errors, temperature (sudo apt install smartmontools).
sudo fstrim -av
Tells SSDs which blocks are free. Usually runs on a weekly timer.
mount | grep ^/dev
What is mounted where and with which options.
sudo dmesg -T | grep -i -E "i/o error|nvme"
Disk errors at the kernel level. If they appear, plan a replacement.

Network and bandwidth

ip a
IP addresses and interface state (state UP).
ip r
The routing table: which gateway the server uses to reach the internet.
curl -4 ifconfig.me
The public IPv4 address as seen by the internet and the Vast panel.
ping -c 4 8.8.8.8
Is there connectivity and what is the latency. ping -c 4 vast.ai also checks DNS.
ss -tulpen
What is listening on which port: program, PID and address. Replaces the old netstat.
sudo ss -tnp | grep :22
Who is connected to the server over SSH right now.
nc -zv host 22
Checks whether a port is open on another machine — for example, your Vast port from outside, from a VPS.
mtr -rw 8.8.8.8
A traceroute with loss per hop: shows where the provider has a problem (sudo apt install mtr).
speedtest-cli --simple
Bandwidth to the nearest test server (sudo apt install speedtest-cli). For Vast, upload matters more.
iperf3 -c <ip_vps>
Real upload throughput to your VPS in Europe or Asia; -R measures the other direction. The VPS must run iperf3 -s.
wget -O /dev/null http://speedtest.tele2.net/100MB.zip
A quick download speed check without installing anything.
sudo ethtool enp5s0 | grep -i speed
The negotiated link speed: 1000Mb/s or 100Mb/s. A common surprise with a bad cable. Take the interface name from ip a.
dig +short vast.ai
Checks that DNS resolves.
sudo ufw status numbered
Firewall rules with numbers (remove one with sudo ufw delete <number>).
sudo ufw allow 40000:40100/tcp
Opens the port range for Vast rentals. Use your own range from host_port_range.

Services and logs

sudo systemctl status <service>
Service status and the last lines of its log.
sudo systemctl restart <service>
Restarts a service. start and stop start and stop it.
sudo systemctl enable --now <service>
Enables start at boot and starts the service right now.
systemctl list-units --type=service --state=running
What is running right now.
systemctl --failed
Services that failed at boot or later.
sudo journalctl -u <service> -f
A service log, live.
sudo journalctl -u <service> --since "1 hour ago"
The log for the last hour. Also understands today, yesterday, and a date with time.
sudo journalctl -b -1 -e
The previous boot's log from the end: what happened before a reboot or a hang.
sudo journalctl -p err -b
Only errors since boot.
sudo journalctl --vacuum-size=500M
Shrinks the system journal to 500 MB when it fills the disk.
last -n 20
Who logged in and when, plus reboots.
who
Who is logged in right now.
sudo tail -f /var/log/auth.log
Login attempts, including SSH password brute force.
sudo fail2ban-client status sshd
How many addresses are banned for password guessing.

Packages and updates

sudo apt update
Refreshes the package lists. Installs nothing.
apt list --upgradable
What can be upgraded. If nvidia-driver is in the list, run apt-mark hold first.
sudo apt upgrade
careful Installs upgrades. On a host with rentals, update the driver and kernel only in a maintenance window with the machine unlisted.
sudo apt install <package>
Installs a package. sudo apt remove <package> removes it.
apt search <word>
Finds a package by name or description.
dpkg -l | grep <word>
Is a package installed and which version.
sudo apt-mark hold <package>
Freezes a package so upgrades leave it alone. unhold unfreezes it. Mandatory for the NVIDIA driver.
sudo apt autoremove
Removes unneeded dependencies and old kernels that take space in /boot.
cat /etc/apt/apt.conf.d/20auto-upgrades
Are automatic updates enabled (Unattended-Upgrade "1"). On a host, disable them or keep the driver on hold.
sudo reboot
careful Reboots. Every rental on the machine is interrupted: unlist it first and wait for the orders to finish.
sudo shutdown -h now
careful Shuts the server down. sudo shutdown -r +10 reboots in 10 minutes.

Access: SSH, files, sessions

ssh user@host -p 2222
Connects to the server; -p is needed when the port is not 22.
ssh-keygen -t ed25519
Creates a key pair on your own computer.
ssh-copy-id user@host
Installs your public key on the server so you can log in without a password.
scp file user@host:/path/
Copies a file to the server. Swap the arguments to download.
rsync -avP src/ user@host:/dst/
Syncs a directory with resume and progress. Better than scp for large transfers.
tmux
A session that survives an SSH disconnect: start, disconnect, come back with tmux attach. Detach with Ctrl+B, then D.
screen -S name
Like tmux, but simpler. Return with screen -r name, detach with Ctrl+A, then D.
nohup command &
Runs a command in the background so it survives logout. Output goes to nohup.out.
sudo -i
Become root until you type exit.
passwd
Changes your password.
exit
Ends the session (or Ctrl+D).

Combinations that save time

Ctrl+C
Interrupts the current command.
Ctrl+R
Searches command history: start typing and the last match appears.
Tab
Autocompletes paths and commands. Press twice to list every option.
command | less
Pages through long output: q to quit, / to search.
command | grep word
Keeps only the output lines that contain the word.
command > out.txt 2>&1
Saves all output, errors included, to a file.
command1 && command2
The second command runs only if the first succeeded.
watch -n 5 'df -h; nvidia-smi -L'
Repeats a command every 5 seconds and shows the result on one screen.
alias gpu='nvidia-smi --query-gpu=index,temperature.gpu,power.draw,utilization.gpu --format=csv'
A short name for a long command. Add the line to ~/.bashrc to keep it.
man command
The manual for a command; shorter — command --help.

Seven commands to memorise first

  • nvidia-smi — how the cards are doing.
  • df -h — how much space is left.
  • htop — what is loading the CPU and memory.
  • sudo systemctl status vastai — is the Vast daemon alive.
  • sudo journalctl -u vastai -f — what it is writing right now.
  • ss -tulpen — what is listening on which port.
  • docker ps — which rentals are running.

Where to go next

If the server is not built yet, start with the free Ubuntu 22.04 Server installation guide: https://vast.sale/articles/a16?lang=en. Reaching a server without a public IP is covered here: https://vast.sale/articles/a17?lang=en. The full host setup from hardware to the first order and machine hardening are in the paid guides: https://vast.sale/articles/a6?lang=en and https://vast.sale/articles/a8?lang=en. Prefer not to do it yourself — we connect your hardware to Vast.ai turnkey: https://vast.sale/services?lang=en.

Need help with the move?

We will set it up turnkey with a launch guarantee.

View services
© Vast Sale · Source: https://vast.sale/articles/a19
Copying, paraphrasing and AI processing are permitted only with an active link to https://vast.sale