Notes to Self

Alex Sokolsky's Notes on Computers and Programming

Proxmox Set-up

Update repositories

You will get an error during the update:

Err:4 https://enterprise.proxmox.com/debian/pve buster InRelease
  401  Unauthorized [IP: 144.217.225.162 443]

It comes from /etc/apt/sources.list.d/pve-enterprise.list:

root@pve02:/etc/apt/sources.list.d# cat pve-enterprise.list
deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

You can disable it in the GUI and add there ‘No Subscription’ option. Alternatively, change it to (mind version specific buser below):

root@pve02:/etc/apt/sources.list.d# cat pve-enterprise.list
# deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

And then:

root@pve02:/etc/apt# cat sources.list
deb http://ftp.us.debian.org/debian buster main contrib

deb http://ftp.us.debian.org/debian buster-updates main contrib

# security updates
deb http://security.debian.org buster/updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

Install useful tools

apt install ifupdown2 ethtool silversearcher-ag emacs-nox htop lm-sensors \
  nvme-cli linux-cpupower cpufrequtils sysstat

Set the CPU governor

cat << 'EOF' > /etc/default/cpufrequtils
GOVERNOR="powersave"
EOF

Configure lm-sensors

sensors-detect

And finally:

watch -n 1 sensors

or just

root@pve02:/etc/apt/sources.list.d# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +36.0°C  (high = +94.0°C, crit = +100.0°C)
Core 0:        +35.0°C  (high = +94.0°C, crit = +100.0°C)
Core 1:        +35.0°C  (high = +94.0°C, crit = +100.0°C)
Core 2:        +36.0°C  (high = +94.0°C, crit = +100.0°C)
Core 3:        +34.0°C  (high = +94.0°C, crit = +100.0°C)
Core 4:        +32.0°C  (high = +94.0°C, crit = +100.0°C)
Core 5:        +32.0°C  (high = +94.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1:        +27.8°C  (crit = +119.0°C)

pch_cannonlake-virtual-0
Adapter: Virtual device
temp1:        +44.0°C

PCI Pass Through Config

See pcie-passthrough.

Add the server to Datacenter / Cluster

Make sure the host has:

Data Center Storage

Add connection to the NFS server: Datacenter\Storage Add NFS

Schedule Backups

Datacenter\Backup, Add

Cron

/etc/cron.daily

smart

#!/bin/sh
/usr/sbin/smartctl -t long /dev/sda >/dev/null
/usr/sbin/smartctl -t long /dev/sdb >/dev/null

/etc/cron.weekly

fstrim

#!/bin/sh
# trim all the mounted file systems which support it
/usr/sbin/fstrim --all || true

systemps script

Requires:

apt install nvme-cli
#!/bin/sh
# relies on sensors, nvme, hddtemp

# cleanup
rm -f /tmp/temps_????????_??????

now=$(date +"%Y%m%d_%H%M%S")
TMPFILE="/tmp/temps_$now"
(
    #echo "To: your email address"
    subject="`hostname` temperatures `date +'%Y-%m-%d %H:%M:%S'`"
    #echo "Subject: $subject"
    #echo " "
    echo $subject
    echo "$(uname -nsrm), $(uptime -p)"
    cat /proc/loadavg | awk '{ print "System load:",$1,$2,$3 }'

    # this does not work if CPU is virtualized
    temp=$(sensors|grep Package|awk '{print $4}')
    freqs=$(grep "cpu MHz" /proc/cpuinfo|awk -F'[\.| ]' '{print $3}'|paste -sd ' ')
    echo "CPU: $temp, $freqs MHz"

    temp=$(nvme smart-log /dev/nvme0|grep temperature|awk '{print $3}')
    echo "NVME: $temp°C"

    echo 'HDDs:'
    HDDS="/dev/sda /dev/sdb /dev/sdc /dev/sdc /dev/sdd /dev/sde"
    for disk in $HDDS
    do
        if [ -b $disk ]; then
            hddtemp ${disk}
        fi
    done

) > $TMPFILE

#sendmail -t < $TMPFILE
cat $TMPFILE

Check for IO bottleneck:

Use iostat:

root@fuji:~# iostat -mxy 10
Linux 5.15.83-1-pve (fuji) 	01/31/2023 	_x86_64_	(8 CPU)


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.00    0.00    0.48    0.05    0.00   98.47

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
dm-0             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-1             0.00      0.00     0.00   0.00    0.00     0.00   96.40      0.38     0.00   0.00    0.32     4.03    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.03   0.36
dm-10            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-11            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-12            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-13            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-14            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-15            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-16            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-17            0.00      0.00     0.00   0.00    0.00     0.00   12.00      0.05     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.04
dm-18            0.00      0.00     0.00   0.00    0.00     0.00    0.20      0.00     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.08
dm-19            0.00      0.00     0.00   0.00    0.00     0.00    0.20      0.00     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.08
dm-2             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-3             0.00      0.00     0.00   0.00    0.00     0.00   15.30      0.06     0.00   0.00    0.05     4.03    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.44
dm-4             0.00      0.00     0.00   0.00    0.00     0.00   15.30      0.06     0.00   0.00    0.05     4.03    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.44
dm-6             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-7             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-8             0.00      0.00     0.00   0.00    0.00     0.00    2.90      0.01     0.00   0.00    0.28     4.14    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.28
dm-9             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme0n1          0.60      0.07     0.00   0.00    0.83   128.00   46.50      0.44    65.20  58.37    1.27     9.68    0.00      0.00     0.00   0.00    0.00     0.00    0.30    0.67    0.06   0.92
sda              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00