After few weeks I have already plotted big enough sample of different k-sizes so last thing left was finding biggest files from each category. I have a single harvester that has all the drives pooled with mergerfs so the fastest way was to list all files sorting them by size with ls -lS --block-size=M /mnt/chia/farm and pick largest ones (rounding them a bit) manually. My google-fu failed me this time and not being able to find efficient and satysfying solution online I came up with this script:

from collections import Counter

#disk_space = 4658 # G
#plot_sizes = {'k34': 430, 'k33': 209, 'k32': 102}
disk_space = 4767650 # M
plot_sizes = {'k34': 440220, 'k33': 213900, 'k32': 103900}

res = {}

def get_combinations(total_size, history, last_size):
  for plot_size, file_size in plot_sizes.items():
    if file_size <= last_size:
       if total_size + file_size <= disk_space:
          get_combinations(total_size + file_size, history + [plot_size], file_size)
       else:
          free_space = disk_space - total_size
          if free_space < min(plot_sizes.values()):
            if str(free_space) not in res:
              res[str(free_space)] = []
            res[str(free_space)].append(sorted(Counter(history).items(),reverse = True))
            break

get_combinations(0, [], plot_sizes['k34'])

sortedKeys = list(res.keys())
sortedKeys.sort(key=int)
for key in sortedKeys:
    print(key, res[key])

And here is the output:

The outcome was that instead of losing 90GB from each 5TB disk, like it was with OG non-NFT plots, I’m leaving max few gigs free.

These instructions are for Microsoft SQL Server but same can be done easily with other DBMSes like MySQL or PostgreSQL. I’m assuming Elasticsearch is already installed somewhere. If it’s listening on localhost head to /etc/elasticsearch/elasticsearch.yml and add:

network.host: _site_
discovery.type: single-node

Restart elastic instance. Now lets install Logstash

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt install logstash

To be able to talk with SQL Server we will need to use appropriate JDBC driver. Most recent version available at time of writing is 9.2 and supports SQL Server versions starting from 2012 and newer. Logstash is currently packaged with Java 11.

mkdir -p /opt/sqljdbc_9.2/enu/
wget https://github.com/microsoft/mssql-jdbc/releases/download/v9.2.1/mssql-jdbc-9.2.1.jre11.jar -P /opt/sqljdbc_9.2/enu/

Logstash allows to select column that will be used to sync from SQL Server only inserted/updated records, in this case it’s “updated_at”. By default this value is stored in ~/.logstash_jdbc_last_run. Location of this file can be changed using last_run_metadata_path variable - I like to have all things related to particular service stored in one place so lets put it in /var/lib/logstash/jdbc_last_runs/ directory (should be taken from path.data)

mkdir /var/lib/logstash/jdbc_last_runs

Here’s sample config file:

input {
    jdbc {
        jdbc_connection_string => "jdbc:sqlserver://192.168.1.124\MSSQLSERVER;database=Test;user=sa;password=Passwd123"
        jdbc_user => nil
        jdbc_driver_library => "/opt/sqljdbc_9.2/enu/mssql-jdbc-9.2.1.jre11.jar"
        jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"

        statement => "SELECT * FROM dbo.sampletable WHERE updated_at >= :sql_last_value"
        # statement_filepath
        last_run_metadata_path => "/var/lib/logstash/jdbc_last_runs/sampletable"
        use_column_value => true
        tracking_column => "@timestamp"
        shedule => "0 * * * *"
    }
}

output {
    elasticsearch {
        hosts => ["192.168.1.121:9200"]
        index => "test"
    }
    stdout { codec => rubydebug }
}

/etc/logstash/conf.d/sampletable.conf

If there is no id column in our table we should add it to update rows that we have already fetched instead of inserting them again. Something like “SELECT [table id] as id, * FROM…” should do the job.  Now lets test our settings doing initial import:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sampletable.conf

If there were no errors we should have our data in Elasticsearch.

Useful links

https://docs.microsoft.com/en-us/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server https://docs.microsoft.com/en-us/sql/connect/jdbc/microsoft-jdbc-driver-for-sql-server-support-matrix

Recently OVH upgraded their Sandbox PI line and renamed it to Discovery:

Besides making price bigger they have provided us with option to get connections faster than 100mbps and now middle variant is perfect fit for my WAN.

After some testing I decided to use OpenVPN under Ubuntu on both sides - OVH d2-4 as VPN server and LXC container as VPN client. Installation is quite straightforward, so on both ends:

apt install openvpn bridge-utils

Since on this tunnel I don’t want more traffic than necessary, I will have only one client connecting to server. Simple static key auth will be enough:

openvpn --genkey --secret /etc/openvpn/static.key
cat > /etc/openvpn/server.conf << EOF
#verb 4
dev tap0
dev-type tap
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-128-CBC
auth SHA1
lport 1194
up server.sh
secret static.key
log /var/log/openvpn/server.log
status /var/log/openvpn/server.status
EOF

We need to bridge interface in vRack (private network) with TAP adapter. Easiest and in this case probably cleanest way is to let OpenVPN do it:

cat > /etc/openvpn/server.sh << EOF
#!/bin/bash

ip link add br0 type bridge
ip link set tap0 master br0
ip link set ens4 master br0
ip link set br0 up
ip link set tap0 up
EOF
chmod u+x /etc/openvpn/server.sh

Now we should configure firewalls (e.g. UFW and OVH firewall) to reflect our infrastructure, e.g. filtering with static list of IPs allowed to talk with port 1194. Finally we can start OpenVPN server:

systemctl enable openvpn@server --now
tail /var/log/openvpn/server.log

Now lets configure client, assuming it is using vpn.yourserver.com subdomain:

cat > /etc/openvpn/client.conf << EOF
dev tap0
persist-tun
persist-key
cipher AES-128-CBC
auth SHA1
script-security 3
up client.sh
remote vpn.yourserver.com 1194 udp
secret static.key
log /var/log/openvpn/client.log
status /var/log/openvpn/client.status
EOF

cat > /etc/openvpn/client.sh << EOF
#!/bin/bash

ip link add br0 type bridge
ip link set tap0 master br0
ip link set ens19 master br0
ip link set br0 up
ip link set tap0 up
ip link set ens19 up
EOF
chmod u+x /etc/openvpn/client.sh

systemctl enable openvpn@client --now
tail /var/log/openvpn/client.log

Similar config may be used for other scenarios, like for example connecting OVH vRack to dedicated server from different provider.

Probably some of our packets won’t be forwarded because of their size, we should fragment them then. Easiest way to find max packet size is to ping starting from size we know that should be passed with default MTU in our network, usually it’s 1500 - so this gives 1500 bytes (Ethernet MTU) - 20 byte (IP header) - 8 byte (ICMP header) = 1472 bytes max . Lets start with this number and lower it on every retry till we will receive answer. From Windows:

ping -f -n 1 -l 1472 host.behind.vpn 

Or from Linux:

ping -M do -c 1 -s 1472 host.behind.vpn 

In my case it was AFAIR 1350 so we have to both config files (this setting can’t be pushed), client and server:

fragment 1350

Other thing we might want is tunneling data between from all vlans, to do this add to both scripts started by OpenVPN (assuming we already have VLAN-aware bridge):

ip link set tap0 promisc on
bridge vlan add vid 2-4094 dev tap0

I’ll be using below configuration in my examples:

To add additional IP addresses to interface, we can create e.g. /etc/systemd/network/eth0.network.d/ips.conf file:

[Network]
Address=1.2.3.4/29
Address=5.6.7.8/27

Now just restart networkd and check if everything went as we’ve planned

systemctl restart systemd-networkd && systemctl status systemd-networkd && ip a

Another case where such functionality is helpful is when we want to add some static routes - lets say we have two interfaces: eth0 with public ip and isp gateway and eth1 for private/internal network with its own router for, inter alia, vpn access. Because traffic from all potential private networks will go only through this second interface lets configure them in /etc/systemd/network/eth1.network.d/static.conf

[Route]
Destination=10.0.0.0/8
GatewayOnLink=true
Gateway=10.7.17.1

[Route]
Destination=172.16.0.0/12
GatewayOnLink=true
Gateway=10.7.17.1

[Route]
Destination=192.168.0.0/16
GatewayOnLink=true
Gateway=10.7.17.1

Apply new settings, show potential errors and display current routes:

systemctl restart systemd-networkd && systemctl status systemd-networkd && ip r

Proxmox Virtual Environment  API is a core element that rest of management interfaces is built on. It provides us with an easy way to automate tasks like making and deleting snapshots at programmed intervals. Lets have a look at endpoints that we will need for managing snapshots of LXC container:

In a similar way we can list, create and delete KVM snapshots:

To test how it works we can use pvesh - shell interface for PVE API

Nice! With all this knowledge all what’s left is scripting requests to API. Since I’m trying to use Python for most backend stuff, after quick search I’ve found Proxmoxer that will take care of authentication and making requests. What’s left is adding some basic logic and after a while I came up with this short script making snapshots of all VMs and CTs in my cluster:

import proxmoxer, datetime, time

pve = proxmoxer.ProxmoxAPI('10.7.17.11', user='snapshoter@pve', password='passw0rd', verify_ssl=False)

for vm in pve.cluster.resources.get(type='vm'):
    if vm['type'] == 'lxc':
        cur_vm = pve.nodes(vm['node']).lxc(vm['vmid'])
    else vm['type'] == 'qemu':
        cur_vm = pve.nodes(vm['node']).qemu(vm['vmid'])

    cur_vm.snapshot.post(snapname='auto_'+datetime.datetime.now().strftime("%Y%m%d%H"))

Now just to add task to cron that will execute this script every four hours and wait. This is what I saw after two days:

Now I just need a way to prune snapshots older than two days. To keep things simple I have created another script that is executed once a day during the night:

import proxmoxer, datetime, time

pve = proxmoxer.ProxmoxAPI('10.7.17.11', user='snapshoter@pve', password='passw0rd', verify_ssl=False)

for vm in pve.cluster.resources.get(type='vm'):
    if vm['type'] == 'lxc':
        cur_vm = pve.nodes(vm['node']).lxc(vm['vmid'])
    else vm['type'] == 'qemu':
        cur_vm = pve.nodes(vm['node']).qemu(vm['vmid'])

    for snap in cur_vm.snapshot.get():
        if snap['name'] != 'current' and snap['name'][:5] == 'auto_' and snap['name'] < 'auto_'+(datetime.datetime.now()-datetime.timedelta(days=2)).strftime("%Y%m%d%H"):
            cur_vm.snapshot(snap['name']).delete()
            time.sleep(10)

ceph osd pool set  size 1
ceph osd pool set  min_size 1

However, recently I got Advance STOR-1 with single 500GB NVMe and four 4TB HDDs from OVH, mainly because I’ve decided to stop using multiple ARM-2T for OSDs in my Ceph cluster. One of the reasons was that  with bigger traffic it was too unstable and nonexistent support outweighed benefits coming from the low cost.

After installing Proxmox 6.1 via IPMI on main NVMe drive and adding four SATA drives as OSDs next step was changing failure domain to osd from default host. Because I wanted to use erasure coded pool to gain as much storage as I can simplest way was to start from creating pool with interesting me profile (in my case it was default K3M1) which caused inserting relevant rule to crush map. Of course such pool won’t be usable and Ceph will be reporting it as unhealthy because at the moment we need at least four OSDds on separate hosts to be able to use it. Having all rules that we’ll need we can move to adjusting crush map. First we have to dump and decompress it:

ceph osd getcrushmap -o crush_map_compressed
crushtool -d crush_map_compressed -o crush_map_decompressed

Next we will have to modify lines in our rules for replicated_rule and erasure-code starting with  “step chooseleaf”

sed -r 's/step chooseleaf (firstn|indep) 0 type host/step chooseleaf \1 0 type osd/g' crush_map_decompressed > new_crush_map_decompressed

After this change we can compress and import our new crush map

crushtool -c new_crush_map_decompressed -o new_crush_map_compressed
ceph osd setcrushmap -i new_crush_map_compressed

This way we’ve forced Ceph to work as a simple local software RAID and now cluster shouldn’t report errors.

Right image: camera zooming on caliper (next to big three below 10” screen)

Results were pretty satisfying so I have ordered 1/4-20 UNC threaded inserts that I wanted like to add to design before releasing it to wider public and since it will be delivered in about month or so, I’ve put it back on shelf till then.

It is wise to run service under dedicated and unprivileged user so lets create one

useradd autossh -g nogroup -s /bin/false -m /etc/autossh autossh

We will use SSH key to connect to remote machine

sudo -u autossh ssh-keygen -t ed25519 -C autossh

Next let’s define host that we will be forwarding ports to (or from)

Host    homelab-vps
        HostName        10.10.10.10
        User            root

/etc/autossh/.ssh/config

Now lets test connection. Additionally our machine will be added to known_hosts

sudo -u autossh ssh homelab-vps

Before closing connection we have to modify /etc/ssh/sshd_config on machine we will be forwarding ports to by adding “GatewayPorts yes”.

Systemd unit will allow us to start tunnel automatically with operating system

cat > /etc/systemd/system/autossh@.service << EOF
[Unit]
Description=Keeps an ssh tunnel to %I open
After=network-online.target ssh.service

[Service]
User=autossh
# no monitoring
Environment="AUTOSSH_PORT=0"
# Disable gatetime behaviour
Environment="AUTOSSH_GATETIME=0"
# Enable logging
Environment="AUTOSSH_DEBUG=1"
Environment="AUTOSSH_LOGFILE=/var/log/autossh/%i.log"
EnvironmentFile=/etc/autossh/%i.conf
RestartSec=3
Restart=always

# -NT Just open the connection and do nothing (not interactive, no tty alloc)
# use /usr/bin/ssh instead of autossh is good as well
ExecStart=/usr/bin/autossh -NT -o "ExitOnForwardFailure=yes" $SSH_OPTIONS ${TARGET_HOST} $FORWARDS
TimeoutStopSec=10

[Install]
WantedBy=multi-user.target
EOF

For each connection we should create config file

cat > /etc/autossh/homelab-vps.conf << EOF
TARGET_HOST=homelab-vps
FORWARDS=-R 80:localhost:80 -R 443:localhost:443
SSH_OPTIONS=-o "ServerAliveInterval=10" -o "ServerAliveCountMax=3"
AUTOSSH_PORT=0
AUTOSSH_GATETIME=0
EOF

Finally, to activate tunnel:

systemctl enable autossh@homelab-vps
systemctl start autossh@homelab-vps

We can of course forward ports in different direction - from remote to local machine.

This time we have to modify ssh server config on machine that will be used to forward ports. One thing to note: privileged ports (1-1023) can only be forwarded by root, in this scenario we will use dynamic port

cat >> /etc/ssh/sshd_config << EOF
Match User autossh
    PasswordAuthentication no
    GatewayPorts yes
EOF

cat >> /etc/autossh/.ssh/config << EOF
Host    mariadb
        HostName        10.1.1.50
        User            root
EOF

Config in such scenario may look like this (in case of mysql or mariadb there’s a reason not to use localhost - all connection will be seen as coming from it by dbms)

cat >> /etc/autossh/mariadb.config << EOF
TARGET_HOST=mariadb
FORWARDS=-L 0.0.0.0:3306:10.1.1.50:3306
SSH_OPTIONS=-o "ServerAliveInterval=10" -o "ServerAliveCountMax=3"
AUTOSSH_PORT=0
AUTOSSH_GATETIME=0
EOF

Linux

With Linux distros (it works fine with other systems that are hosting ssh server, including Windows) I usually use paramiko - it’s quite easy to use and supports all what one might need (including key-based auth, which inter alia gives more granular control which makes it far more reasonable solution than using passwords). Here’s simple script to check how much of space we’re using on /tmp

import paramiko, re

hostname = '192.168.1.110'
username = 'piotr'
password = 'piotr'

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=hostname, username=username, password=password)
stdin, stdout, stderr = ssh.exec_command('df /tmp')
print(re.findall(r'\s(\d+)%', stdout.read().decode('ascii'))[0])
ssh.close()

Windows

For Windows-based machines I prefer pypsexec - it doesn’t require too much setup on remote machine. There’s great manual at Python Package Index so I don’t think there’s a point in replicating it’s contents here. Lets check how much of storage is used on drive C:

import pypsexec.client

hostname = '192.168.1.80'
username = 'Piotr'
password = 'Piotr'

c = pypsexec.client.Client(hostname, username=username, password=password, encrypt=False)
c.connect()
try:
	c.create_service()
	stdout = c.run_executable("powershell.exe", arguments="-", stdin=
"""$disk = Get-WmiObject Win32_LogicalDisk -Filter "DeviceID='C:'"
[math]::round(($disk.Size-$disk.FreeSpace)/$disk.Size*100)
exit 0
""".encode('utf-8'), timeout_seconds=30)
	print(stdout[0].decode('utf-8').strip())
finally:
	c.remove_service()
	c.disconnect()