Servers

Over the summer I got a R1500 G3 ups for my server it is quite over kill however i hope to connect my POE switch and starlink internet to also keep wifi up during a power outage. To that end it would be rather nice if my server actually shut down as the power was running out rather than just continuing until it dies when the battery runs out. The last piece of this puzzle was a usb a to usb b cable which I recently got. This connection goes from my ups to my r720 on which i decided to install nut. The server is running proxmox as its primary OS so I was able to install nut with apt however the first few guides I followed ended up not working for what ever reason. The one that finally worked was this one. Once I had that sorted I setup a web nut docker container in terraform which all went smoothly I hope to eventually add the stats to graphana for data visulisation but for now i am happy.

One of my big goals with my homelab is resiliency as in that i could lose everything and have the ability to rebuild quickly on a new piece of hardware. To this end i have previously had a gitea repo which contained: a docker compose; a deploy target txt; and a env.sh loader script. My setup currently looks like a main.ft with indivial files pre container. I am also using a gitea action to run terraform and commit the new tfstate file to the repo. I think this will be able to scale up with me for now once i finish my migration.

I had for a while been noticing large spikes in the disk backlog according to netdata as wells as swap useage errors. The swap useage errors especially were wierd because the VM can ballon up to 200GB of RAM and i know i am not using that much. These errors combined were leading to system freezes where i had to go into proxmox and stop the VM and then start it. So to start investigating i ran ...

So in my process of preparing to switch to starlink I setup headscale to replace my wireguard vpn. I thought that I would have to use tailscale as I presumed the headscale server would require a port. It does but it just so happens to use https which I can tunnel with cloudflare. So with that in mind I setup headscale in a docker container and connected my phone with no issue. The server side was slightly harder as a i used the tailscale docker container and accidentally removed the envs so i spent a while trying to figure out why the state file wasn’t saving properly. That is fixed now and it works fine. However on further investigation I realised that I could not use it for my main purpose as cloudflare tunnels do not support web socket PUSH.

I am not mad I will be keeping ipv4 for my LAN however for reasons beyond my control I am moving from a wired internet connection to starlink that uses carrier grade nat. This leaves me with a range of problems I need to solve to get my homelab working without access to public ports. The first solution I am going to use is cloudflare tunnels. So i go from… to… This is not terrible and was easy to setup and integrate in my CI/CD pipeline however it is not a 1 size fits all solution as I still have game severs, mail, VPN, git ssh, remote access and sftp. I have some ideas such as headscale or similar for VPN. Git ssh i can tunnel but it would not be public so i feel it is pointless. The others I will have to try ipv6 when i change to starlink.

The setup was simple I deployed the panel in a docker container to a fresh docker host. I had to set the application url env to the public url not the private one. Then I installed the wings software on the same VM as the panel. I had to manually copy the config file from the panel. Everything else it automated with gitea actions. I made sure to change some of the files in the config files to my nfs share. Unfortuantly I then ran into a massive bug where I couldn’t start any server. I hope this is fixed soon.

I recently decided that I wanted to explore the entire hashicorp stack. I have been using vault and terraform since I moved to gitea for container management. However I decided that learning the entire stack would be interesting and teach me more about IaC and cloud services. I chose to do a install of all the software in a VM running arch that has 200GB of ballooning ram and 20 CPU cores at 3 GHz. It will run the containers in docker on the same server. I managed to install nomad and consul quickly however vault took a minute as the config file is just in /etc not /etc/vault.d. I then also had to add tls_disable = true to the config files which I am currently storing in gitea and deploying with gitea actions. I initialized vault with 5 keys and 3 minimum. I decided not to use my existing vault as I want to start clean and fresh. I installed vagrant and boundary-bin from the AUR and had to curl the binary for waypoint which is currently had coded to version 11.4 but i hope to make dynamic at some point. I then setup waypoint with waypoint install -accept-tos -platform nomad -nomad-host-volume=waypoint-data -nomad-runner-host-volume=waypoint-data-client. However I needed to set export NOMAD_ADDR=http://192.168.1.13:4646 first.

I have been using nginx reverse proxy for quite a while and it works great. However with my recent switch to automation and git managment i have decieded to switch to caddy i know many people love traefik however i have tried it in the past and had some errors so decided to investigate caddy. My current setup works as such in each projects git repo I store the caddy file their is then an action which will copy the caddy file to the correct folder before running caddys hot reload command caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile.

Since I started storing all my docker compose files in git i have wanted to create a public mirror as a disaster recovery option. To that end i needed a way to remove sensitive envs from the files. To achive this I set the docker env to a shell env like so TEMP: ${TEMP} This shell env was set by a script which contained export TEMP=$(vault kv get -field=temp kv/data/docker). This vault currently runs as a single node in docker and has to be manually unsealed as needed.

It started when i tired to set up my emails on Thunderbird mail on arch I noticed it wouldn’t connect. I checked the logs and found “2025-01-16 21:04:11,919 WARN exited: amavis (exit status 1; not expected)” I fixed this relatively quickly by removing “amavisd.pid”. After this it booted but I still couldn’t connect. I tracked this down by using dig and whats my ip to see that the dns record was wrong. I fixed this by switching to a cname record set to @ so i only need 1 ddns setup. This finally fixed it.