Server

One of my big goals with my homelab is resiliency as in that i could lose everything and have the ability to rebuild quickly on a new piece of hardware. To this end i have previously had a gitea repo which contained: a docker compose; a deploy target txt; and a env.sh loader script. My setup currently looks like a main.ft with indivial files pre container. I am also using a gitea action to run terraform and commit the new tfstate file to the repo. I think this will be able to scale up with me for now once i finish my migration.

I had for a while been noticing large spikes in the disk backlog according to netdata as wells as swap useage errors. The swap useage errors especially were wierd because the VM can ballon up to 200GB of RAM and i know i am not using that much. These errors combined were leading to system freezes where i had to go into proxmox and stop the VM and then start it. So to start investigating i ran ...

So in my process of preparing to switch to starlink I setup headscale to replace my wireguard vpn. I thought that I would have to use tailscale as I presumed the headscale server would require a port. It does but it just so happens to use https which I can tunnel with cloudflare. So with that in mind I setup headscale in a docker container and connected my phone with no issue. The server side was slightly harder as a i used the tailscale docker container and accidentally removed the envs so i spent a while trying to figure out why the state file wasn’t saving properly. That is fixed now and it works fine. However on further investigation I realised that I could not use it for my main purpose as cloudflare tunnels do not support web socket PUSH.

I am not mad I will be keeping ipv4 for my LAN however for reasons beyond my control I am moving from a wired internet connection to starlink that uses carrier grade nat. This leaves me with a range of problems I need to solve to get my homelab working without access to public ports. The first solution I am going to use is cloudflare tunnels. So i go from… to… This is not terrible and was easy to setup and integrate in my CI/CD pipeline however it is not a 1 size fits all solution as I still have game severs, mail, VPN, git ssh, remote access and sftp. I have some ideas such as headscale or similar for VPN. Git ssh i can tunnel but it would not be public so i feel it is pointless. The others I will have to try ipv6 when i change to starlink.

I recently decided that I wanted to explore the entire hashicorp stack. I have been using vault and terraform since I moved to gitea for container management. However I decided that learning the entire stack would be interesting and teach me more about IaC and cloud services. I chose to do a install of all the software in a VM running arch that has 200GB of ballooning ram and 20 CPU cores at 3 GHz. It will run the containers in docker on the same server. I managed to install nomad and consul quickly however vault took a minute as the config file is just in /etc not /etc/vault.d. I then also had to add tls_disable = true to the config files which I am currently storing in gitea and deploying with gitea actions. I initialized vault with 5 keys and 3 minimum. I decided not to use my existing vault as I want to start clean and fresh. I installed vagrant and boundary-bin from the AUR and had to curl the binary for waypoint which is currently had coded to version 11.4 but i hope to make dynamic at some point. I then setup waypoint with waypoint install -accept-tos -platform nomad -nomad-host-volume=waypoint-data -nomad-runner-host-volume=waypoint-data-client. However I needed to set export NOMAD_ADDR=http://192.168.1.13:4646 first.

I have been using nginx reverse proxy for quite a while and it works great. However with my recent switch to automation and git managment i have decieded to switch to caddy i know many people love traefik however i have tried it in the past and had some errors so decided to investigate caddy. My current setup works as such in each projects git repo I store the caddy file their is then an action which will copy the caddy file to the correct folder before running caddys hot reload command caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile.

Since I started storing all my docker compose files in git i have wanted to create a public mirror as a disaster recovery option. To that end i needed a way to remove sensitive envs from the files. To achive this I set the docker env to a shell env like so TEMP: ${TEMP} This shell env was set by a script which contained export TEMP=$(vault kv get -field=temp kv/data/docker). This vault currently runs as a single node in docker and has to be manually unsealed as needed.

It started when i tired to set up my emails on Thunderbird mail on arch I noticed it wouldn’t connect. I checked the logs and found “2025-01-16 21:04:11,919 WARN exited: amavis (exit status 1; not expected)” I fixed this relatively quickly by removing “amavisd.pid”. After this it booted but I still couldn’t connect. I tracked this down by using dig and whats my ip to see that the dns record was wrong. I fixed this by switching to a cname record set to @ so i only need 1 ddns setup. This finally fixed it.

I decided that I wanted to use Proxmox as my main operating system as it has a large community and a large available library of tutorials. So i followed this guide (https://www.youtube.com/watch?v=sZcOlW-DwrU&t=487s) which made installing relitivly easy. The only small issue i found post install was that the 4 drives i had setup for proxmox were set up as 2 storage sections called local and local-zfs. I still don’t fully understand what happened here but i just put important files on local-zfs and it seem to work fine.

Ever since I setup mailcow I have been annoyed that it runs with docker volumes and updates with a script. This makes it imposible to run how I want. I feel the main benefit of docker should be all the files needed are backed up and I can use the docker compose file to spin the container up anywhere. I have tried may times to fix these problems in mailcow and have always failed. So I finally gave up with mailcow and started looking at other options. I quite quickly found docker mailserver as my main option. I don’t know if I didn’t see it the first time round or if I was scared of by the lack of a GUI. What ever the case now I am happy to use the command line for everything. ...