R720 on Keldo.uk

NUT

Thu, 15 Jan 2026 00:00:00 +0000

Over the summer I got a R1500 G3 ups for my server it is quite over kill however i hope to connect my POE switch and starlink internet to also keep wifi up during a power outage. To that end it would be rather nice if my server actually shut down as the power was running out rather than just continuing until it dies when the battery runs out. The last piece of this puzzle was a usb a to usb b cable which I recently got. This connection goes from my ups to my r720 on which i decided to install nut. The server is running proxmox as its primary OS so I was able to install nut with apt however the first few guides I followed ended up not working for what ever reason. The one that finally worked was this one. Once I had that sorted I setup a web nut docker container in terraform which all went smoothly I hope to eventually add the stats to graphana for data visulisation but for now i am happy.

ZFS Ram consumtion

Wed, 30 Jul 2025 00:00:00 +0000

I had for a while been noticing large spikes in the disk backlog according to netdata as wells as swap useage errors. The swap useage errors especially were wierd because the VM can ballon up to 200GB of RAM and i know i am not using that much. These errors combined were leading to system freezes where i had to go into proxmox and stop the VM and then start it. So to start investigating i ran

Headscale

Thu, 17 Jul 2025 00:00:00 +0000

So in my process of preparing to switch to starlink I setup headscale to replace my wireguard vpn. I thought that I would have to use tailscale as I presumed the headscale server would require a port. It does but it just so happens to use https which I can tunnel with cloudflare. So with that in mind I setup headscale in a docker container and connected my phone with no issue. The server side was slightly harder as a i used the tailscale docker container and accidentally removed the envs so i spent a while trying to figure out why the state file wasn’t saving properly. That is fixed now and it works fine. However on further investigation I realised that I could not use it for my main purpose as cloudflare tunnels do not support web socket PUSH.

Hashicorp Install

Tue, 10 Jun 2025 00:00:00 +0000

I recently decided that I wanted to explore the entire hashicorp stack. I have been using vault and terraform since I moved to gitea for container management. However I decided that learning the entire stack would be interesting and teach me more about IaC and cloud services. I chose to do a install of all the software in a VM running arch that has 200GB of ballooning ram and 20 CPU cores at 3 GHz. It will run the containers in docker on the same server. I managed to install nomad and consul quickly however vault took a minute as the config file is just in /etc not /etc/vault.d. I then also had to add tls_disable = true to the config files which I am currently storing in gitea and deploying with gitea actions. I initialized vault with 5 keys and 3 minimum. I decided not to use my existing vault as I want to start clean and fresh. I installed vagrant and boundary-bin from the AUR and had to curl the binary for waypoint which is currently had coded to version 11.4 but i hope to make dynamic at some point. I then setup waypoint with waypoint install -accept-tos -platform nomad -nomad-host-volume=waypoint-data -nomad-runner-host-volume=waypoint-data-client. However I needed to set export NOMAD_ADDR=http://192.168.1.13:4646 first.

Caddy Setup

Sun, 11 May 2025 00:00:00 +0000

I have been using nginx reverse proxy for quite a while and it works great. However with my recent switch to automation and git managment i have decieded to switch to caddy i know many people love traefik however i have tried it in the past and had some errors so decided to investigate caddy. My current setup works as such in each projects git repo I store the caddy file their is then an action which will copy the caddy file to the correct folder before running caddys hot reload command caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile.

Secrets in Vault

Mon, 24 Mar 2025 00:00:00 +0000

Since I started storing all my docker compose files in git i have wanted to create a public mirror as a disaster recovery option. To that end i needed a way to remove sensitive envs from the files. To achive this I set the docker env to a shell env like so TEMP: ${TEMP} This shell env was set by a script which contained export TEMP=$(vault kv get -field=temp kv/data/docker). This vault currently runs as a single node in docker and has to be manually unsealed as needed.

Mail Server Down

Thu, 16 Jan 2025 00:00:00 +0000

It started when i tired to set up my emails on Thunderbird mail on arch I noticed it wouldn’t connect. I checked the logs and found “2025-01-16 21:04:11,919 WARN exited: amavis (exit status 1; not expected)” I fixed this relatively quickly by removing “amavisd.pid”. After this it booted but I still couldn’t connect.

I tracked this down by using dig and whats my ip to see that the dns record was wrong. I fixed this by switching to a cname record set to @ so i only need 1 ddns setup. This finally fixed it.

Proxmox Install

Thu, 26 Dec 2024 00:00:00 +0000

I decided that I wanted to use Proxmox as my main operating system as it has a large community and a large available library of tutorials. So i followed this guide (https://www.youtube.com/watch?v=sZcOlW-DwrU&t=487s) which made installing relitivly easy. The only small issue i found post install was that the 4 drives i had setup for proxmox were set up as 2 storage sections called local and local-zfs. I still don’t fully understand what happened here but i just put important files on local-zfs and it seem to work fine.

Docker Mailserver

Fri, 22 Nov 2024 00:00:00 +0000

Ever since I setup mailcow I have been annoyed that it runs with docker volumes and updates with a script. This makes it imposible to run how I want. I feel the main benefit of docker should be all the files needed are backed up and I can use the docker compose file to spin the container up anywhere.

I have tried may times to fix these problems in mailcow and have always failed. So I finally gave up with mailcow and started looking at other options. I quite quickly found docker mailserver as my main option. I don’t know if I didn’t see it the first time round or if I was scared of by the lack of a GUI. What ever the case now I am happy to use the command line for everything.

Pterodactyl 504

Tue, 08 Oct 2024 00:00:00 +0000

It started when I went back to my pterodactyl panel after a long time. I saw a red heart and a invalid ssl cert warning in console. I renewed the cert and the heart went green. However my servers still wern’t loading. So I changed the forwarding from cloudflare to nginx proxy. This still didn’t work. Then I renewed the cert as rsa type and noticed it put it a folder that ended in 00001. So I deleted the one with out the numbers and then renamed the one left to remove the numbers. One restart later and it was all working.

Redoing My VPN

Tue, 13 Aug 2024 00:00:00 +0000

During a backup test I did recently I found that wireguard wasn’t working properly. I have no idea how long this has been the case but to fix it I started trying to change the config however when that lead to no improvment I decided to reinstall. This is something I have considered before as my current setup doesn’t have a gui which would be nice so I use the repository (https://github.com/wg-easy/wg-easy?tab=readme-ov-file). The install was over all easy however the password hashing didn’t work on a super strong password. This seems to be working now and I will probably redo the backup test for wireguard soon.

Kasm storage complications

Thu, 01 Aug 2024 00:00:00 +0000

When I revisted my kasm server I found I was unable to launch a session. After running docker ps I saw that the guac container was contantly restarting. After searching the error message I found this link (https://kasmweb.atlassian.net/servicedesk/customer/portal/3/article/8126468) which shows a fix. After following this guide I got it working again.

However the same day one of my friends tried to download a game on a steam continer and I don’t know why but the entire vm froze up and I had to hard stop it. When I restarted it I found the config files filled with @ and nothing else.

Wazuh File Premissions

Mon, 29 Jul 2024 00:00:00 +0000

I noticed today that Wazuh wasn’t working after I restarted my server. I saw in the logs it was failing the bootstrap test. I thought the issue was the mass of file permission error that were appearing before this error showed. However after messing around with file permissions I noticed a error about sysctl vm.max_map_count after running “sysctl -w vm.max_map_count=262144” it started working however I have no idea how this wasn’t a issue until now.

Proxmox no valid subscription warning

Mon, 22 Jul 2024 00:00:00 +0000

I decided to finally remove that annoying pop up that appears every time I log into my server. I found this guide (https://www.youtube.com/watch?v=5ftWGupG66A). i ignored the first section about the repositories but the section about removing the warning was clear and easy to follow. Once I had restarted the service I went to check the web UI and I was still getting the warning so I checked in a incognito tab and saw it was gone I just need to wait for my cache to update.

Wazuh

Mon, 22 Jul 2024 00:00:00 +0000

I was originaly planing to try security onion but while reasurching it I came across Wazuh which runs on docker which I always prefer. Once I installed it using a edited version of the compose file in the github repository.

I managed to install the agent on two of my servers however on my main proxmox host I didn’t understand that the server I was the ip of the main host not the server I was installing it on. This lead to a botched install I haven’t been able to fix yet.

Portainer Docker exec

Fri, 19 Jul 2024 00:00:00 +0000

One day when I tried to use the console section of portainer I got the error “Unable to retrieve image details”. After looking online I saw that I needed to update portainer so I started looking for the compose file and couldn’t find it. Well that was a bit annoying so I ran “docker run –rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose …” which worked as expected. I then pulled the latest image and trimed the compose file. When I removed and restarted the container I was annoyed to see it still wasn’t working.

Wordpress https

Wed, 17 Jul 2024 00:00:00 +0000

This has for a long time been probably my second biggest issue with my homelab. My main website that your reading this on has been running on http. This happened because i use the domain keldo.uk which is also the domain I use for email. This ment I couldn’t route through cloudflare like everything else.

My original attempt to fix this was with trafik but when I got as far as forwarding the website all I got was a white page. I presumed that nginx proxy manager would be the same however I was proven wrong when I tried today.

Ollama Install

Sun, 14 Jul 2024 00:00:00 +0000

I started with the docker compose file from https://github.com/open-webui/open-webui. I then edited this changing the volumes to mounts on my nas and changing the port. I then installed a range of models some designed for general conversations and others for coding.

I then wanted to add tab auto complete to vscode so i used a section of technotim’s guide (https://technotim.live/posts/ai-stack-tutorial/). This worked fine and was all round a simple process. I look forward to trying to customise some of the models.

Ansible Install

Wed, 10 Jul 2024 00:00:00 +0000

I started by following this video https://www.youtube.com/watch?v=NyOSoLn5T5U. Once i customised the compose file I managed to setup my first playbook. I decided to store playbooks on my nas for safety and the first one I made was a apt update.

I then started tinkering with more complicated ones for proxmox and my dns settings. This is still what I am doing as proxmox is working through proxmoxer and if not exactly simple but I am happy with the progress I am making.

Don't delete random docker Volumes

Mon, 24 Jun 2024 00:00:00 +0000

It started when I was trying to install change detection. I was getting a error about not having space on the drive so I decided to delete some unused docker volumes. This lead to a new far more concerning error “mkdir /data/compose: no such file or directory”.

After looking online I realised this could be bad so I decided to download a copy of passbolt and then reboot. The first thing I noticed was proxmox stopped working but I realised this was just my dns. The when I did a docker ps it showed that everything was starting fine. However when I went to Portainer it told me to setup a new install.

Broken mailcow

Thu, 30 May 2024 00:00:00 +0000

I began by trying to move my mailcow install from running on docker volumes to storing data on my nas. I started copying the files however after around the 5th volume I remembered the continers were still online. After I continued even finding a copy of the docker compose.

However when I ran the compose I got serveral errors about mysql. I realised I had corrupted the mysql database. I tried several edits to the mysql files and failed. After that I went to launch the old copy and realised that was corrupted as well.

ICloudpd Notifications

Tue, 28 May 2024 00:00:00 +0000

The one feature of Icloudpd that I dislike is the requirement to constantly redo 2fa. I also would have no way of knowing when this is untill I set up one of the notification options.

I decided since there was no smtp which would be my normal choice I would try out the discord web hook function which I hadn’t used before. The setup went fine untill I went to test it I restarted the container and got a notification that it started however, almost strait after that I got a message that it couldn’t access the folder that stored my photos. I re aplied the acl which fixed it but I need to find a away to re do the acl on a schedual.

SyncTrayzor

Mon, 27 May 2024 00:00:00 +0000

I have been using the default syncthing windows client for a while. However, the task sceudular setup had never worked properly and there for I had been forced to run it manually every now and then.

Luckily I came across a reddit post that mentionted SyncTrayzor at first I thought it was a different software but then I found out it worked with my existing syncthing server. I downloaded the installer and ran it. It worked even better than expected as it managed to use my existing config with the only change needed was to add an allow newer config flag in the settings.

Kasm install

Sat, 11 May 2024 00:00:00 +0000

I wanted to setup a more simple shareable alternative to VMs. Kasm was a great option which I was very excited to try. I began by setting up the normal docker compose with my normal bind mounts. However it kept getting stuck in a loop after accepting the terms. So I had to switch to a full install on a vm.

After that I followed the stock install guide and it all went fine. I setup the workspaces I wanted and they installed fine. However the other issue I ran into was when I tried to add ssh to a vm. No matter what I tried it just wouldn’t work so in the end I decided I could just use the Ubuntu workspace and manually ssh in.

iCloudpd

Fri, 10 May 2024 00:00:00 +0000

I wanted to try and find a better way to backup my photos from my iPhone to my NAS. I had been using Nextcloud with the app on my phone to do this however is was slow and nextcloud pritiy much always ran so slow a page could take around 60 seconds to load.

I had seen iCloudpd before as a application on TrueNAS and so wanted to try it out. I set up a docker compose and signed in. However I then started getting a message to create the .mounted file which I had already done. This prity much instantly told me that there was somthing wrong with my mount. However when I looked it all seemed fine so I was forced to ssh in and relook at the new mount I had made to my NAS. After some failed ideas I noticed that the mount only had empty folders in it. This finaly reminded me that my Photos folder was acutualy a dataset so I switched the nsf share to that and it started to run a backup.

Passbolt update

Mon, 06 May 2024 00:00:00 +0000

I began to notice issues when my passwords in the ios passbolt app when it was’t showing my passwords. I quickly found a range of posts about historical issues with ios 17 batas but as far as I could see they had been solved in app version 15 and I was on 18.

The next step I took was to run the health check this found that I had two issues: my install didn’t force https which is fine because if uses cloudflare; and my install was very out of date. Updating is one of the big draw backs of docker in my opinion and while there was a page about updating on the passbolt website I decided to just figure it out through portainer.

Mailcow ssl

Sat, 13 Apr 2024 00:00:00 +0000

The first thing I noticed was wrong was that my Iphone was giving me a pop up telling me it could verify the identity of mail.keldo.uk. My first though was that the ssl cert had expired but as far as I knew Mailcow should automaticly refresh the cert. Luckily it was easy to debug as Mailcow has a nice selection of logs and I just open the acme logs and Imedialty saw the issue “mail.keldo.uk - Found AAAA record for mail.keldo.uk: 2a00:23c7:82c2:6401:be24:11ff:fe6d:10fa - skipping A record check” was apearing every hour. I setup ipv6 records when I set up mailcow however since I have a dynamic IP and haven’t set up a ddns for ipv6 it was seeing ipv6 skiping ipv4 and then seeing the ipv6 didn’t match. The simple solution was I just removed the ipv6 record and then the next hour it was fixed.

Hard Drive Errors

Wed, 10 Apr 2024 00:00:00 +0000

It began when I checked my emails and saw a email from TrueNAS it told me that one of my drives had failed and the pool was degraded. When I investigated I saw that the drive only had 13 errors but since I had 6 drives I wasn’t using I decided to swap it out I passed through another 2 drives and began the replacement tool. It took around three hours but went smoothly with the pool up the entire time.

Metasploitable 2

Fri, 05 Apr 2024 00:00:00 +0000

For a addition to my cyber security home lab I added a metasploitable VM. I made a VM with weak specs (but it should be fine) and then downloaded, unziped and converted it then just added it to the VM. I made sure to put it in VLAN 30 then booted it launched and I could access the UI. I look forward to practising with this.

Kiwix

Thu, 04 Apr 2024 00:00:00 +0000

I began by changing the command to a docker compose it was easy apart from a random *.zim that was at the end. I just skipped over it and presumed it would be fine but there was no url so I thought I needed a zim for it to share so I downloaded all of Wikipedia which took around 1 hour so not to bad but after a restart there was still no page. I returned to that random .zim and after some googling and trial and error I found I needed to add the line “command: ‘.zim’” after that it worked fine and I added ask Ubuntu as well.

Recoved WP db

Mon, 01 Apr 2024 00:00:00 +0000

I started by spinning up phpmyadmin I didn’t care about the settings as I was only going to use it for a bit. At first i tried moving files 1 by 1 but when that gave error I moved all the files I had in my mount and tried to login it said my login was wrong so I tried my wordpress login which worked. I navigated to the pages table and was horrified to see it only have 4 rows. As a last hope I restored a older copy of the db I had from just after the fail. When I checked this time there were 159 posts and a quick check showed I was all there!!

Opnsense

Sun, 31 Mar 2024 00:00:00 +0000

I began with goal of making a cyber security home lab for learning on my r720. The first part of this was to set up a separate network so I could change settings without taking down my familys internet. So I installed Opnsense on a vm and added two network device my normal bridge(WAN) and a bridge that is bound to nothing(LAN) I booted into the live environment and ran the installer. When it rebooted I checked the assignments and found that the wan and lan were the wrong way round. I changed that and the interface ip and then connected my kali box to the LAN and accessed the interface.

Ollama K80

Fri, 22 Mar 2024 00:00:00 +0000

This project has been a roll-coaster and one of the hardest things I have attempted my goal when I set out was simple to run local Ollama models on a cheap £60 k80 GPU. My first attempted end very quickly when I found out that the GPU I got was faulty and it spewed out smoke within a second of first boot.

After I confirmed the rest of my server was unharmed I got a full refund and purchased another K80. This one did not fail on first boot and so I was able to start setting up my GPU. I spent the next month trying on and off failing every time.

Nextcloud

Mon, 12 Feb 2024 00:00:00 +0000

Today I installed nextcloud through docker I’m using the Linux server image and the install was very smooth however when I got to the web UI it was slow and I mean anywhere from 5 seconds to 60 per click! But I continued thinking it was just running a background install and would soon speed up.

I added my NAS as external storage which worked fine and then setup the recognize app which is basically facial regognision like your phone does where it groups the same people and lets you name them.

Removing My Applications Dataset

Fri, 02 Feb 2024 00:00:00 +0000

So I recently tried to remove my Application Dataset on my TrueNAS scale NAS. It was just annoying seeing it constantly when i didn’t have any applications running through TrueNAS so went to remove it and it said it was busy. I thought that can’t be right I know there are no applications installed so I spent a while online and found out how to force remove the dataset and I did.

Mailcow Install

Sat, 13 Jan 2024 00:00:00 +0000

I decided I wanted to be able to get emails from my services as that is the mail method most of them support. For my mail server i decided to use mailcow rather than try to stitch together my own group of programs. The install was easy with no errors by simply just using a Docker compose file. Its smtp work fine with TrueNAS, Pterodactyl, Nextcloud, Proxmox and Passbolt however i could not get it working with Mealie.

Dynamic DNS

Thu, 04 Jan 2024 00:00:00 +0000

Today I set up my DDNS for my domains that need to link to my dynamic public ip address. I followed this guide https://www.youtube.com/watch?v=rI-XxnyWFnM by NetworkChuck which worked amazingly. I didn’t want to run it on its own vm so I made a ubuntu container in Proxmox and apart from mixing up api and global api keys I managed everything first try.

Pterodactyl Install

Thu, 04 Jan 2024 00:00:00 +0000

I began by trying to install pterodactyl as 2 separate containers on my main docker VM however this ended up having far to many issues for me to try and solve. My next idea was to use the ubuntu container and install them in there however this also didn’t work. In the end i just created a new VM and using this script (https://github.com/pterodactyl-installer/pterodactyl-installer). Which after a bit of fiddling with the ssl record and making the panel and the wing have unique domains then it finally worked. I also made sure that the servers are stored on a nfs share for easy backups.

Pi Hole

Thu, 28 Dec 2023 00:00:00 +0000

I chose to use Pi hole for ad blocking and local dns as it has a large community and I have used it in the past. The install went fine and then i used http://firebog.net to get a list of ad blocks. The only issues I have run into is that I blacklisted a google link that stopped a lot of surchs i did and that it only prompts you to re loggin after you fail to do something.

TrueNAS Install

Tue, 26 Dec 2023 00:00:00 +0000

The install for TrueNAS was very simple as it was something i had practiced multiple times as a VM on my PC to mess around with smb shares. I decided to install scale as the choice between core and scale seemed very easy for me as i felt that while core was a small bit faster scale had a far more promising future and had plenty of people working to improve its speed.

R720

Mon, 25 Dec 2023 00:00:00 +0000

For christmas 2023 i got a second hand R720 from ebay i cost around £150 it has: 2 E5-2690 v2s for a total of 20 cores at 3 GHz; 256gb of ram; duel 10gig sfp+ ports; and 16x900gb 2.5" sas hdds (However only 2 drive sleds).

My plan for this server was to use proxmox to build an environment that gave me space to run some useful services as well as mess around with new software.