Tag Archives: cybersecurity

Proxmox Time Sync

Posted on by
Reply

Syncing time is essential on Proxmox servers, especially in a cluster — time drift can cause serious issues with quorum, backups, and HA.

Here’s how to ensure accurate time sync on your Proxmox nodes:

✅ Recommended Method: Use systemd-timesyncd (built-in on Debian)

🔧 1. Enable and start time sync:

sudo timedatectl set-ntp true

Verify:

timedatectl status

Look for:

NTP service: active
System clock synchronized: yes

This uses systemd-timesyncd, which is light and reliable for Proxmox servers.

🧰 Alternative: Use chrony (Recommended for more control)

chrony is more robust, especially if your nodes might be offline/online often or you’re doing HA.

1. Install chrony:

sudo apt install chrony

2. Edit config (optional):

sudo nano /etc/chrony/chrony.conf

Add or edit NTP servers:

server time.google.com iburst
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst

3. Restart and enable:

sudo systemctl enable chrony
sudo systemctl restart chrony

4. Check sync status:

chronyc tracking

🧪 Verify Time Sync on Proxmox

On each node:

timedatectl

All nodes should show:

  • NTP service: active
  • System clock synchronized: yes
  • Same or very close times

🧠 Bonus: Sync RTC (hardware clock)

To sync system time to the hardware clock:

sudo hwclock --systohc

Let me know if you’re using a cluster and want to designate one Proxmox node as an NTP server for others, or if you’d like to sync from your Pi-hole box or router!

Perfect — turning one of your Proxmox nodes into an NTP server is a great idea for local time sync, especially in a cluster or if internet access is limited.

Here’s how to do it using chrony, which is ideal for Proxmox nodes:

🧭 Overview

  • One Proxmox node will be the NTP server.
  • The others will sync from it as NTP clients.

🖥️ On the Proxmox NTP Server (e.g., pve1)

1. Install chrony if not already installed:

sudo apt update
sudo apt install chrony

2. Edit the config to allow LAN clients

sudo nano /etc/chrony/chrony.conf

Add or uncomment these lines:

# Allow LAN subnet (adjust to your LAN range)
allow 192.168.1.0/24

# Make this machine act as an NTP server
local stratum 10

Optional: Use a known upstream NTP source for initial sync:

server time.google.com iburst
server 0.pool.ntp.org iburst

3. Restart chrony and enable it

sudo systemctl restart chrony
sudo systemctl enable chrony

🖥️ On the Proxmox Clients (e.g., pve2, pve3)

1. Install chrony:

sudo apt install chrony

2. Configure it to sync from the Proxmox NTP server:

sudo nano /etc/chrony/chrony.conf

Comment out existing server lines, and add:

server 192.168.1.10 iburst   # Replace with the IP of your Proxmox NTP server

3. Restart and enable:

sudo systemctl restart chrony
sudo systemctl enable chrony

4. Verify sync status:

chronyc sources
chronyc tracking

You should see something like:

Name/IP address             Stratum Poll Reach LastRx Last sample
===============================================================================
192.168.1.10                     10   6   377    32    -34us[ -44us] +/-  200us

🧪 Bonus: Test if the server is responding

From another node:

chronyc sources

Or:

ntpq -p 192.168.1.10

Let me know if you want to also allow the Pi-hole box or other clients on your LAN to sync from this NTP server too — it’s as easy as expanding the subnet in allow.

The Hardened (Home) Data Center

Posted on by
Reply

What’s a good plan for remote access to my smart home during a power outage when I’m on an extended vacation?

Jump to What I Did.

Since you’ll be away for a while, you need a robust failover plan to ensure continuous remote access to your smart home and Proxmox cluster. Here’s a multi-layered approach using your existing equipment:

🌐 1️⃣ Internet Failover

Since you have:

  • Primary ISP: AT&T Fiber (Nokia BGW320-505)
  • Backup ISP: T-Mobile 5G Hotspot (30GB/month before throttling)
  • Spare ASUS RT-AC68U Router (WAN Failover Capable)

🛠 Steps to Set Up Failover

  1. Enable Dual WAN Failover on ASUS Router
    • Set AT&T as primary WAN and T-Mobile Hotspot as secondary WAN
    • Choose failover mode (not load balancing)
    • Adjust health check settings to detect AT&T failures
  2. Avoid Double NAT Issues
    • If AT&T’s Nokia router is in router mode, enable bridge mode on the ASUS router
    • Assign static routes if needed to prevent conflicts

🔌 2️⃣ Power Resilience

Since you have a UPS, ensure:

  • Proxmox nodes, routers, and modem are on the UPS
  • Use a smart plug on your modem & router (in case they lock up)

🛠 Automation Ideas

  • Smart Plug Auto-Reboot
    • Use a TP-Link Kasa or Tasmota plug
    • Set up a rule: If internet is down for 10+ min, cycle power to the router/modem

🔐 3️⃣ Remote Access Setup

🛠 VPN Options

  1. Tailscale VPN (Easiest & Most Reliable)
    • Run Tailscale on your Raspberry Pi (or Proxmox VM)
    • Enables access even if your IP changes
    • No port forwarding needed
  2. PiVPN (WireGuard or OpenVPN)
    • Run on a Raspberry Pi
    • Requires port forwarding, so it may fail if your ISP changes your IP
  3. Cloudflare Tunnel (Great Backup Plan)
    • Provides a free tunnel to access your smart home services
    • Works even if your public IP changes

🚨 4️⃣ Automatic Recovery from Failures

🛠 Proxmox & Server Auto-Recovery

  • Use Smart Plugs for Power Cycling
  • Watchdog Timer on Proxmox
    • If a Proxmox node stalls, it will auto-reboot

🛠 Network Watchdog on Raspberry Pi

Run a simple script to detect network failures and auto-reboot devices:

#!/bin/bash
if ! ping -c 5 8.8.8.8 &> /dev/null; then
  echo "Internet down! Rebooting router..."
  ssh user@smartplug "turn off && sleep 10 && turn on"
fi

(Schedule with crontab -e to run every 10 minutes)

📌 Summary: Best Remote Access Plan

✅ Failover Internet → ASUS RT-AC68U handles automatic switch to T-Mobile 5G
✅ UPS Protection → Keeps critical devices online during power outages
✅ Smart Plug Automation → Auto-reboots router/modem if unresponsive
✅ VPN Access (Tailscale) → Provides secure access even if ISP changes IP
✅ Proxmox Watchdog → Ensures servers auto-recover from crashes
✅ Cloudflare Tunnel (Optional) → Secondary remote access method

🚀 Next Steps

  • Do a test run before your trip
  • Monitor logs remotely (Tailscale + Home Assistant)
  • Set up alerts (Home Assistant + Pushover/Telegram)

Would you like help setting up any of these steps? 🚀

I implemented dual Wan on the ASUS router and used IP passthrough on the Nokia router. Failover mode works. Most of my automation works with Alexa. I have the UPS. I’ll implement the RPi watchdog. I think I’ll shutdown Proxmox during a power outage to preserver the UPS. I think I can get the temperature from my Ambient Weather station if it’s on the UPS. Two of my blink cameras work on batteries. I’ll need an Echo on UPS.

That sounds like a solid plan for maintaining remote access and automation during an outage. If your ASUS router supports custom scripts, you might consider adding automation to gracefully shut down Proxmox when the UPS battery level gets too low. Let me know if you want help setting that up!

how would you detect the battery level?

You can detect the UPS battery level using Network UPS Tools (NUT), which allows you to monitor your UPS from a Raspberry Pi, your Proxmox servers, or another system on your network. Here’s how you can set it up:

Step 1: Install NUT on Proxmox or a Raspberry Pi

On your Proxmox node or Raspberry Pi (whichever is connected to the UPS via USB):

sudo apt update
sudo apt install nut nut-client

Step 2: Configure NUT to Read UPS Data

Edit the NUT configuration file to recognize your UPS:

sudo nano /etc/nut/ups.conf

Add an entry for your UPS. For example, if it’s an APC UPS, it may look like this:

[myups]
    driver = usbhid-ups
    port = auto
    desc = "My UPS"

Save and exit.

Restart the NUT service:

sudo systemctl restart nut-server

Step 3: Check Battery Status

Run the following command to check UPS status:

upsc myups

Look for a value like:

battery.charge: 85

This tells you the current battery percentage.

Step 4: Automate Proxmox Shutdown on Low Battery

Create a script (/usr/local/bin/ups-shutdown.sh) that will shut down Proxmox when the battery is low:

#!/bin/bash
BATTERY_LEVEL=$(upsc myups battery.charge)

if [ "$BATTERY_LEVEL" -lt 20 ]; then
    echo "Battery low ($BATTERY_LEVEL%). Shutting down Proxmox..."
    /usr/sbin/poweroff
fi

Make it executable:

chmod +x /usr/local/bin/ups-shutdown.sh

Step 5: Run This Script Periodically

Add this to cron to check the battery level every minute:

sudo crontab -e

Add this line at the bottom:

* * * * * /usr/local/bin/ups-shutdown.sh

This setup will ensure Proxmox shuts down when the UPS battery gets too low.

Would you like help with setting up email or push notifications when the power goes out?

To get email or push notifications when the power goes out, you can integrate the UPS monitoring system with sendmail (for email notifications) or use Pushover or Telegram (for mobile push notifications).

1. Send Email Notification on Power Outage

You’ll need msmtp and mailutils to send emails from the Raspberry Pi or Proxmox server.

Install Required Packages

sudo apt update
sudo apt install msmtp mailutils

Configure msmtp

Edit the configuration file:

sudo nano /etc/msmtprc

Add the following, adjusting for your SMTP provider (Gmail, Outlook, etc.):

defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp.log

account default
host smtp.gmail.com
port 587
from your-email@gmail.com
user your-email@gmail.com
password your-email-password

Save and exit.

Test Email Sending

Try sending an email:

echo "Test email from Raspberry Pi" | mail -s "UPS Test Email" your-email@example.com

If you receive the email, the setup is working.

Modify the Shutdown Script to Send an Email

Edit the /usr/local/bin/ups-shutdown.sh script to send an email when the UPS is on battery:

#!/bin/bash
BATTERY_LEVEL=$(upsc myups battery.charge)
ONLINE_STATUS=$(upsc myups ups.status)

if [[ "$ONLINE_STATUS" != "OL" ]]; then
    echo "Power outage detected! UPS running on battery. Battery level: $BATTERY_LEVEL%" | mail -s "Power Outage Alert" your-email@example.com
fi

if [ "$BATTERY_LEVEL" -lt 20 ]; then
    echo "Battery low ($BATTERY_LEVEL%). Shutting down Proxmox..." | mail -s "Critical Battery Alert - Shutting Down" your-email@example.com
    /usr/sbin/poweroff
fi

2. Push Notifications Using Telegram

If you prefer mobile push notifications, you can use Telegram bots:

Create a Telegram Bot

  1. Open Telegram and search for BotFather.
  2. Start a chat and use the command /newbot.
  3. Give your bot a name and username.
  4. Copy the API token from BotFather.

Find Your Chat ID

  1. Open https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates in a browser.
  2. Send your bot a message.
  3. Refresh the browser; your chat ID will appear.

Modify the Shutdown Script to Send Telegram Messages

Edit /usr/local/bin/ups-shutdown.sh:

#!/bin/bash
BATTERY_LEVEL=$(upsc myups battery.charge)
ONLINE_STATUS=$(upsc myups ups.status)
TELEGRAM_BOT_TOKEN="YOUR_BOT_TOKEN"
TELEGRAM_CHAT_ID="YOUR_CHAT_ID"

send_telegram() {
    curl -s -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage" \
        -d chat_id="$TELEGRAM_CHAT_ID" \
        -d text="$1"
}

if [[ "$ONLINE_STATUS" != "OL" ]]; then
    send_telegram "⚠️ Power outage detected! UPS running on battery. Battery level: $BATTERY_LEVEL%"
fi

if [ "$BATTERY_LEVEL" -lt 20 ]; then
    send_telegram "🔴 Critical Battery Alert! Battery level: $BATTERY_LEVEL%. Shutting down Proxmox."
    /usr/sbin/poweroff
fi

3. Running the Script on a Schedule

Edit cron:

sudo crontab -e

Add this line:

* * * * * /usr/local/bin/ups-shutdown.sh

This checks the UPS status every minute.

Would you like to add any additional notification methods, like Pushover or Home Assistant alerts?

What I Did

Sure! Here’s a quick and clear guide to set up SSH key-based login from your local machine to a remote Linux server, so you can skip password prompts.

🔑 Step-by-Step: SSH Key Setup

1. Check for existing SSH keys on your local machine

ls ~/.ssh/id_rsa.pub

If it exists, skip to step 3.

2. Generate a new SSH key pair (if needed)

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
  • Press Enter to accept defaults.
  • You can set a passphrase (or leave it blank for no prompt).

This creates:

  • Private key: ~/.ssh/id_rsa
  • Public key: ~/.ssh/id_rsa.pub

3. Copy your public key to the remote server

ssh-copy-id user@remote-host
  • Replace user with your username and remote-host with the IP or hostname.
  • It will prompt for your password once to copy the key.
Alternate method (if ssh-copy-id isn’t available):
cat ~/.ssh/id_rsa.pub | ssh user@remote-host 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys && chmod 700 ~/.ssh'

4. Test it

ssh user@remote-host

You should now log in without entering a password.

✅ Optional: Configure Passwordless sudo

If you want to run shutdown commands via SSH without typing the sudo password:

  1. On the remote server, run:
sudo visudo
  1. Add this line at the bottom (change user to your username):
user ALL=(ALL) NOPASSWD: /sbin/shutdown, /sbin/poweroff

Now your remote shutdown command works like this:

ssh user@remote-host 'sudo /sbin/shutdown -h now'

Crontab

  1. timedatectl # check time zone
  2. timedatectl set-timezone America/Chicago
  3. * * * * * date >> /home/pi/tz.txt #verify how crontab sees time

DHCP Failover with Pi-hole

Posted on by
Reply

I

Let’s get that Pi-hole DHCP reservation syncing set up! I’ll walk you through creating a sync script to keep the DHCP reservations (and any other Pi-hole configurations) identical between your Proxmox VM and Raspberry Pi 4. Here’s how we’ll do it!

✅ Step 1: Set up SSH key-based authentication

To allow the Raspberry Pi to pull the DHCP config from the Proxmox VM without a password:

👉 On the Raspberry Pi 4, generate an SSH key (if you haven’t already):

ssh-keygen -t rsa -b 4096 -C "pi-hole-sync"
  • Press Enter to save it to the default location (~/.ssh/id_rsa).
  • No passphrase is needed for this.

👉 Copy the key to your Proxmox Pi-hole VM:

ssh-copy-id <username>@<proxmox_vm_ip>
  • This allows the Raspberry Pi to SSH into the Proxmox VM without a password.

✅ Step 2: Create the sync script

👉 On the Raspberry Pi, create a script (e.g., pi-hole-sync.sh):

#!/bin/bash

# Define variables
PRIMARY_PIHOLE="<proxmox_vm_ip>"
PIHOLE_CONFIG_DIR="/etc/dnsmasq.d"
LOCAL_BACKUP_DIR="/home/pi/pihole_backup"

# Create backup directory if it doesn't exist
mkdir -p $LOCAL_BACKUP_DIR

# Sync DHCP reservations and Pi-hole config files
rsync -avz --delete $PRIMARY_PIHOLE:$PIHOLE_CONFIG_DIR/ $LOCAL_BACKUP_DIR/
rsync -avz --delete $LOCAL_BACKUP_DIR/ $PIHOLE_CONFIG_DIR/

# Restart Pi-hole to apply changes
pihole restartdns

👉 Make the script executable:

chmod +x pi-hole-sync.sh

✅ Step 3: Schedule automatic syncing with cron

👉 Open the crontab on the Raspberry Pi:

crontab -e

👉 Add a cron job to sync every 5 minutes (or whatever interval you prefer):

*/5 * * * * /path/to/pi-hole-sync.sh

✅ Step 4: Test the setup

👉 Run the sync script manually to confirm it works:

./pi-hole-sync.sh

👉 Check the DHCP reservations on the Raspberry Pi Pi-hole to ensure they match the Proxmox VM.

✅ Step 5: DHCP failover

👉 Now that both Pi-hole instances have the same DHCP reservations:

  • Set the Proxmox Pi-hole as the primary DHCP server.
  • Keep the Raspberry Pi Pi-hole DHCP disabled by default.
  • Use a failover script (as mentioned before) to detect when the primary goes down and enable DHCP on the Pi dynamically.

✨ Final result:

  • DHCP reservations stay synced automatically.
  • No more IP conflicts or lost reservations during a failover!
  • Your network stays resilient even if the Proxmox node goes down.

👉 Let me know if you’d like to add monitoring or notifications to this — we can polish this home lab setup to perfection! 🚀

Failover Script

Let’s set that up! I’ll give you a failover script that’ll:
✅ Periodically check if the primary Pi-hole VM is online.
✅ Enable DHCP on the Raspberry Pi if the primary goes down.
✅ Disable DHCP on the Raspberry Pi when the primary comes back online.

✨ Step 1: Create the failover script

👉 On the Raspberry Pi, create a new file:

nano /home/pi/dhcp-failover.sh

👉 Add the following code:

#!/bin/bash

# Primary Pi-hole VM IP address
PRIMARY_PIHOLE_IP="<proxmox_vm_ip>"

# Check if the primary Pi-hole is online
if ping -c 3 $PRIMARY_PIHOLE_IP &> /dev/null; then
    echo "$(date) - Primary Pi-hole is up."
    
    # Check if DHCP is running on the Pi - if so, disable it
    if pihole-FTL dhcp-discover | grep -q "Pi-hole DHCP server: active"; then
        echo "$(date) - Disabling DHCP on the Raspberry Pi (primary is up)."
        pihole disable-dhcp
    fi
else
    echo "$(date) - Primary Pi-hole is DOWN."
    
    # Check if DHCP is running on the Pi - if not, enable it
    if ! pihole-FTL dhcp-discover | grep -q "Pi-hole DHCP server: active"; then
        echo "$(date) - Enabling DHCP on the Raspberry Pi (primary is down)."
        pihole enable-dhcp
    fi
fi

✅ This script:

  • Pings the Proxmox Pi-hole VM.
  • Enables DHCP on the Pi-hole (Raspberry Pi) if the VM is unreachable.
  • Disables DHCP on the Pi-hole (Raspberry Pi) when the VM is back online.

👉 Save and exit the file (CTRL + X → Y → Enter).

✨ Step 2: Make the script executable

chmod +x /home/pi/dhcp-failover.sh

✨ Step 3: Set up a cron job for periodic checking

👉 Open the crontab:

crontab -e

👉 Add this line to check every 5 minutes:

*/5 * * * * /home/pi/dhcp-failover.sh >> /var/log/dhcp-failover.log 2>&1

✅ This logs the DHCP failover status to /var/log/dhcp-failover.log.

✨ Step 4: Test the failover

👉 Run the script manually to verify it works:

/home/pi/dhcp-failover.sh

👉 Then, simulate a failure by shutting down the Proxmox Pi-hole VM and checking if the Raspberry Pi takes over DHCP.

✅ Final result:

  • Primary Pi-hole VM on Proxmox runs DHCP under normal conditions.
  • Secondary Pi-hole (Raspberry Pi) automatically enables DHCP if the primary goes offline.
  • DHCP is disabled again when the primary comes back online — no IP conflicts, just smooth failover!

👉 Let me know if you want to add email alerts or system monitoring to this! 🚀

VPN Redux

Posted on by
Reply

Goodbye CGNAT! I limped along for over a year with T-Mobile 5G Home Internet for a miserly $30 per month. However my downloads dropped from double-digit mbps to single-digits and I cried uncle. I’m in a T-Mobile fringe area for LTE/5G signal and probably in an area with insufficient capacity for Home Internet. I’ve stepped up to AT&T fiber. As important to me, the change from Commercial Grade NAT to standard NAT means I’ll be able use Dynamic DNS to use my personal domain name to point to my home network. For the uninitiated, your CGNAT IP address changes so frequently that DDNS is useless. This means I can now set up a VPN tunnel into my network for remote control and file sharing. Woohoo!

To use DDNS with NameCheap, go to domain management, Dynamic DNS and create an A + Dynamiic DNS Record with host @ and your IP address, then save. This NameCheap section has the secret password needed for the DDNS client like ddclient for linux.

WireGuard seems to be the go-to VPN server and I installed it on a Proxmox VM using PiVPN. I’m convinced this is the simplest way to setup and manage a VPN. I found excellent WireGuard links on Mike Royal’s github page. Even better is a video from Crosstalk Solutions which adds how to add access to your local network. Next I created a port forwarding entry on my router pointing my WireGuard port to my WireGuard server. Next up is to install the app on my notebook computer to see if it works.

Update 12/30/24

I installed ddclient on my file server and it keeps the ip address lynnhargrove.site up-to-date. I also had a “duh” moment regarding IP reservations on my home network. I ran out of IP reservations on the AT&T router and struggled with servers changing addresses. Pi-hole has a very nice DCHP server that I switched to. I had to edit the dhcp.conf to add a secondary DNS server in case Pi-hole is down. Unfortunately, it reverts to the original dhcp.conf when changes are made to Pi-hole. I originally installed Pi-hole for quicker DNS resolution, but I hadn’t use it to blacklist any websites. I used Steven Black’s unified list of porn, gambling, fake news and social hosts for my blacklist.