Skip to content

Repository files navigation

CI/CD Status IaC Lint & Test Coverage Infrastructure Uptime Virtualization Zero-Trust Security Local AI License: MIT


Infrastructure platform with Proxmox VE virtualization on x86_64, enterprise firewall routing (OPNsense perimeter NGFW + Proxmox VE defense-in-depth), ZFS storage arrays, declarative Terraform/Ansible automation, and eBPF runtime observability.

Live Interactive Web Architecture ViewerArchitecture BlueprintCyber Forensics SuiteSecurity Policy

Active Workloads Automated Tests ELO Tools Last Sync


Table of Contents

  1. Mission & Design Principles
  2. End-to-End Architecture & Network Topology
  3. Physical Hardware Fleet & Power Delivery
  4. LXC Containers & VM Workloads Resource Matrix
  5. Storage Architecture & ZFS Pool Optimization
  6. Network Segmentation & Inter-VLAN Firewall Matrix
  7. Ingress Traffic, Zero-Trust Authentication & Split-Horizon DNS
  8. Infrastructure as Code (Terraform & Ansible)
  9. Kubernetes & GitOps Deployment Lifecycle
  10. LGTM Observability Stack & Telemetry Pipeline
  11. 3-2-1 Backup Strategy, Sanoid & Disaster Recovery
  12. Cybersecurity Test Environment, SOC & eBPF Security
  13. Local GPU AI LLM Runtime (Ollama CT 110)
  14. Chaos Engineering & Resiliency Validation
  15. Environmental Telemetry & Closed-Loop Fan Control
  16. Security Hardening & Cryptographic Integrity
  17. Static IP & Ports Directory
  18. Cold-Start Runbook & Operational Cheat Sheet
  19. Troubleshooting FAQ
  20. Monorepo Layout & Engineering Portfolio

1. Mission & Design Principles

flowchart LR
 subgraph Principles["HOMELAB ENGINEERING PRINCIPLES"]
 direction LR
 P1["RESOURCE EFFICIENCY<br/>• Minimal footprint via Alpine LXC<br/>• ZFS ZSTD & ZRAM lz4 compression<br/>• Sub-100ms GPU local LLM inference"]
 P2["DEFENSE-IN-DEPTH<br/>• OPNsense default-deny firewall<br/>• Kernel-level eBPF telemetry<br/>• DMZ Deception & FIDO2 Zero-Trust"]
 P3["GITOPS & AS-CODE<br/>• 100% declarative Terraform & Ansible<br/>• Zero manual click-ops<br/>• Instant rollback & CI security scans"]
 end
Loading
  • Resource Efficiency: High-density virtualization utilizing minimal CPU/RAM footprints. Alpine Linux and Debian slim containers maximize performance on constrained silicon.
  • Defense-in-Depth: Strict L2/L3 segmentation across 5 VLANs, CrowdSec real-time IP reputation bouncers, Suricata intrusion detection, and kernel-level Cilium Tetragon tracing.
  • Declarative GitOps: Every container, VM, firewall rule, dashboard, and secret is managed declaratively through version-controlled Terraform, Ansible, and Docker manifests.
  • High Availability & Fault Tolerance: Automated disaster recovery snapshots, virtual IP failover, cold-start runbooks, and UPS battery backup with controlled sequential shutdown.

2. End-to-End Architecture & Network Topology

flowchart TB
    subgraph WAN_Edge["Perimeter & External Ingress"]
        CF["Cloudflare WAF / CDN"] -->|Encrypted Tunnel| VPS["VPS WireGuard Gateway"]
        VPS -->|Dual-Homed VPN| OPN["OPNsense Firewall (VM 200)<br/>Suricata IDS/IPS · WireGuard · Unbound"]
    end

    subgraph Network_VLANs["Segmented Virtual Local Area Networks"]
        V10["VLAN 10: Management & Storage<br/>Proxmox VE · OMV NAS · IPMI"]
        V20["VLAN 20: Core Microservices<br/>NPM · Authentik · Vaultwarden · Nextcloud"]
        V30["VLAN 30: CyberLab & Sandboxes<br/>Wazuh SIEM · Atomic Red Team · CAPEv2"]
        V40["VLAN 40: DMZ Deception<br/>T-Pot Multi-Honeypots · AbuseIPDB"]
        V50["VLAN 50: IoT & Edge Sensors<br/>ESP32 mmWave · Zigbee · Home Assistant"]
    end

    subgraph Compute_Layer["Hybrid Multi-Node Virtualization Fleet"]
        Node1["Node 1: Proxmox Primary (x86_64)<br/>Intel Core i3-10100F · 12GB RAM<br/>NVIDIA GTX 1050 Ti GPU (Passthrough)"]
        Node2["Node 2: OMV NAS Storage<br/>ASUS Laptop · Celeron N2830 · 2GB RAM<br/>500GB ZFS Pool · Kiwix Wikipedia"]
        Node4["Node 4: Talos Linux Worker<br/>AMD Athlon II X2 · 4GB RAM<br/>k3s-agent · eBPF Tetragon Sensor"]
    end

    OPN -->|VLAN 10: 192.168.1.0/24| V10
    OPN -->|VLAN 20: 192.168.20.0/24| V20
    OPN -->|VLAN 30: 192.168.30.0/24| V30
    OPN -->|VLAN 40: 192.168.40.0/24| V40
    OPN -->|VLAN 50: 192.168.50.0/24| V50

    V10 -.-> Node1
    V10 -.-> Node2
    V10 -.-> Node4
    V20 -.-> Node1
    V30 -.-> Node1
    V40 -.-> Node1
    V50 -.-> Node1
Loading

2.3 OPNsense Enterprise Architecture (5 Security Pillars)

The perimeter firewall OPNsense (VM 200 · 192.168.1.134) implements a unified enterprise defense suite running in the FreeBSD kernel (pf):

flowchart TB
 subgraph OPN["OPNsense Enterprise Security Core (192.168.1.134)"]
 direction TB
 subgraph P1["1. Threat Intel & Perimeter"]
 SURI["Suricata NIDS/IPS (v8.0)<br/>• ET Open Rules & Promiscuous"]
 CS["CrowdSec LAPI Bouncer<br/>• Dynamic pf table drops"]
 GEO["GeoIP Kernel Drop<br/>• Ingress block for high-risk zones"]
 end
 subgraph P2["2. Observability & Health"]
 TELE["Telegraf Prometheus Exporter<br/>• pf state table telemetry (:9273)"]
 MONIT["Monit Self-Healing Watchdog<br/>• Daemon auto-restart & ntfy alerts"]
 end
 subgraph P3["3. GitOps & Disaster Recovery"]
 GIT["os-git-backup<br/>• GPG encrypted config.xml snapshots"]
 end
 subgraph P4["4. Privacy & DNS"]
 DOT["Unbound DNS-over-TLS<br/>• Quad9 (9.9.9.9:853) & DNSSEC"]
 DHCP["Kea DHCP Auto DynDNS<br/>• Auto registration of *.homelab.local"]
 end
 subgraph P5["5. Zero-Trust & Kubernetes"]
 BGP["FRRouting BGP Peering<br/>• MetalLB & Cilium LoadBalancers"]
 TS["Tailscale Subnet Router<br/>• Encrypted mesh across all VLANs"]
 end
 end
Loading

| Strategic Pillar | Technology & Module | Cluster Role & Functionality | Port / Protocol | Threat Intel | Suricata 8.0 + CrowdSec + GeoIP | Deep packet inspection, collaborative IP reputation, and GeoIP drop | WAN / VLAN Promisc | GitOps & DR | os-git-backup (GPG Encrypted) | Automatic Git versioning of config.xml on every administrative change | Git SSH Hook | Zero-Trust Mesh | FRRouting BGP + Tailscale Subnet | Dynamic K8s MetalLB routing and remote mesh access without open ports | :179 BGP / Mesh |

2.4 OPNsense 802.1Q VLAN Micro-Segmentation & Security Policies

The perimeter firewall OPNsense (VM 200 · 192.168.1.134) enforces zero-trust 802.1Q micro-segmentation across 5 isolated VLANs using strict Packet Filter (pf) rules:

OPNsense 802.1Q VLAN Micro-Segmentation

| VLAN ID | Network Segment | Subnet CIDR | Gateway | Attached Workloads | Security Policy | VLAN 10 | Management & Storage Subnet | 192.168.1.0/24 | 192.168.1.1 | Proxmox Core (x86_64), OMV NAS, Managed Switches | Isolated from IoT & Guest subnets | VLAN 30 | Cyber Security & Sandboxes (CyberLab) | 192.168.30.0/24 | 192.168.1.134:8443 | Wazuh XDR SIEM (1514), Suricata IDS, Atomic Red Team, CAPEv2 / Cuckoo Sandbox (Win10 + INetSim) | Promiscuous SPAN mirror port, no outbound WAN access for sandboxes | VLAN 50 | IoT & Physical Edge Devices | 192.168.50.0/24 | 192.168.1.134 (OPNsense) | ESP32 mmWave Radar, ESP32 Irrigation Relays, Zigbee Gateway | MQTT communication strictly restricted to Home Assistant (CT 106) |


3. Hybrid Multi-Cloud Architecture (Azure, GCP, AWS)

The on-premise cluster is extended into a true hybrid multi-cloud topology across Microsoft Azure, Google Cloud Platform (GCP), and Amazon Web Services (AWS) using declarative, modular Infrastructure as Code (IaC) located in cloud/ and terraform/:

flowchart TB
 subgraph OnPrem["ON-PREMISE HYBRID HOMELAB"]
 direction TB
 OPN["OPNsense Firewall (192.168.1.134:8443)<br/>Suricata IDS/IPS · WireGuard · Unbound"]
 PVE["Proxmox VE Primary (x86_64)<br/>ZRAM lz4 · Dynamic VirtIO Ballooning"]
 ZFS["ZFS Storage Mirror & Local Backups<br/>NFS / SMB Shares · PBS Target"]
 OPN --- PVE --- ZFS
 end

 subgraph Azure["MICROSOFT AZURE (cloud/azure/)"]
 direction TB
 AKV["Azure Key Vault (Cloud HSM)<br/>Step-CA Root CA & LUKS Escrow"]
 ABS["Blob Storage Archive Tier<br/>Cold ZFS Disaster Recovery"]
 EID["Entra ID SSO Federation<br/>Authentik SAML / OIDC"]
 ARC["Azure Arc Integration<br/>Defender for Cloud Security"]
 end

 subgraph GCP["GOOGLE CLOUD PLATFORM (cloud/gcp/)"]
 direction TB
 GCS["Cloud Storage Bucket WORM<br/>Ransomware-Proof PBS Retention"]
 WIF["Workload Identity Federation<br/>Keyless CI/CD (GitHub & Woodpecker)"]
 DNS["Cloud DNS Managed Zone<br/>Split-Horizon DNS Fallback"]
 BQ["BigQuery Security Sink<br/>T-Pot & Wazuh SIEM Analytics"]
 end

 subgraph AWS["AMAZON WEB SERVICES (cloud/aws/)"]
 direction TB
 S3["S3 Glacier Deep Archive<br/>Encrypted Off-Site Cold DR"]
 OIDC["IAM OIDC Keyless Role<br/>Least-Privilege AssumeRole"]
 VPN["Site-to-Site IPsec VPN<br/>Encrypted Tunnel to OPNsense"]
 end

 OnPrem -->|"IPsec / WireGuard VPN"| Azure
 OnPrem -->|"OIDC Token / HA VPN"| GCP
 OnPrem -->|"Glacier Sync / IPsec Tunnel"| AWS
Loading

Cloud Integration & Zero-Cost Tiering Matrix

| Cloud Provider | IaC Directory | Core Declarative Resources | Cost Optimization Tier | Microsoft Azure | cloud/azure/ | azurerm_key_vault (Cloud HSM Root CA & LUKS), azurerm_storage_blob (Archive Tier DR), azuread_application (SSO Authentik), azurerm_arc_machine (Defender for Cloud) | Archive Tier + Free Tier HSM | Amazon Web Services | cloud/aws/ | aws_s3_bucket (Glacier Deep Archive 365d), aws_iam_openid_connect_provider (Keyless CI/CD AssumeRole), aws_vpn_connection (Site-to-Site IPsec OPNsense) | Glacier Deep Archive + Free STS |


4. Enterprise CI/CD Quality Matrix (9 Automated Workflows)

Infrastructure and application code are validated continuously across 9 GitHub Actions CI/CD workflows running 36+ parallel automated quality gates:

| # | Workflow File | Pipeline Name | Automated Quality Guarantees & Checks | 1 | .github/workflows/homelab-ci-cd-matrix.yml | Enterprise Quality Matrix | terraform fmt & validate (on-prem + multi-cloud), Checkov IaC Security, Trivy Misconfig, Docker Compose validation, ShellCheck, Secret Leakage, ELO Matrix (Python 3.9-3.13) | 3 | .github/workflows/cd.yml | Continuous Deployment | GitOps Reconciliation, Container Image Packaging on GHCR, Automated Rollback Verification | 5 | .github/workflows/security-scan.yml | CodeQL SAST Analysis | GitHub Advanced Security CodeQL engine for deep static vulnerability scanning (Python & TypeScript) | 7 | .github/workflows/deploy-pages.yml | Deploy GitHub Pages | Angular 19 production build & zero-downtime deployment to GitHub Pages | 9 | .github/workflows/readme-sync.yml | Documentation Sync | Automated documentation sync and badge validation across all 5 supported languages |


9. Physical Hardware Fleet & Power Delivery

Hardware Specifications Matrix

| Node Identifier | Form Factor / Chassis | CPU Architecture | Accelerator / GPU | RAM Allocation | Storage Configuration | Primary Purpose | pve (Node 1) | Custom ATX Tower | Intel Core i3-10100F (4C/8T @ 4.30 GHz) | NVIDIA GeForce GTX 1050 Ti (4GB VRAM) | 12 GB DDR4-2133 (12,288 MB) | 512 GB NVMe SSD (local-lvm) | Primary Hypervisor: Windows Server 2025 Datacenter AD, OPNsense, Ollama GPU (CT 110), Immich AI | kubernetes (Node 4) | Custom ATX Chassis | AMD Athlon II X2 220 (2C/2T @ 2.80 GHz) | NVIDIA GeForce GTS 250 (1GB) | 4 GB DDR3-1333 | 80 GB HDD (NFS Root) | Immutable Talos Linux / k3s worker, batch cron workloads, eBPF security probing |

Power Delivery & NUT Controlled Shutdown Sequence

flowchart TD
 Mains["Mains Utility Power 230V AC"] --> UPS["Coldex Pure Sine Wave 1200VA UPS<br/>+ External 100Ah Deep-Cycle Battery"]
 UPS --> PDU["Smart Energy Metered PDU"]
 PDU --> Node1 & Node2 & Node4 & Switch["Managed PoE+ Switch"]

 UPS -.->|"USB HID Telemetry"| NUT_Master["NUT Server (Network UPS Tools)<br/>Node 1 (192.168.1.132)"]
 NUT_Master -->|"Power Outage Event"| Timer{"On Battery > 15 Mins OR<br/>Battery Charge < 25%"}
 
 Timer -->|"YES"| Graceful_Shutdown["Controlled Sequential Shutdown Sequence"]
 Graceful_Shutdown --> S1["1. Stop Non-Critical LXCs (Media, Nextcloud)"]
 S1 --> S2["2. Stop Core Databases & Storage (PostgreSQL, OMV)"]
 S2 --> S3["3. Gracefully Stop VMs (Windows Server, OPNsense)"]
 S3 --> S4["4. Proxmox VE Host Poweroff via 'poweroff'"]
Loading

10. LXC Containers & VM Workloads Resource Matrix

Granular LXC Container Roster (Node 1 — x86_64 Primary: CT 100 - CT 174)

| VMID | Hostname | Base OS | vCPU | RAM Allocation | Storage Pool | Static IP | Subsystem Category | Primary Workload | 100 | immich | Alpine 3.24 | 2 | 256 MB | local-lvm:40G | 192.168.1.15 | Storage / AI | Photo Library + Machine Learning Face Recognition | 102 | homeassistant | Alpine 3.24 | 2 | 128 MB | local-lvm:16G | 192.168.1.10 | Automation | Smart Home Hub, Zigbee & ESP32 Telemetry | 104 | scrutiny | Alpine 3.24 | 1 | 96 MB | local-lvm:3G | 192.168.1.18 | Monitoring | Scrutiny S.M.A.R.T. Drive Health Agent | 106 | ollama | Debian 13 | 4 | 2048 MB | local-lvm:16G | 192.168.1.110 | Local AI | Ollama GPU LLM Runtime (Qwen2.5-Coder & DeepSeek-R1) | 108 | whisper | Debian 13 | 2 | 1024 MB | local-lvm:8G | 192.168.1.112 | Local AI | Faster-Whisper Speech-to-Text CUDA API | 110 | paperless-ai | Alpine 3.24 | 1 | 64 MB | local-lvm:1G | 192.168.1.56 | Local AI | Paperless-AI Automated OCR & DeepSeek Document Tagging | 112 | proxmox-backup-server | Alpine 3.24 | 2 | 512 MB | local-lvm:4G | 192.168.1.116 | Storage / Backup | Proxmox Backup Server (PBS Enterprise Deduplication & Verification) | 114 | woodpecker-k0s | Alpine 3.24 | 2 | 512 MB | local-lvm:8G | 192.168.1.118 | CI/CD | Woodpecker CI Server & Runner on Alpine Linux backed by k0s Kubernetes Engine | 116 | actualbudget | Alpine 3.24 | 1 | 256 MB | local-lvm:4G | 192.168.1.116 | Finance / Budgeting | Zero-based personal budgeting application with real-time transaction tracking. | 118 | changedetection | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.118 | Monitoring / Web-Watch | Monitors targeted web pages and APIs for structural changes and triggers alerts. | 120 | vaultwarden | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.120 | Security / Vault / Passwords | Self-hosted zero-knowledge password vault providing cross-device synchronization. | 122 | authelia | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.122 | Security / Sso / 2Fa | Identity provider enforcing two-factor authentication and single sign-on (SSO). | 124 | gatus | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.124 | Monitoring / Status | Health dashboard actively probing HTTP endpoints and TLS certificates. | 126 | linkding | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.126 | Bookmarks / Search | Fast bookmark manager with automatic title scraping and tag indexing. | 128 | beszel | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.128 | Monitoring / Metrics | Aggregates microsecond-resolution system resource metrics across the fleet. | 130 | homepage | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.130 | Dashboard / Portal | Centralized service portal and dashboard displaying real-time server health. | 132 | memos | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.132 | Notes / Microblog | Privacy-first micro-note platform for instant thought capturing and journaling. | 134 | syncthing | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.134 | Storage / Sync | Continuous file synchronization service replicating document folders securely. | 136 | vikunja | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.136 | Productivity / Kanban / Tasks | Collaborative project and task management application with Kanban boards. | 138 | yourspotify | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.138 | Analytics / Music | Self-hosted music analytics platform recording personal Spotify history. | 140 | opengist | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.140 | Pastebin / Git / Snippets | Self-hosted pastebin and code snippet repository powered by Git version control. | 142 | whoogle | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.142 | Search / Privacy | Privacy-preserving search gateway proxying Google search queries anonymously. | 144 | pingvin-share | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.144 | File-Share / Privacy | Privacy-focused file sharing platform enabling secure link-based transfers. | 146 | excalidraw | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.146 | Diagrams / Whiteboard | Collaborative whiteboard drawing tool for sketching system architectures. | 148 | transmission | Alpine 3.24 | 1 | 256 MB | local-lvm:4G | 192.168.1.148 | Media / Bittorrent | Isolated BitTorrent download gateway and management interface. | 150 | stirling-pdf | Alpine 3.24 | 1 | 256 MB | local-lvm:2G | 192.168.1.150 | Documents / Pdf | Powerful local web application for performing PDF merging, splitting, and OCR. | 152 | tubearchivist | Alpine 3.24 | 1 | 256 MB | local-lvm:4G | 192.168.1.152 | Media / Youtube / Archive | Self-hosted YouTube media archiver with semantic indexing. | 154 | cyberchef | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.154 | Cyber / Tools / Crypto | Web app for encryption, encoding, compression, and data analysis. | 156 | romm | Alpine 3.24 | 1 | 256 MB | local-lvm:4G | 192.168.1.156 | Gaming / Retro / Roms | Retro gaming ROM manager with metadata enrichment and game covers. | 158 | paperless-ngx | Alpine 3.24 | 1 | 256 MB | local-lvm:4G | 192.168.1.158 | Documents / Dms / Ocr | Document management system transforming physical documents into searchable archives. | 160 | meilisearch | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.160 | Search / Indexing | Lightning-fast, hyper-relevant search engine for documents and logs. | 162 | searxng | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.162 | Search / Metasearch | Privacy-respecting, hackable metasearch engine aggregating 70+ search engines. | 164 | rustdesk | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.164 | Remote-Desktop / Relay | Open-source virtual/remote desktop infrastructure and signal relay. | 166 | wg-easy | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.166 | Vpn / Wireguard / Gui | WireGuard VPN management dashboard with automated QR client profiles. | 168 | dozzle | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.168 | Logging / Containers | Real-time log viewer for Docker and Podman container workloads. | 170 | hedgedoc | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.170 | Collaboration / Markdown | Collaborative markdown editor for real-time document authoring. | 172 | gotify | Alpine 3.24 | 1 | 128 MB | local-lvm:1G | 192.168.1.172 | Notifications / Push | Simple server for sending and receiving push notifications over WebSockets. | 174 | grocy | Alpine 3.24 | 1 | 128 MB | local-lvm:2G | 192.168.1.174 | Inventory / Groceries / Erp | Enterprise resource planning (ERP) system for household grocery and pantry tracking. |

Kubernetes Cloud-Native Platform & OpenStack Private Cloud

| Platform Component | Technology & Distribution | Node / Host Target | Port / Exposure | Primary Capability | ArgoCD GitOps | ArgoCD v2.12.3 Operator | Hybrid Cluster (Node 1 & Node 4) | :8080 (HTTPS) | Declarative continuous delivery, auto-sync and self-healing directly from Git repository | Cilium eBPF CNI | Cilium v1.16.1 eBPF Engine | Kernel-space (kube-system) | :9962 / :12000 (Hubble) | High-performance CNI replacing kube-proxy, WireGuard transparent encryption & L3-L7 security | Twingate ZTNA | Twingate Connector v1 | Remote Access (twingate) | Internal P2P Mesh | Enterprise Zero-Trust Network Access for secure remote operations without inbound firewall holes | OpenStack Cloud | OpenStack 2024.1 Caracal (Kolla) | Node 1 (VM 205 · QEMU KVM) | :80 / :5000 (Keystone) | Enterprise IaaS private cloud virtualization (Nova, Neutron, Keystone, Glance, Horizon Dashboard) |

QEMU / KVM Virtual Machines & VirtIO Memory Ballooning

| VMID | VM Name | Operating System | vCPU | RAM Max | Balloon Min | Passthrough / Hardware | Primary Role | 200 | opnsense | Hardened FreeBSD 14 | 4 Cores | 4,096 MB | 2,048 MB | VirtIO Net Multi-VLAN | Perimeter Firewall, Zenarmor NGFW (L7 Shun-Tuned), AdGuard Home + Unbound Split-DNS (:5335), FQ_CoDel Traffic Shaper, CrowdSec IPS + Threat Feeds, FRR BGP/OSPF, LLDP Discovery, iperf3, Encrypted Git/Nextcloud Backup | 202 | rhel | RHEL 9.8 Enterprise | 2 Cores | 2,048 MB (2 GB) | 1,024 MB (1 GB) | VirtIO SCSI Single IOThread | SELinux Enforcing, Podman Rootless, Enterprise Workload (1-2 GB) | 204 | nixos | NixOS 24.11 Minimal | 2 Cores | 1,024 MB (1 GB) | 512 MB | VirtIO SCSI Single (22 GB) | Declarative Immutable Linux, Flakes Reproducible Builds, Atomic Rollbacks | 206 | metasploitable2 | Metasploitable 2 (Ubuntu 8.04) | 1 Core | 512 MB | 512 MB | VirtIO Net + IDE (8 GB) | Intentionally Vulnerable Linux Target, Penetration Testing & IDS/IPS Tuning | 208 | securityonion | Security Onion 3.2 / Wazuh SIEM | 4 Cores | 8,192 MB (8 GB) | 4,096 MB (4 GB) | VirtIO Net + SCSI (50 GB) | Enterprise SIEM, HIDS, Log Analysis, Network Security Monitoring (Zeek, Suricata, Elastic, Kibana)

Consolidated Enterprise Virtualization on Node 1: All microservices and utility containers (CT 100–174) are unified on Node 1 (x86_64). Active enterprise VMs (VM 200–209) leverage VirtIO dynamic memory ballooning, while consolidated containers (CT 115–174) are configured with onboot: 0 for zero-overhead on-demand activation without consuming baseline RAM.

Host Memory Tuning: ZRAM / ZSWAP Fast RAM Compression

  • Compression Algorithm: Ultra-fast lz4 with < 1% CPU overhead.
  • Node 1 (x86_64) ZRAM: /dev/zram0 (6.0 GB RAM compressed swap, priority 100, vm.swappiness = 60, vm.vfs_cache_pressure = 50).
  • NVMe Lifespan Protection: High-frequency memory pages are compressed directly in RAM before touching NVMe storage, eliminating SSD wear and IO blocking.

Zero-Trust Security & Enterprise Test Environment

  1. HashiCorp Vault / OpenBao:
  • Centralized secret management with zero .env files stored on local disks.
  • Automated dynamic token generation and ephemeral credential injection for Terraform, Ansible, and Woodpecker CI.
  1. WireGuard Kernel Module on OPNsense with Automated Key Rotation:
  • Zero-downtime periodic rotation of Curve25519 cryptographic keypairs and pre-shared keys (PSK) via Ansible and cron.
  1. Mutual TLS (mTLS) Inter-Service Communication:
  • Mandatory cryptographic client-certificate verification between ingress gateways and critical backend services in VLAN 20.
  1. Canary Honeytokens & Directory Decoys:
  • Deceptive decoy files (passwords.csv, aws_keys.env, id_rsa_backup) placed in DMZ containers and SMB shares that trigger instant Telegram/ntfy webhooks upon access.
  1. RenovateBot On-Premise GitOps Automation:
  • Continuous dependency scanning engine inspecting internal Gitea repositories and filing automated Pull Requests for new Docker images and Terraform modules.

9. Storage Architecture & ZFS Pool Optimization

flowchart TD
 subgraph ZFSTopology["ZFS STORAGE POOL TOPOLOGY"]
 direction TB
 Pools["ZFS Storage Architecture"]
 
 subgraph RPool["rpool (NVMe SSD · Proxmox Root & OS)"]
 R1["• recordsize: 128k"]
 R2["• compression: zstd-3"]
 R3["• atime: off · autotrim: on"]
 end

 subgraph DataPool["datapool (ZFS Mirror · OpenMediaVault)"]
 D1["• recordsize: 1M (Media Streams)"]
 D2["• recordsize: 16k (Databases)"]
 D3["• compression: zstd · ashift: 12"]
 end

 Pools --> RPool
 Pools --> DataPool
 end
Loading

Granular ZFS Filesystem Tuning Rules

  • PostgreSQL / MySQL / SQLite Data: recordsize=16k matching DB page sizes to eliminate write amplification.
  • Large Media Streams (Jellyfin / Kiwix): recordsize=1M for sequential streaming throughput.
  • Compressratio: compression=zstd delivering ~1.85x space efficiency with zero noticeable CPU latency.
  • ZFS ARC Ceiling: Capped dynamically via /etc/modprobe.d/zfs.conf (zfs_arc_max=2147483648 — 2GB) to protect VM allocations.

10. Network Segmentation & Inter-VLAN Firewall Matrix

flowchart LR
 VLAN10["VLAN 10: Mgmt & Storage<br/>192.168.1.0/24"]
 VLAN20["VLAN 20: Core Microservices<br/>192.168.20.0/24"]
 VLAN30["VLAN 30: CyberLab & Sandboxes<br/>192.168.30.0/24"]
 VLAN40["VLAN 40: DMZ Deception<br/>192.168.40.0/24"]
 VLAN50["VLAN 50: IoT Sensors<br/>192.168.50.0/24"]

 VLAN10 -->|"Full Admin Access"| VLAN20 & VLAN30 & VLAN40 & VLAN50
 VLAN20 -->|"Restricted Ports: 53, 443"| VLAN10
 VLAN30 -->|"NO OUTBOUND WAN / Isolated"| VLAN10 & VLAN20
 VLAN40 -->|"DROP ALL Traffic to LAN"| VLAN10 & VLAN20 & VLAN30
 VLAN50 -->|"MQTT Only :1883"| VLAN20
Loading

Inter-VLAN Firewall Policy Table (Default-Deny)

| Source VLAN | Destination VLAN | Allowed Destination Ports | Protocol | Firewall Action | VLAN 10 (Management) | ALL VLANs | ANY | ANY | PASS (Stateful) | VLAN 20 (Core Services) | VLAN 50 (IoT) | 1883 (MQTT Broker) | TCP | PASS | VLAN 30 (CyberLab) | WAN | HTTP :8080 via INetSim Fake Gateway | TCP | PASS (Simulated) | VLAN 50 (IoT) | ANY Internal VLAN | 1883 (Home Assistant MQTT Only) | TCP | PASS

9. Ingress Traffic, Zero-Trust Authentication & Split-Horizon DNS

Ingress Forward-Authentication Sequence

sequenceDiagram
 autonumber
 actor Client as External User
 participant CF as Cloudflare WAF
 participant NPM as OPNsense Nginx Ingress (VM 200)
 participant Auth as Authentik / Authelia (CT 108)
 participant Srv as Target Internal Service

 Client->>CF: HTTPS Request (https://service.homelab.local)
 CF->>NPM: Forwarded through WireGuard Tunnel
 NPM->>Auth: Auth Subrequest (/api/verify)
 alt No Valid Session Cookie
 Auth-->>NPM: 401 Unauthorized / Redirect to Login
 NPM-->>Client: 302 Redirect to Authentik Login
 Client->>Auth: Authenticate with FIDO2 WebAuthn / Passkey
 Auth-->>Client: Set Secure HttpOnly Session Token
 else Valid Session & 2FA Confirmed
 Auth-->>NPM: 200 OK + Remote-User Headers
 NPM->>Srv: Reverse Proxy HTTP Request
 Srv-->>NPM: 200 Response Payload
 NPM-->>Client: Protected Application Data
 end
Loading

Split-Horizon DNS Schema

  • External Resolution: Public domain records hosted on Cloudflare DNS point exclusively to protected VPS reverse proxy endpoints.
  • Internal Resolution: OPNsense Unbound DNS and AdGuard Home sinkholes resolve *.homelab.local directly to internal RFC1918 IPs (192.168.1.134), bypassing external bandwidth entirely.

10. Infrastructure as Code (Terraform & Ansible)

All infrastructure is provisioned declaratively using Terraform with the bpg/proxmox provider.

terraform/
├── main.tf                    # Root composition
├── providers.tf               # Proxmox VE provider & Remote S3 backend
├── backend-config.hcl.example # Remote MinIO S3 backend template
├── variables.tf               # Cluster endpoints & credentials
├── terraform.tfvars.example   # Template variables
├── lxc_services.tf            # Declarative LXC container definitions
├── vm_workloads.tf            # Declarative VM definitions
└── modules/
    ├── proxmox_lxc/           # Reusable LXC container module
    └── proxmox_vm/            # Reusable QEMU VM module

Centralized & Encrypted Remote State with DynamoDB Locking

To prevent race conditions during concurrent CI/CD executions and guarantee enterprise reproducibility, Terraform state is stored on an internal MinIO S3 bucket (CT 161) with AES-256 encryption and state locking:

terraform {
  backend "s3" {
    bucket                      = "terraform-state"
    key                         = "infrastructure/terraform.tfstate"
    region                      = "us-east-1"
    endpoint                    = "http://192.168.1.161:9000" # MinIO CT 161
    dynamodb_endpoint           = "http://192.168.1.161:9000" # Lock table
    dynamodb_table              = "terraform-locks"
    encrypt                     = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    use_path_style              = true
  }
}

Policy-as-Code Guardrails (OPA & Conftest)

All Terraform declarations and Kubernetes manifests undergo mandatory pre-flight policy evaluation via Open Policy Agent (conftest):

  • Rootless Containment: Blocs any workload with runAsNonRoot: false or runAsUser: 0 (policy/kubernetes/security.rego).
  • Immutable Version Pinning: Forbids mutable tags (:latest) or untagged images.
  • Network Isolation: Prohibits unauthorized hostPort, hostNetwork: true, or host namespace leaks.

Quick Bootstrap Runbook

# 1. Clone repository
git clone https://github.com/stefanutc1/infrastructure.git
cd infrastructure/terraform

# 2. Initialize with remote backend
terraform init -backend-config=backend-config.hcl

# 3. Plan & Apply
terraform plan -out=tfplan.binary
terraform apply tfplan.binary

9. Kubernetes & GitOps Deployment Lifecycle

flowchart LR
    Dev["Engineer Commit"] -->|"Push to main"| GH["GitHub Repository"]
    GH -->|"Trigger"| CI["GitHub Actions / Conftest OPA<br/>Trivy · Gitleaks · Hadolint · OPA"]
    CI -->|"Pass Quality Gates"| Argo["ArgoCD / Flux GitOps Operator"]
    Argo -->|"Continuous Reconciliation"| K3s["Talos Linux / K3s Cluster"]
    K3s -->|"Strict mTLS L7"| Workloads["Distroless Microservices & Agents"]
Loading
  • Talos Linux OS (kubernetes/talos/cluster.yaml): Immutable, zero-SSH operating system managed strictly via gRPC APIs.
  • Cilium eBPF CNI & Strict mTLS Service Mesh:
    • Seamless SPIFFE/SPIRE mutual TLS authentication enforced on all inter-workload traffic (kubernetes/apps/cilium/cilium-strict-mtls-vlan20.yaml).
    • Enforces authentication.mode: required between Talos pods and critical VLAN 20 microservices (NPM, Authentik, Vaultwarden), completely eliminating cleartext inter-container communication.

10. LGTM Observability Stack & Telemetry Pipeline

flowchart TD
 subgraph Collectors["Telemetry Collectors"]
 NE["node_exporter (Host Hardware)"]
 PE["postgres_exporter"]
 RE["redis_exporter"]
 VB["Vector Log Forwarder"]
 OTEL["OpenTelemetry SDKs"]
 end

 subgraph LGTM_Backend["LGTM Backend Storing & Indexing"]
 PROM["Prometheus TSDB (:9090)<br/>High-resolution metrics"]
 LOKI["Grafana Loki (:3100)<br/>Indexed log streams"]
 TEMPO["Grafana Tempo (:3200)<br/>OTLP distributed traces"]
 end

 subgraph Visualization["Unified Glass Pane"]
 GRAF["Grafana Core Dashboard (:3000)"]
 ALARM["Alertmanager Routing"]
 end

 NE & PE & RE --> PROM
 VB --> LOKI
 OTEL --> TEMPO

 PROM & LOKI & TEMPO --> GRAF
 PROM --> ALARM
 ALARM -->|"High Severity Alert"| TG["Telegram / Discord Webhook Channel"]
Loading

11. 3-2-1 Backup Strategy, Sanoid & Disaster Recovery

flowchart LR
 PVE["Proxmox VE (Node 1 NVMe)"] -->|"Hourly ZFS Snapshots"| Sanoid["Sanoid / Syncoid Policy Engine"]
 Sanoid -->|"Daily vzdump Archive"| PBS["OpenMediaVault NAS (Node 2 ZFS Mirror)"]
 PBS -->|"Nightly Encrypted Restic/Rclone"| R2["Off-Site Cloudflare R2 / AWS S3 Glacier"]
Loading
  • 3 Copies: Primary NVMe SSD, Secondary OMV NAS ZFS Mirror, Remote Cloudflare R2 Bucket.
  • 2 Formats: Live ZFS Snapshots + compressed zstd .vma.zst archives.
  • 1 Off-Site: Encrypted, immutable cloud backup with 90-day object lock.
  • Automated DR Verification (scripts/disaster-recovery/dr_vzdump_restore.sh): Weekly CI script restores the newest snapshot into an isolated test VLAN 99, tests DB consistency and HTTP 200 endpoints, and reports results to Telegram.

12. Digital Forensics, Cyber Defense Proving Ground & Threat Intelligence

The Datacenter operates an integrated Security Operations Center (SOC), automated deception mesh, and four in-depth real-world digital forensics investigations hosted directly within cyber/.

flowchart TD
    subgraph External_Threats["External Threat Landscape & Investigations"]
        Attacker["Threat Actor / Scanning Botnets"]
        ScamNet["Task Scam & Phishing Networks<br/>(BitM, Vishing, MRR Fraud)"]
    end

    subgraph Perimeter_Defense["Perimeter Defense & Deception"]
        TPot["T-Pot DMZ Multi-Honeypot (VM 207)<br/>Cowrie · Dionaea · Honeytrap · Suricata"]
        OPN["OPNsense Firewall (VM 200)<br/>CrowdSec Bouncer · DoT · Zenarmor L7"]
        PVEFW["Proxmox VE Host Firewall & eBPF<br/>Rate Limiting · SYN-Flood Guards · IPset"]
    end

    subgraph Detection_Analysis["Detection, SIEM & Reverse Engineering"]
        Wazuh["Wazuh SIEM / XDR Manager (CT 100)<br/>FIM · Log Anomaly Correlation"]
        SecOnion["Security Onion Grid (VM 208)<br/>Zeek Network Metadata · Arkime PCAP"]
        Remnux["REMnux Malware Analysis (VM 209)<br/>Static/Dynamic Deobfuscation · Ghidra"]
        Tetra["Cilium Tetragon eBPF Runtime Sensor<br/>Syscall Monitoring (execve, openat, tcp_connect)"]
    end

    subgraph Autonomous_SOAR["Automated SOAR Playbooks & Containment"]
        SOAR["SOAR Playbook Engine (Shuffle / n8n)"]
        FirewallAPI["Automated Drop Rule & IPS Banning"]
        ThreatFeed["AbuseIPDB & Threat Intel Sync"]
    end

    Attacker -->|Probes Port 22, 445, 3389, 5060| TPot
    Attacker -->|Perimeter Ingress| OPN
    OPN -->|Defense-in-Depth Inspection| PVEFW
    ScamNet -.->|Case Studies & IoCs| Remnux

    TPot -->|Decoy Telemetry| Wazuh
    Tetra -->|Kernel Anomaly Events| Wazuh
    PVEFW -->|NetFlow / Syslog| SecOnion
    OPN -->|Alerts| Wazuh

    Wazuh -->|Correlated High-Severity Alert| SOAR
    SOAR -->|1. Inject IPset / Null-Route| FirewallAPI
    SOAR -->|2. Report Malicious Source| ThreatFeed
Loading

12.1 Real-World Digital Forensics & Threat Investigations (cyber/)

The cyber/ directory contains four end-to-end investigative case studies into active cybercrime campaigns, reverse engineered using the Datacenter's forensic sandbox tooling:

  • Threat Classification: Cybercrime Infrastructure, Leaky REST APIs, Crypto Money Laundering.
  • Incident Summary: In-depth anatomical breakdown of fraudulent "task-farming" platforms operated by organized cybercrime syndicates targeting mobile users through Telegram funnels.
  • Technical Exploitation & Findings:
    • Exposed Backend APIs: Reverse engineering unauthenticated administrative endpoints (/api/v1/user/task, /api/admin/recharge) that leaked internal server architecture, agent referral trees, and database schemas.
    • SQL Injection (SQLi) Discovery: Identified severe vulnerabilities in backend transaction endpoints allowing full parameter extraction and administrative bypass.
    • Cryptocurrency Money Laundering Flow: Traced illicit USDT deposits on the TRC-20 (Tron) network across multi-hop mixing structures into centralized exchange consolidation hot wallets.
    • UI Manipulation: Documented client-side JavaScript trickery that dynamically fabricated fake trading balances and simulated VIP commission payouts.
  • Repository Deliverables: Full case study, API exposure audit, SQLi proof of concept, OSINT infrastructure mapping, and Docker Compose test fixture.
  • Threat Classification: Voice Phishing (Vishing), International SIP Telephony Fraud, 3D Secure Bypass.
  • Incident Summary: Complete forensic reconstruction of an advanced social engineering phone attack where threat actors spoofed official European banking support numbers to intercept real-time SMS one-time passwords (OTP).
  • Technical Exploitation & Findings:
    • Caller ID Spoofing via International SIP Trunks: Dissected how rogue VoIP softswitches manipulate the SIP From: and P-Asserted-Identity headers to present legitimate bank caller IDs on victim smartphones.
    • Real-Time 3DS Intercept: Documented step-by-step social engineering call flows engineered to induce panic, prompting victims to authorize pending credit card charges while believing they were canceling a fraud event.
    • PCAP & Call Flow Analysis: Extracted session initiation protocol packets (INVITE, 180 Ringing, 200 OK, ACK, BYE), analyzed SDP audio codec negotiation (G.711u / PCMU), and mapped caller User-Agent signatures.
    • Carrier Traceback & Takedown: Outlined the administrative and telecommunication subpoena processes used to isolate upstream rogue carriers.
  • Repository Deliverables: Complete incident timeline, technical analysis, carrier response and takedown documentation, formal regulatory report, and simulated VoIP call-flow reproduction lab.
  • Threat Classification: Monthly Recurring Revenue (MRR) Deception, Viral Social Funnels, Payment Gateway Abuse.
  • Incident Summary: Investigation of automated social media ad networks promoting misleading software utilities and "AI tools" that enroll unsuspecting users into predatory recurring weekly and monthly subscription charges.
  • Technical Exploitation & Findings:
    • Bot Cloaking & User Fingerprinting: Unpacked client-side JavaScript fingerprinting scripts designed to detect and serve benign, compliant pages to TikTok/Meta ad review crawlers while serving predatory landing pages to organic mobile users.
    • Dark Pattern Payment Redirection: Analyzed multi-stage redirect chains masking merchant category codes (MCCs) to bypass payment processor risk scoring.
    • Chargeback Avoidance Tactics: Identified how scammers artificially delay initial recurring billing cycles to exceed consumer dispute windows and maintain merchant acquiring accounts.
  • Repository Deliverables: Comprehensive prevention guide, payment gateway abuse analysis, funnel traffic breakdown, LLM course synthesis, and architectural case study.
  • Threat Classification: Browser-in-the-Middle (BitM / AitM), Steam OpenID 2.0 Credential & Session Theft.
  • Incident Summary: Dissection of an aggressive phishing campaign targeting gaming accounts by rendering a simulated, interactive pop-up browser window within the active HTML DOM.
  • Technical Exploitation & Findings:
    • Synthetic Browser Canvas: Attackers drew an entirely fake, draggable Chrome browser window (complete with custom minimize/maximize controls, URL address bar, and spoofed green SSL padlock) entirely in HTML5/CSS, rendering standard URL inspection useless.
    • Real-Time Session Relaying: Intercepted OpenID 2.0 authentication exchanges, harvesting steamLoginSecure cookies and session tokens while automatically passing SteamGuard mobile 2FA challenges.
    • Automated Family View Lockout: Captured sessions were immediately automated via headless scripts to activate Steam Family View with an attacker-selected PIN, preventing legitimate account recovery.
  • Repository Deliverables: Full executive summary, technical reverse-engineering report, deobfuscated payload source, Suricata IDS detection signatures, and interactive HTML5 demonstration lab.

12.2 Proving Ground Defense Stack & Threat Intelligence Feeding

Findings from these four forensic investigations directly inform the proactive defense configurations across the Datacenter:

| Security Layer | Host / Virtual Machine | Engine & Role | Defensive Functionality | Perimeter IDS/IPS | VM 200 (OPNsense) | Suricata 8.0.3 + CrowdSec | Drops active BitM synthetic popup URLs and blocks malicious IP lists via threat feeds. | Deception Honeynet| VM 207 (T-Pot) | Cowrie, Dionaea, Honeytrap | Exposes decoy honeypots in isolated DMZ (vmbr3) to harvest live scanner payloads. | Enterprise SIEM/XDR| CT 100 (Wazuh) | Wazuh Manager + Elastic Stack | Centralized syslog/FIM correlation across all 95 services with automated active response. | Host Zero-Trust FW | Node 1 (192.168.1.132)| Proxmox VE Cluster Firewall | Global DROP policy, rate-limited ICMP, SYN-flood guards, IPset bastion access control.

12.3 Security Auditing & Detection Tests (cyber/red-team/)

Verification scripts for container isolation and detection rules:

  • Container Audit (cyber/red-team/container_audit.py):
    • Audits Linux capabilities (CAP_SYS_ADMIN, CAP_SYS_PTRACE, CAP_SYS_MODULE, CAP_DAC_OVERRIDE).
    • Scans for mounted Docker/containerd UNIX control sockets (/var/run/docker.sock).
    • Validates cgroup isolation (release_agent), host namespace leakage (hostPID, hostNetwork), and Seccomp/AppArmor enforcement.
  • Security Detection Tests (cyber/red-team/sec_tests.py):
    • Runs controlled validation checks (T1059.004 Unix Shell, T1082 System Discovery, T1046 Network Service Discovery, T1552 Canary Token Search).
    • Validates alert ingestion in Wazuh SIEM (Rule 80710) and CrowdSec portscan decisions.
  • Privilege Audit (cyber/red-team/priv_check.py):
    • Evaluates writable system PATH directories, verifies private key permissions, and scans environment variables for plaintext secrets.

13. Local GPU AI LLM Runtime (Ollama CT 110)

Ollama is running inside container CT 110 on Proxmox Node 1 (192.168.1.110:11434), utilizing direct NVIDIA GeForce GTX 1050 Ti GPU acceleration:

# Verify active models inside CT 110
pct exec 110 -- ollama list

# Output:
# NAME ID SIZE MODIFIED 
# llama3.2:1b baf6a787fdff 1.3 GB Active 
# qwen2.5-coder:1.5b d7372fd82851 986 MB Active 

# Execute instant test query via REST API:
curl -s http://192.168.1.110:11434/api/generate -d '{"model": "qwen2.5-coder:1.5b",
 "prompt": "Write a Python script to check Proxmox container status",
 "stream": false
}'

14. Chaos Engineering & Resiliency Validation

Automated continuous resiliency testing is enforced both locally and via a dedicated CI/CD pipeline (.github/workflows/chaos-scheduled.yml) scheduled via cron 0 3 * * 0 (Sunday nights at 03:00 UTC).

# 1. Inject 100% CPU stress & 80% RAM pressure
./scripts/chaos/chaos_runner.sh cpu-stress 30
./scripts/chaos/chaos_runner.sh ram-pressure 30

# 2. Inject service fault (SIGKILL) & verify automated self-healing without human intervention
./scripts/chaos/chaos_runner.sh service-kill 5 staging-workload
./scripts/chaos/chaos_runner.sh auto-healing-check 30 staging-workload

# 3. Simulate network latency (150ms) and packet loss (15%) via kernel netem
./scripts/chaos/chaos_runner.sh network-latency 30 eth0 150ms
./scripts/chaos/chaos_runner.sh packet-loss 30 eth0 15%

# 4. Validate incident alerting pipeline across Uptime Kuma, Ntfy, and Telegram
./scripts/chaos/chaos_runner.sh alert-webhook-validate

15. Environmental Telemetry & Closed-Loop Fan Control

flowchart LR
 ESP["ESP32 Sensors<br/>DHT22 Temp · mmWave Radar"] -->|"MQTT :1883"| HA["Home Assistant (CT 106)"]
 HA -->|"Thermal Threshold Calculation"| PWM["ESP32 PWM Fan Controller"]
 PWM -->|"Dynamic Duty Cycle: 20% - 100%"| Fans["Noctua 120mm Server Rack Cooling"]
Loading
  • Rack Tamper Monitoring: Optical microswitch on server chassis logs physical cabinet door state; triggers snapshot on security cameras if opened unexpectedly.

16. Security Hardening & Cryptographic Integrity

  • Linux Kernel Hardening (/etc/sysctl.d/99-proxmox-hardening.conf):
  • Complete ASLR randomization (kernel.randomize_va_space=2).
  • Strict memory restriction (kernel.kptr_restrict=2, kernel.dmesg_restrict=1).
  • SYN flood cookies enabled (net.ipv4.tcp_syncookies=1).
  • Source routing and ICMP redirects disabled.
  • SSH Hardening: Password authentication disabled across all nodes; SSH restricted to Ed25519 cryptographic keys only (ssh-audit rated 100/100).
  • Storage Encryption: LUKS encrypted data volumes unlocked automatically via Clevis/Tang Network-Bound Disk Encryption (NBDE).

17. Static IP & Ports Directory

| IP Address | Hostname / Resource | Exposed Ports | Subsystem Role | 192.168.1.1 | Gateway Router | 80, 443 | Default LAN Gateway | 192.168.1.9 | homeassistant (CT 106) | 8123, 1883 | Home Automation & MQTT Broker | 192.168.1.134 (OPNsense) | pve (Node 1 Host) | 8006, 22 | Proxmox VE Web Management

18. Cold-Start Runbook & Operational Cheat Sheet

Cold-Start Sequential Boot Sequence

  1. Phase 1 (Power & Networking): Turn on Coldex UPS $\to$ Power on Managed Switch $\to$ Verify OPNsense Firewall (VM 200) WAN connectivity.
  2. Phase 2 (Storage & DNS): Power on OMV NAS (Node 2) $\to$ Wait for NFS mounts $\to$ Verify AdGuard Home & Unbound DNS on OPNsense (VM 200).
  3. Phase 3 (Core Hypervisors): Power on Node 1 (x86_64) $\to$ Verify ZFS pool status (zpool status).
  4. Phase 4 (Security & Authentication): Start Authentik (CT 108) $\to$ Start Wazuh SIEM (CT 105) $\to$ Ingress Reverse Proxy active on OPNsense (VM 200).
  5. Phase 5 (Workloads & AI): Start Ollama (CT 110), Home Assistant (CT 106), and user microservices.

Proxmox Daily CLI Commands

# List all active containers and VMs
pct list && qm list

# Check ZFS storage pools health
zpool status -v

# Inspect Ollama LLM logs inside CT 110
pct exec 110 -- journalctl -u ollama -f -n 50

# Perform immediate vzdump backup of critical container
vzdump 110 --storage local-lvm --mode snapshot --compress zstd

19. Troubleshooting FAQ

Q: How do I resolve temporary DNS resolution errors inside LXC containers? Ensure the container nameserver is set to the local DNS resolver (`192.168.1.1` or `192.168.1.4`) via pct set <VMID> -nameserver 192.168.1.1 and verify /etc/resolv.conf contains valid nameservers.
Q: How do I verify GPU Passthrough for Ollama inside CT 110? Run pct exec 110 -- /usr/local/bin/ollama run qwen2.5-coder:1.5b "test" and check nvidia-smi on the Proxmox host to observe GPU compute utilization.
Q: How do I trigger an emergency snapshot restore in an isolated VLAN? Execute the automated Disaster Recovery script: ./scripts/disaster-recovery/dr_vzdump_restore.sh proxmox /mnt/pve/backup-nfs/dump.

20. Monorepo Layout & Engineering Portfolio

.
├── .github/workflows/ # CI/CD pipelines (Trivy, Gitleaks, Shellcheck, CD)
├── cyber/ # SOC, SIEM, Honeypots (T-Pot), eBPF & Sandbox
├── elo/ # Autonomous AI Agent Control Plane & Tools
├── hypervisors/ # Proxmox sysctl hardening & kernel profiles
├── kubernetes/ # Talos Linux & K3s manifests
├── opencore/ # OpenCore EFI bootloader for macOS Monterey KVM (/opencore/EFI)
├── scripts/ # Disaster Recovery & Chaos Engineering runners
├── services/ # Docker Compose & container configurations
├── terraform/ # Declarative Proxmox LXC & VM IaC modules
├── vms/ # NixOS & Windows Server configurations
└── web/ # Angular 20 Standalone Interactive Web App

This repository serves as a production-grade engineering portfolio and personal infrastructure lab, designed and maintained by @stefanutc1 to showcase hybrid cloud architecture, SecOps, GitOps, and resilient self-hosted platforms.


Author: @stefanutc1 Released under the MIT License.


Photo Gallery: Management Panels, Services & Loki Telemetry

All hardware nodes, virtual machines, and containers execute live on physical infrastructure. Below are direct interface captures of core control planes, running microservices, and centralized Grafana Loki log aggregation streams.

Core Management Panels

| Grafana: Homelab Nodes (12GB x64) | Grafana: OPNsense Perimeter Defense | Grafana Nodes Dashboard | Grafana OPNsense Dashboard | :---: | :---: | Pi-hole DNS Sinkhole & FTL (192.168.1.4:8080) | Home Assistant Automation Hub (192.168.1.10:8123) | Pi-hole Admin | Home Assistant | :---: | :---: | OPNsense: WireGuard Kernel VPN Mesh | OPNsense: Unbound DNS-over-TLS (DoT) | OPNsense WireGuard VPN | OPNsense Unbound DNS |


Core & Networking

| Nginx Proxy Manager | Pi-hole DNS Sinkhole | Nginx Proxy Manager | Pi-hole DNS | :---: | :---: | OPNsense Core Gateway | OPNsense Unbound DoT | OPNsense Core Gateway | OPNsense Unbound DoT | :---: | :---:

Storage & Backup

| Nextcloud Hub | Paperless-ngx Document OCR | Nextcloud Hub | Paperless-ngx | :---: | :---: | Syncthing File Sync | Proxmox Backup Server (PBS) | Syncthing | Proxmox Backup Server |


Automation & AI

| Ollama LLM Runtime | Open-WebUI AI Interface | Ollama LLM | Open-WebUI | :---: | :---: | Home Assistant Automation Hub | RenovateBot GitOps Engine | Home Assistant | RenovateBot |


Observability & Monitoring

| Grafana Enterprise Dashboard | Prometheus Metrics Engine | Grafana Enterprise | Prometheus Metrics | :---: | :---: | Gatus Status Healthchecker | Beszel Lightweight Metrics | Gatus Status | Beszel Metrics | :---: | :---: | Dozzle Real-Time Log Viewer | NetAlertX Network Scanner & Intrusion Monitor | Dozzle Log Viewer | NetAlertX |


Security & Cyber Lab

| OPNsense Suricata 8 NIDS/IPS | OPNsense CrowdSec LAPI Bouncer | Suricata IDS/IPS | CrowdSec Bouncer | :---: | :---: | CyberChef Cryptographic Utility | DFIR Dynamic Malware Sandbox | CyberChef | DFIR Sandbox | :---: | :---:

Media & Utilities

| Stirling-PDF Manipulation Suite | Kavita Digital Library | Stirling-PDF | Kavita Library | :---: | :---: | Transmission BitTorrent Client | Calibre-Web E-Book Manager | Transmission | Calibre-Web | :---: | :---: | Code-Server VS Code Cloud IDE | Draw.io Architecture Designer | Code-Server | Draw.io Designer | :---: | :---: | Trillium Structured Knowledge Base | ChangeDetection Web Monitor | Trillium Knowledge Base | ChangeDetection | :---: | :---: | Memos Lightweight Note Stream | Wallos Subscription Tracker | Memos Note Stream | Wallos Subscriptions | :---: | :---: | Flame Application Launcher | RustDesk Self-Hosted Remote Desktop | Flame Launcher | RustDesk | :---: | :---: | Kiwix Offline Wikipedia & Archive | Flatnotes Headless Wiki | Kiwix Archive | Flatnotes | :---: | :---: | Ntfy Real-Time Push Notifications | Bark iOS Alert Gateway | Ntfy Notifications | Bark Push | :---: | :---: | OpenGist Self-Hosted Pastebin | pgAdmin 4 PostgreSQL Manager | OpenGist Pastebin | pgAdmin |


Specialized Operating Systems & Telemetry (Loki Telemetry & Runtime Logs)

| Windows Server 2025 Datacenter (VM 201 · Loki Telemetry) | Red Hat Enterprise Linux 9.8 (VM 202 · Loki Telemetry) | Windows Server 2025 Datacenter Telemetry | RHEL 9.8 Telemetry | :---: | :---: | OpenStack 2024.1 Caracal (VM 205 · Cloud Horizon) | Metasploitable 2 (VM 206 · Vulnerable Target) | OpenStack Cloud Horizon | Metasploitable 2 | :---: | :---: | REMnux v7 Noble (VM 209 · Reverse Engineering) | OPNsense Core Gateway & Firewall (VM 200) | REMnux Malware Analysis | OPNsense Core Gateway | :---: | :---: | Proxmox VE 9.2 Primary (Node 1 · x86_64 Hypervisor) | Proxmox VE Primary x86_64 |


About the Author

Designed, engineered, and operated by @stefanutc1.

  • Focus: Infrastructure Engineering, Enterprise Virtualization (Proxmox VE x86_64 12GB DDR4-2133), Zero-Trust Network Defense (OPNsense, Suricata, CrowdSec, WireGuard), Smart Home (Home Assistant), DNS Filtering (Pi-hole), GitOps & IaC (Terraform, Ansible, CI/CD).
  • Purpose: Production-grade engineering portfolio showcasing on-premise and hybrid systems architecture.

About

Hybrid system architecture, isolated network segmentation, runtime monitoring, and local AI execution. Physical systems managed entirely through configuration scripts and automated pipelines.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages