ITeas Tools & Infrastructure

Appearance

Foreman & Puppet Setup ​

Documentation for the Foreman and Puppet configuration management setup.

Overview ​

Foreman is a complete lifecycle management tool for physical and virtual servers, integrated with Puppet for configuration management.

Setup Date: 2025-10-20

Architecture ​

┌─────────────────────────────┐
│            Gitlab           │
│       (Puppet module)       │
└─────────────────────────────┘
               │
               │  Pushes Puppet modules
               │  to environments based on
               │  Git commits and tags
               ▼
┌─────────────────────────────┐
│        Foreman Server       │
│        (Web UI, ENC)        │
└─────────────────────────────┘
               │
               │ Reports & Facts
               │ Node Classifications
               ▼
┌─────────────────────────────┐            ┌─────────────────────────────┐
│        Puppet Server        │◄───────────│      Hashicorp Vault        │
│         (Puppet CA)         │            │    (Secrets via Hiera)      │
│      (Hiera Backend)        │   Secrets  │                             │
└─────────────────────────────┘            └─────────────────────────────┘
               │   Catalog Compilation
               │   Agent Communication
      ┌────────┘─────────┐
      ▼                  ▼
 ┌─────────┐        ┌─────────┐
 │ Agent 1 │        │ Agent 2 │
 └─────────┘        └─────────┘

Components ​

Foreman Server ​

Purpose: Web-based management interface for infrastructure

Key Features:

  • Node inventory and classification
  • External Node Classifier (ENC) for Puppet
  • Reporting and auditing
  • Provisioning (bare metal, VMs)
  • Smart proxies for remote management

Puppet Server ​

Purpose: Configuration management and automation

Key Features:

  • Catalog compilation
  • Certificate Authority (CA)
  • Agent communication
  • Module management
  • Hiera data backend

Installation ​

ITeas Setup

Diese Installation wurde auf Rocky Linux durchgeführt und verwendet den Foreman Installer.

Prerequisites

  • Linux server (Ubuntu/CentOS/Rocky Linux)
  • Minimum 4GB RAM, 2 CPU cores
  • DNS properly configured
  • Firewall ports opened (443, 8140, 8443)
  • Git (muss bei Rocky Linux manuell nachinstalliert werden)

Foreman Installation ​

Rocky Linux / CentOS / RHEL ​

See: https://theforeman.org/manuals/3.16/index.html#2.Quickstart

bash
# Enable Puppet's 8.x repository:
dnf -y install https://yum.puppet.com/puppet8-release-el-9.noarch.rpm

# Enable the Foreman repositories:
dnf -y install https://yum.theforeman.org/releases/3.16/el9/x86_64/foreman-release.rpm

# Install Foreman Installer
dnf install -y foreman-installer

# Run Foreman Installer
foreman-installer -l info --foreman-foreman-service-puma-workers=3 --foreman-foreman-service-puma-threads-min=16 --foreman-foreman-service-puma-threads-max=16

# Install Foreman Maintenance Tool
yum install rubygem-foreman_maintain

Configuration ​

Foreman Configuration ​

OICD Anmeldung ​

Installation der OICD Anmeldung:

bash
# Install OICD Anmeldung
yum install mod_auth_openidc keycloak-httpd-client-install -y
Konfiguration in Microsoft Entra ID ​
  1. Im Entra Admin Center zu „Identity" → „Applications" → „App registrations" navigieren.
  2. Eine neue App-Registrierung erstellen. Der Anwendung einen Namen geben (z. B. Foreman). Die Redirect-URI auf „Web" und „https://your-foreman-fqdn/users/extlogin/redirect_uri" konfigurieren. Auf „Register" klicken.
  3. Die Application ID und Tenant ID notieren, diese werden später benötigt.
  4. Zu „Authentication" navigieren. ID-Token aktivieren und auf „Save" klicken.
  5. Zu „Certificates & secrets" navigieren. Ein neues Client Secret generieren.
  6. Das soeben generierte Client Secret notieren, es wird später benötigt.
  7. Zu „Token configuration" navigieren. Auf „Add optional claim" klicken und die Claims „email", „given_name", „family_name" und „preferred_username" hinzufügen. Die MS Graph-Berechtigungen „email" und „profile" müssen aktiviert werden, wenn dazu aufgefordert wird.
  8. Zu „API permissions" navigieren. Admin-Zustimmung für die Organisation für die MS Graph-API-Berechtigungen erteilen.
Konfiguration in Foreman ​

Anlegen der Config /etc/httpd/conf.d/05-foreman-ssl.d/openidc.conf

Config

apache
OIDCProviderMetadataURL https://login.microsoftonline.com/*tenant-id*/v2.0/.well-known/openid-configuration
OIDCClientID xxx
OIDCClientSecret yyy
OIDCRedirectURI https://foreman.iteas.tools/users/extlogin/redirect_uri
OIDCCryptoPassphrase zzz
OIDCRemoteUserClaim preferred_username
OIDCScope "openid email profile"
OIDCResponseType id_token

<Location /users/extlogin>
    AuthType openid-connect
    Require valid-user
    LogLevel debug

    RequestHeader set REMOTE_USER %{OIDC_CLAIM_preferred_username}e
    RequestHeader set REMOTE_USER_EMAIL %{OIDC_CLAIM_email}e
    RequestHeader set REMOTE_USER_FIRSTNAME %{OIDC_CLAIM_given_name}e
    RequestHeader set REMOTE_USER_LASTNAME %{OIDC_CLAIM_family_name}e
</Location>

Module in Apache aktiveren Pfad /etc/httpd/conf.modules.d/10-auth_openidc.conf

Hinzufügen von

apache
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_user_module modules/mod_authz_user.so
Foreman Web UI Konfiguration ​

In der Foreman Web UI zu Administer → Settings → Authentication gehen und folgende Einstellungen vornehmen:

Authorize login delegation = Yes
Authorize login delegation auth source user autocreate = External
OIDC JWKs URL = https://login.microsoftonline.com/*tenant-id*/discovery/v2.0/keys
OIDC Audience = *application-ID*
OIDC Issuer = https://login.microsoftonline.com/*tenant-id*/v2.0
OIDC Algorithm = RS256

Neustart von Apache systemctl restart httpd

Login unter https://your-foreman-fqdn/users/extlogin

Location: /etc/foreman/

Key configuration files:

  • settings.yaml - Main Foreman settings
  • database.yml - Database configuration
  • encryption_key.rb - Encryption key for sensitive data

Puppet Server Configuration ​

Puppet Server wird immer durch "foreman-installer" und deren Parameter konfiguriert. Manuelle Änderungen werden überschrieben

Hashicorp Vault Integration ​

Vault ist als Hiera-Backend im Puppet Server integriert und ermöglicht das sichere Verwalten von Secrets und sensiblen Daten.

Für generelle Vault-Dokumentation siehe: Hashicorp Vault

Vault Hiera Backend Setup ​

Die Puppet-Hiera Integration basiert auf: https://github.com/voxpupuli/puppet-hiera_vault

Installation ​
bash
# Install vault gems on Puppet Server
/opt/puppetlabs/puppet/bin/gem install --user-install vault
/opt/puppetlabs/puppet/bin/gem install --user-install debouncer
puppetserver gem install vault
puppetserver gem install debouncer
# Restart Foreman services
foreman-maintain service restart
Vault Configuration for Puppet ​

Info: Die Konfiguration kann auch in der Web-UI erfolgen.

1. Enable KV Secrets Engine:

bash
# Enable KV v2 secrets engine for puppet
vault secrets enable -path=secret kv-v2

# Verify
vault secrets list

2. Create Puppet Server Policy:

Create puppet-policy.hcl:

hcl
# Policy for Puppet Server to read secrets
path "secret/data/puppet/*" {
  capabilities = ["read"]
}

Apply the policy:

bash
vault policy write puppet puppet-policy.hcl

3. Create Token for Puppet Server:

bash
# Create a renewable token with puppet-server policy
vault token create \
  -policy=puppet \
  -period=768h \
  -renewable \
  -display-name="puppet-server"

# Save the token securely on Puppet Server
echo "hvs.xxxxxxxxxxxxxxxxxxxxxxxx" > /etc/puppetlabs/vault/hiera_token.txt
chown puppet:puppet /etc/puppetlabs/vault/hiera_token.txt
chmod 600 /etc/puppetlabs/vault/hiera_token.txt
Hiera Configuration ​

Edit hiera.yaml in your specific environment:

yaml
---
version: 5
defaults:
  data_hash: yaml_data
hierarchy:
  - name: "Hiera-vault lookup"
    lookup_key: hiera_vault
    options:
      ssl_verify: false
      address: https://vault.iteas.cloud
      token: /etc/puppetlabs/vault/hiera_token.txt
      #default_field: value
      v1_lookup: false
      v2_guess_mount: false
      mounts:
        secret/data:
          - "puppet/%{environment}/%{::trusted.certname}"
          - "puppet/%{environment}/common"
          - "puppet/common"
  - name: "Per-Node data"
    path: "nodes/%{trusted.certname}.yaml"
  - name: "Common data"
    path: "common.yaml"

Using Vault Secrets in Puppet ​

Storing Secrets in Vault ​
bash
# Store production database credentials
vault kv put puppet/production/foreman-test-client/database \
  password="SuperSecretPassword123" \
  username="dbuser" \
  host="db.iteas.tools"

# Store API key common to all environments
vault kv put puppet/common/foreman-test-client/api \
  token="api-key-xxxxxxxxx" \
  endpoint="https://api.example.com"

# Store node-specific credentials
vault kv put puppet/production/foreman-test-client \
  ssh_key="-----BEGIN RSA PRIVATE KEY-----..." \
  admin_password="NodeSpecificPassword"
Accessing Secrets in Puppet Code ​

In Puppet Manifest:

puppet
# Lookup from Vault via Hiera
$db_password = lookup('database')['password']
$db_username = lookup('database')['username']
$db_host     = lookup('database')['host']

# Use in resource
class { 'mysql::server':
  root_password => $db_password,
}

mysql::db { 'myapp_db':
  user     => $db_username,
  password => $db_password,
  host     => $db_host,
}

In Hiera YAML (referencing Vault):

Da Vault direkt in der Hiera-Hierarchie eingebunden ist, werden Secrets automatisch geladen. YAML-Dateien werden für nicht-sensitive Daten verwendet.

Path Structure Best Practices ​

Empfohlene Pfadstruktur in Vault:

puppet/
├── common/                      # Shared across all environments
│   ├── api/                     # External API credentials
│   └── certificates/            # Common SSL certs
├── production/
│   ├── common/                  # Production-wide secrets
│   │   ├── database/
│   │   └── smtp/
│   ├── node1.iteas.tools/      # Node-specific secrets
│   └── node2.iteas.tools/
└── development/
    ├── common/
    │   ├── database/
    │   └── smtp/
    └── testnode.iteas.tools/

Token Management ​

Install CLI vault client first: https://developer.hashicorp.com/vault/install

Automated Token Renewal ​

Create /etc/cron.daily/vault-token-renew:

bash
#!/bin/bash
VAULT_TOKEN=$(cat /etc/puppetlabs/vault/hiera_token.txt)
VAULT_ADDR=https://vault.iteas.cloud

export VAULT_ADDR
export VAULT_TOKEN

# Renew token
vault token renew

# Log result
if [ $? -eq 0 ]; then
  logger "Vault token renewed successfully"
else
  logger "ERROR: Failed to renew Vault token"
  exit 1
fi

Make executable:

bash
chmod +x /etc/cron.daily/vault-token-renew

Troubleshooting Vault Integration ​

Token Expired ​

Symptoms: Puppet runs fail with authentication errors

Solution:

bash
# Create new token
vault token create -policy=puppet -period=768h -renewable

# Update token file
echo "s.newtoken" > /etc/puppetlabs/vault/hiera_token.txt

# Restart Puppet Server
systemctl restart puppetserver
Secret Not Found ​

Symptoms: Hiera lookup returns nil/empty

Solution:

  1. Verify secret exists: vault kv get puppet/production/mysecret
  2. Check Hiera hierarchy in hiera.yaml for your environment
  3. Verify policy allows read access to the path
  4. Check Puppet Server logs: /var/log/puppetlabs/puppetserver/puppetserver.log
Debug Vault Lookups ​

Enable debug logging:

bash
# Check Puppet Server logs for Vault operations
tail -f /var/log/puppetlabs/puppetserver/puppetserver.log | grep -i vault

# Test Vault connectivity from Puppet Server
vault status
vault token lookup
vault kv get puppet/common/api

Environment Configuration ​

Puppet Environment Konfiguration erfolgt durch Gitops.

Repo: https://git.styrion.net/puppet/puppet-control

Jeder Branch entspricht einer Environment in Puppet. Der Sync erfolgt mit r10k im Minutentakt.

*/1  *  *  *  * root /opt/puppetlabs/bin/r10k_polling_deploy_all.sh

Puppet Module ​

GitLab Integration ​

Die Environment-Konfiguration von Puppet und die Puppet-Module selbst werden in GitLab verwaltet.

Die Namenskonvention der Gitlab Projekte ist hier:

  • puppet-control - Steuert Environments über Branches, im Puppetfile werden die Puppet-Module für diese Environment festgelegt
  • puppet-module-* - Stellt jeweils ein Puppet-Modul dar

Jedes Puppet-Modul stellt ein Projekt in GitLab dar zb https://git.styrion.net/puppet/puppet-module-iteasapps Diese Puppet-Module wiederrum werden im puppet-control Repo im jeweiligen Branch/Environment im Puppetfile zugewiesen.

Alle Puppet Module befinden sich in der Gitlab Gruppe https://git.styrion.net/puppet

Workflow:

  1. Module werden in GitLab im development Branch entwickelt
  2. Module werden auf einen Client im development Environment getestet
  3. Wenn Tests positiv sind wird der Code in den production Branch gemerged und getaggt
  4. Puppet-Modul wird dann im puppet-control Projekt mit jeweiligen Tag versehen

Deployment-Prozess:

  • Puppet Server pollt r10k Konfiguration von Gitlab im Minutentakt
  • Environments werden automatisch aktualisiert
  • Änderungen sind sofort für Puppet Agents verfügbar

Managing Modules ​

Using Puppetfile ​

Im puppet-control Projekt existiert für jeden Branch ein Puppetfile.
Dieses wird von r10k gelesen und dann automatisch am Puppet Server deployed.

Dieses Puppetfile kann wie folgt aussehen:

ruby
# Gibt die Quelle für Forge-Module an (Standard)
forge "https://forge.puppet.com"

# --- Externe (Forge-) Module ---

mod 'puppetlabs-stdlib', '9.7.0'
mod 'puppetlabs-apt', '11.1.0'
mod 'puppetlabs-powershell', '6.0.2'
mod 'puppetlabs-reboot', '5.1.0'
mod 'puppetlabs-docker', '10.3.0'

# --- Eigene (interne) Module ---

# Eigene Module (werden typischerweise als separate Repositories geführt)
# Wichtig: Hier verwenden Sie den Branch oder Tag, den Sie für die Umgebung verwenden möchten.
mod 'iteas_iteasapps',
    :git => 'https://xxx:yyy@git.styrion.net/puppet/puppet-module-iteasapps.git',
    :ref => 'development'

Best Practice bei Iteas: Custom Module in GitLab entwickeln und über CI/CD deployen.

Agent Setup ​

Installing Puppet Agent ​

bash
wget --no-check-certificate https://foreman.iteas.tools/puppetfiles/puppet_agent_installer_latest && chmod +x puppet_agent_installer_latest && ./puppet_agent_installer_latest --server <puppet-server>

Certificate Signing ​

Via Foreman: https://foreman.iteas.tools/smart_proxies/1-foreman-iteas-tools#puppet-ca

oder Puppet CLI:

bash
# List certificate requests
puppetserver ca list

# Sign certificate
puppetserver ca sign --certname node1.example.com

# Sign all pending
puppetserver ca sign --all

Common Workflows ​

Adding a New Node ​

  1. Install Puppet agent on node (siehe Agent Setup)
  2. Sign certificate on Puppet server (siehe Certificate Signing)
  3. Node appears in Foreman > Hosts
  4. Assign host groups, classes, and parameters in Foreman if needed
  5. Run /opt/puppetlabs/bin/puppet agent --test on node to apply configuration

Applying Configuration ​

Manual run:

bash
/opt/puppetlabs/bin/puppet agent --test

Dry run (noop mode):

bash
/opt/puppetlabs/bin/puppet agent --test --noop

Viewing Reports ​

  • Foreman UI: Monitor > Reports
  • Puppet Server logs: /var/log/puppetlabs/puppetserver/
  • Agent logs: /var/log/puppetlabs/puppet/

Monitoring & Maintenance ​

Health Checks ​

Check Puppet Server status:

bash
systemctl status puppetserver
puppetserver ca list --all

Check Foreman status:

bash
systemctl status foreman
foreman-maintain health check

Backups ​

Important directories to backup:

  • /etc/puppetlabs/ - Puppet configuration
  • /etc/foreman/ - Foreman configuration
  • /opt/puppetlabs/server/data/puppetserver/ - Puppet CA
  • /var/lib/foreman/ - Foreman database and data

Backup script example:

bash
#!/bin/bash
BACKUP_DIR=/backup/puppet-$(date +%Y%m%d)
mkdir -p $BACKUP_DIR

# Backup configurations
tar czf $BACKUP_DIR/puppet-config.tar.gz /etc/puppetlabs/
tar czf $BACKUP_DIR/foreman-config.tar.gz /etc/foreman/

# Backup CA
tar czf $BACKUP_DIR/puppet-ca.tar.gz /opt/puppetlabs/server/data/puppetserver/

# Backup database
sudo -u postgres pg_dump foreman > $BACKUP_DIR/foreman-db.sql

Updates ​

Update Foreman/Puppet: https://theforeman.org/manuals/3.16/index.html#3.6Upgradeto3.16

For the last step foreman-installer run with same arguments as for install

bash
foreman-installer -l info --foreman-foreman-service-puma-workers=3 --foreman-foreman-puma-threads-min=16 --foreman-foreman-service-puma-threads-max=16

Troubleshooting ​

Agent Can't Connect to Server ​

Symptoms: /opt/puppetlabs/bin/puppet agent --test fails with connection error

Solutions:

  1. Check network connectivity: telnet puppet.example.com 8140
  2. Verify DNS resolution: nslookup puppet.example.com
  3. Check firewall rules
  4. Verify Puppet Server is running: systemctl status puppetserver

Certificate Issues ​

Symptoms: SSL certificate errors

Solutions:

  1. Check certificate status: puppetserver ca list --all
  2. Clean certificate on agent: puppet ssl clean
  3. Remove certificate on server: puppetserver ca clean --certname node.example.com
  4. Regenerate certificate on agent: /opt/puppetlabs/bin/puppet agent --test

Catalog Compilation Failures ​

Symptoms: Agent fails to apply catalog

Solutions:

  1. Check Puppet Server logs: /var/log/puppetlabs/puppetserver/puppetserver.log
  2. Validate syntax: puppet parser validate manifests/site.pp
  3. Test on server: puppet agent -t --noop --server puppet.example.com
  4. Check for circular dependencies in modules

Foreman UI Issues ​

Symptoms: Foreman web interface not responding

Solutions:

  1. Check Foreman service: systemctl status foreman
  2. Check Apache/Nginx: systemctl status apache2
  3. Review logs: /var/log/foreman/production.log
  4. Restart services: systemctl restart foreman

Best Practices ​

  1. Version Control - Store Puppet code in Git (bei ITeas: GitLab)
  2. GitLab CI/CD - Nutze automatisiertes Deployment von Modulen über GitLab
  3. Testing - Use development environment before production
  4. Modules - Keep modules small and focused
  5. Hiera - Separate data from code
  6. Documentation - Document custom modules and changes
  7. Monitoring - Monitor Puppet runs and reports
  8. Backups - Regular backups of CA and configurations
  9. Updates - Keep Puppet and Foreman updated

ITeas-Spezifische Best Practices ​

  • Alle Module in GitLab verwalten - Keine manuellen Änderungen auf dem Puppet Server
  • Branch-Strategie nutzen - Development → Testing → Production
  • Git Tags für Versionen - Versionierung über Git Tags
  • Code Reviews - Änderungen über Merge Requests reviewen lassen
  • CI/CD Pipeline - Automatisiertes Testing und Deployment nutzen

Resources ​

Notes ​

ITeas Setup Details

Operating System: Rocky Linux

Installation Method: Foreman Installer

Important: Git muss manuell nachinstalliert werden:

bash
dnf install -y git

Weitere spezifische Details:

  • Server hostnames
  • Network configuration
  • Custom modules used
  • Integration with other systems
  • Team conventions and standards

Iteas Tools Integration Platform Version v1.0.21

Copyright © 2025 ITeas

Version: v1.0.21 Version: v1.0.21
Commit: 7a0e1c11
Deployed at: 2026-09-24T13:56:52Z