Replace The Foreman Self Signed Certificate with a Trusted Certificate

I've installed a few Foreman servers to provide provisioning and configuration management (via Puppet). This document will cover the steps to replace the self signed certificate used for the web interface with a trusted certificate.

For those unfamiliar with Foreman and Puppet, here are snippet from both project pages (http://theforeman.org/learn_more.html and http://puppetlabs.com/puppet/what-is-puppet:

Foreman is an open source project that helps system administrators manage servers throughout their lifecycle, from provisioning and configuration to orchestration and monitoring. Using Puppet or Chef and Foreman's smart proxy architecture, you can easily automate repetitive tasks, quickly deploy applications, and proactively manage change, both on-premise with VMs and bare-metal or in the cloud.

Foreman provides comprehensive, interaction facilities including a web frontend, CLI and RESTful API which enables you to build higher level business logic on top of a solid foundation.

Puppet is IT automation software that defines and enforces the state of your infrastructure throughout your software development cycle. From provisioning and configuration to orchestration and reporting, from initial code development through production release and updates, Puppet frees sysadmins from writing one-off, fragile scripts and other manual tasks. At the same time, Puppet ensures consistency and dependability across your infrastructure.

With Puppet, repetitive tasks are automated away, so sysadmins can quickly deploy business applications, scaling easily from tens of servers to thousands, both on-premise and in the cloud.

By default, the Puppet / Foreman server install uses Puppet's own internal CA for issuing SSL certificates. The Foreman install defaults to using the Puppet CA self signed cert for the web interface. The following steps will replace The Foreman's SSL certificate for the user web interface, but will leave the Puppet CA and SSL certs in place for Puppet related work.

I spent a bit of time trying to get this working, but each attempt resulted in a working web interface and broken Puppet master to client communications and Foreman proxy. Essentially, I was changing the SSL certificate entries in too many locations. Dominic on the #theforeman channel on FreeNode IRC directed me to this Google Groups thread that listed the short list of places to make the change. The following steps are based on that post.

1. Create the SSL key and csr
   

   sudo su - 
   mkdir /root/Incommon-cert
   cd /root/Incommon-cert
   
   openssl req -out $(hostname)-2048.csr -new -newkey rsa:2048 -nodes -keyout $(hostname -f)-2048.key

2. Copy the contents of the csr to the clipboard and use it to request an InCommon SSL certificate
3. Once the cert is approved, download the following files to /root/Incommon-cert on the Puppet / Foreman server:
   • as X509 Certificate only, Base64 encoded
   • as X509 Intermediates/root only, Base64 encoded
4. Rename the files so that we know these are InCommon files
   

   mv puppet.tld.blah.crt puppet.tld.blah-2048-incommon-cert.crt
   mv puppet.tld.blah_interm.crt puppet.tld.blah-2048-incommon-interm.crt
   chown root:root *.crt

5. Copy the files to the appropriate directories
   

   cp puppet.tld.blah-2048.key /var/lib/puppet/ssl/private_keys/
   cp puppet.tld.blah-2048-incommon-cert.crt /var/lib/puppet/ssl/certs/
   cp puppet.tld.blah-2048-incommon-interm.crt /var/lib/puppet/ssl/certs/
   wget https://www.incommon.org/certificates/repository/incommon-ssl.ca-bundle -O /var/lib/puppet/ssl/certs/incommon-ssl.ca-bundle

6. Set the appropriate permissions and SELinux configs for the key
   

   cd /var/lib/puppet/ssl/private_keys/
   chown puppet:puppet *.key
   chmod 640 *.key
   chcon -u system_u -r object_r -t puppet_var_lib_t *.key
   
   ls -lZ
   -rw-r-----. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah-2048.key
   -rw-r-----. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah.pem

7. Set perms and SELinux for the certs
   

   cd /var/lib/puppet/ssl/certs/
   chown puppet:puppet *
   chcon -u system_u -r object_r -t puppet_var_lib_t *.crt
   
   ls -lZ
   -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 ca.pem
   -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 incommon-ssl.ca-bundle
   -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah-2048-incommon-cert.crt
   -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah-2048-incommon-interm.crt
   -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah.pem

8. Next edit the various config files
   • /etc/puppet/node.rb: Change the line :ssl_ca to use the new interm cert
     

     --- /etc/puppet/node.rb.orig 2014-03-24 17:48:09.215000045 -0500
     +++ /etc/puppet/node.rb 2014-06-24 10:24:51.049282905 -0500
     @@ -8,7 +8,8 @@
        :facts        => true,          # true/false to upload facts
        :timeout      => 10,
        # if CA is specified, remote Foreman host will be verified
     -  :ssl_ca       => "/var/lib/puppet/ssl/certs/ca.pem",      # e.g. /var/lib/puppet/ssl/certs/ca.pem
     +  #:ssl_ca       => "/var/lib/puppet/ssl/certs/ca.pem",      # e.g. /var/lib/puppet/ssl/certs/ca.pem
     +  :ssl_ca       => "/var/lib/puppet/ssl/certs/puppet.tld.blah-2048-incommon-interm.crt",      # e.g. /var/lib/puppet/ssl/certs/ca.pem
        # ssl_cert and key are required if require_ssl_puppetmasters is enabled in Foreman
        :ssl_cert     => "/var/lib/puppet/ssl/certs/puppet.tld.blah.pem",    # e.g. /var/lib/puppet/ssl/certs/FQDN.pem
        :ssl_key      => "/var/lib/puppet/ssl/private_keys/puppet.tld.blah.pem"      # e.g. /var/lib/puppet/ssl/private_keys/FQDN.pem

   • /usr/lib/ruby/site_ruby/1.8/puppet/reports/foreman.rb: Change the foreman_ssl_ca = line to use the interm cert
     

     --- /usr/lib/ruby/site_ruby/1.8/puppet/reports/foreman.rb.orig 2014-03-24 17:44:37.494000046 -0500
     +++ /usr/lib/ruby/site_ruby/1.8/puppet/reports/foreman.rb 2014-06-24 10:28:54.497406986 -0500
     @@ -5,7 +5,8 @@
      # URL of your Foreman installation
      $foreman_url='https://puppet.tld.blah'
      # if CA is specified, remote Foreman host will be verified
     -$foreman_ssl_ca = "/var/lib/puppet/ssl/certs/ca.pem"
     +#$foreman_ssl_ca = "/var/lib/puppet/ssl/certs/ca.pem"
     +$foreman_ssl_ca = "/var/lib/puppet/ssl/certs/puppet.tld.blah-2048-incommon-interm.crt"
      # ssl_cert and key are required if require_ssl_puppetmasters is enabled in Foreman
      $foreman_ssl_cert = "/var/lib/puppet/ssl/certs/puppet.tld.blah.pem"
      $foreman_ssl_key = "/var/lib/puppet/ssl/private_keys/puppet.tld.blah.pem"

   • /etc/httpd/conf.d/05-foreman-ssl.conf: Change three lines SSLCertificateFile, SSLCertificateKeyFile and SSLCertificateChainFile to use the new cert, key and CA bundle respectively
     

     --- /etc/httpd/conf.d/05-foreman-ssl.conf.orig 2014-06-24 10:30:59.917531640 -0500
     +++ /etc/httpd/conf.d/05-foreman-ssl.conf 2014-06-24 10:32:36.318164714 -0500
     @@ -35,11 +35,18 @@
      
        ## SSL directives
        SSLEngine on
     -  SSLCertificateFile      /var/lib/puppet/ssl/certs/puppet.tld.blah.pem
     -  SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppet.tld.blah.pem
     -  SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
     +  SSLCertificateFile      "/var/lib/puppet/ssl/certs/puppet.tld.blah-2048-incommon-cert.crt"
     +  SSLCertificateKeyFile   "/var/lib/puppet/ssl/private_keys/puppet.tld.blah-2048.key"
     +  SSLCertificateChainFile "/var/lib/puppet/ssl/certs/incommon-ssl.ca-bundle"
        SSLCACertificatePath    /etc/pki/tls/certs
        SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem
     +
     +#  SSLCertificateFile      /var/lib/puppet/ssl/certs/puppet.tld.blah.pem
     +#  SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppet.tld.blah.pem
     +#  SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
     +#  SSLCACertificatePath    /etc/pki/tls/certs
     +#  SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem
     +
        SSLVerifyClient         optional
        SSLVerifyDepth          3
        SSLOptions +StdEnvVars

9. Restart the services (foreman-proxy restart probably isn't necessary but may as well)
   

   service httpd restart
   service foreman-proxy restart

Use Puppet to Set the Ethernet Speed

The way our network is configured, the workstations perform much better if their nic speed is set, rather than using auto negotiation.

It's easy enough to go out to each of the machines to do this, but it is much easier to let Puppet do the grunt work.

This is a class that will do just that for Red Hat based workstations (the class resides in its own file /etc/puppet/manifests/classes/nic_speed.pp):
class nic_speed {


   case $hostname {
      "host01",
      "host02",
      "host07",
      "host08",
      "host10": {
          # Set the nics to 100 half
          append_if_no_such_line{ eth0:
             file => "/etc/sysconfig/network-scripts/ifcfg-eth0",
             line => 'ETHTOOL_OPTS="speed 100 duplex half autoneg off"'}
      }
  default: { }
  }
}

The hosts listed in the case statement will have the ETHTOOL_OPTS line appended to their ifcfg-eth0 file, all others will use the default, which is to do nothing.

append_if_no_such_line is not native functionality of Puppet and is part of a custom class called cfengine.pp that adds a few commonly used features of CFengine:

# /etc/puppet/manifests/classes/cfengine.pp

define append_if_no_such_line($file, $line, $refreshonly = 'false') {
  exec { "/bin/echo '$line' >> '$file'":
     unless      => "/bin/grep -Fxqe '$line' '$file'",
     path        => "/bin",
     refreshonly => $refreshonly,
  }
}

define delete_lines($file, $pattern) {
  exec { "sed -i -r -e '/$pattern/d' $file":
     path   => "/bin",
     onlyif => "/bin/grep -E '$pattern' '$file'",
  }
}

System Administration with Puppet

I manage quite few Linux workstations (mostly Red Hat EL4 and EL5, CentOS) and found that my home grown scripting approach to maintaining these systems was becoming increasingly difficult.

One obvious problem with using a mountain of scripts to maintain systems, what happens when I want to go on vacation and have to leave the Windows guy in charge? I needed a system that would make it easy for a temporary admin (or new team member) to take the reigns.

A couple recent issues of Linux Magazine (a great publication by the way) reviewed CFengine and Puppet for systems management. I had known that there were open source solutions out there, but had not spent the time to do the research, preferring to handle the latest challenge by writing another script.

The articles finally convinced me to do further research. I ultimately chose Puppet for several reasons, including:

• Written in Ruby
• Very easy to get a base system up and running
• Helpful community
• Active development

I really like the fact that the configuration files use Ruby syntax, so by configuring Puppet, I'm also learning something about Ruby. I'm a Perl guy, but have been interested in learning Ruby for system administration. Now I have the excuse :-)

Example Syntax
A short example of the power of Puppet.

class baseclass {
   service { "sshd":
      enable => true,
      ensure => running,
   }

   file { "/etc/shadow":
      owner => "root",
      group => "root",
      mode => 400,
   }
}

Puppet will ensure that nodes that include this class (baseclass) have the sshd service set to startup on boot and verify that it is running, starting it if necessary. Puppet will also verify the owner, group and permissions of the /etc/shadow file, if they don't match our definition Puppet will make the necessary changes.

These definitions will work regardless of the target operating system (Red Hat EL, Fedora, SuSE, Sun Solaris, BSD, whatever). That's very appealing to administrators because they don't have to script for each unique environment, Puppet handles it in the background.

Puppet uses another Ruby based utility called 'facter' that returns a whole slew of data about the node (architecture, hostname, fqdn, ipaddress, kernelrelease, ...). You can use these facter defined variables in your configurations. For example, Puppet provides for basic 'if' and 'case' statements to allow you to define conditions, for example, only include the class '64bitonly' if the architecure (returned by facter) is x86_64:

case $architecture {
   x86_64: { include 64bitonly }
   default: { }
}


Useful Puppet Links

• Interview with the author of "Pulling Strings with Puppet" by James Turnbull
• Purchase the book at Amazon "Pulling Strings with Puppet", a great read for those new to Puppet
• The Puppet home page at Reductive Labs
• #puppet IRC channel on freenode.net is very helpful