Tuesday, June 24, 2014

Replace The Foreman Self Signed Certificate with a Trusted Certificate

I've installed a few Foreman servers to provide provisioning and configuration management (via Puppet). This document will cover the steps to replace the self signed certificate used for the web interface with a trusted certificate.

For those unfamiliar with Foreman and Puppet, here are snippet from both project pages (http://theforeman.org/learn_more.html and http://puppetlabs.com/puppet/what-is-puppet:

Foreman is an open source project that helps system administrators manage servers throughout their lifecycle, from provisioning and configuration to orchestration and monitoring. Using Puppet or Chef and Foreman's smart proxy architecture, you can easily automate repetitive tasks, quickly deploy applications, and proactively manage change, both on-premise with VMs and bare-metal or in the cloud.

Foreman provides comprehensive, interaction facilities including a web frontend, CLI and RESTful API which enables you to build higher level business logic on top of a solid foundation.
Puppet is IT automation software that defines and enforces the state of your infrastructure throughout your software development cycle. From provisioning and configuration to orchestration and reporting, from initial code development through production release and updates, Puppet frees sysadmins from writing one-off, fragile scripts and other manual tasks. At the same time, Puppet ensures consistency and dependability across your infrastructure.

With Puppet, repetitive tasks are automated away, so sysadmins can quickly deploy business applications, scaling easily from tens of servers to thousands, both on-premise and in the cloud.

By default, the Puppet / Foreman server install uses Puppet's own internal CA for issuing SSL certificates. The Foreman install defaults to using the Puppet CA self signed cert for the web interface. The following steps will replace The Foreman's SSL certificate for the user web interface, but will leave the Puppet CA and SSL certs in place for Puppet related work.

I spent a bit of time trying to get this working, but each attempt resulted in a working web interface and broken Puppet master to client communications and Foreman proxy. Essentially, I was changing the SSL certificate entries in too many locations. Dominic on the #theforeman channel on FreeNode IRC directed me to this Google Groups thread that listed the short list of places to make the change. The following steps are based on that post.

  1. Create the SSL key and csr
    sudo su - 
    mkdir /root/Incommon-cert
    cd /root/Incommon-cert
    
    openssl req -out $(hostname)-2048.csr -new -newkey rsa:2048 -nodes -keyout $(hostname -f)-2048.key
  2. Copy the contents of the csr to the clipboard and use it to request an InCommon SSL certificate
  3. Once the cert is approved, download the following files to /root/Incommon-cert on the Puppet / Foreman server:
    • as X509 Certificate only, Base64 encoded
    • as X509 Intermediates/root only, Base64 encoded
  4. Rename the files so that we know these are InCommon files
    mv puppet.tld.blah.crt puppet.tld.blah-2048-incommon-cert.crt
    mv puppet.tld.blah_interm.crt puppet.tld.blah-2048-incommon-interm.crt
    chown root:root *.crt
  5. Copy the files to the appropriate directories
    cp puppet.tld.blah-2048.key /var/lib/puppet/ssl/private_keys/
    cp puppet.tld.blah-2048-incommon-cert.crt /var/lib/puppet/ssl/certs/
    cp puppet.tld.blah-2048-incommon-interm.crt /var/lib/puppet/ssl/certs/
    wget https://www.incommon.org/certificates/repository/incommon-ssl.ca-bundle -O /var/lib/puppet/ssl/certs/incommon-ssl.ca-bundle
  6. Set the appropriate permissions and SELinux configs for the key
    cd /var/lib/puppet/ssl/private_keys/
    chown puppet:puppet *.key
    chmod 640 *.key
    chcon -u system_u -r object_r -t puppet_var_lib_t *.key
    
    ls -lZ
    -rw-r-----. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah-2048.key
    -rw-r-----. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah.pem
  7. Set perms and SELinux for the certs
    cd /var/lib/puppet/ssl/certs/
    chown puppet:puppet *
    chcon -u system_u -r object_r -t puppet_var_lib_t *.crt
    
    ls -lZ
    -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 ca.pem
    -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 incommon-ssl.ca-bundle
    -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah-2048-incommon-cert.crt
    -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah-2048-incommon-interm.crt
    -rw-r--r--. puppet puppet system_u:object_r:puppet_var_lib_t:s0 puppet.tld.blah.pem
  8. Next edit the various config files
    • /etc/puppet/node.rb: Change the line :ssl_ca to use the new interm cert
      --- /etc/puppet/node.rb.orig 2014-03-24 17:48:09.215000045 -0500
      +++ /etc/puppet/node.rb 2014-06-24 10:24:51.049282905 -0500
      @@ -8,7 +8,8 @@
         :facts        => true,          # true/false to upload facts
         :timeout      => 10,
         # if CA is specified, remote Foreman host will be verified
      -  :ssl_ca       => "/var/lib/puppet/ssl/certs/ca.pem",      # e.g. /var/lib/puppet/ssl/certs/ca.pem
      +  #:ssl_ca       => "/var/lib/puppet/ssl/certs/ca.pem",      # e.g. /var/lib/puppet/ssl/certs/ca.pem
      +  :ssl_ca       => "/var/lib/puppet/ssl/certs/puppet.tld.blah-2048-incommon-interm.crt",      # e.g. /var/lib/puppet/ssl/certs/ca.pem
         # ssl_cert and key are required if require_ssl_puppetmasters is enabled in Foreman
         :ssl_cert     => "/var/lib/puppet/ssl/certs/puppet.tld.blah.pem",    # e.g. /var/lib/puppet/ssl/certs/FQDN.pem
         :ssl_key      => "/var/lib/puppet/ssl/private_keys/puppet.tld.blah.pem"      # e.g. /var/lib/puppet/ssl/private_keys/FQDN.pem
    • /usr/lib/ruby/site_ruby/1.8/puppet/reports/foreman.rb: Change the foreman_ssl_ca = line to use the interm cert
      --- /usr/lib/ruby/site_ruby/1.8/puppet/reports/foreman.rb.orig 2014-03-24 17:44:37.494000046 -0500
      +++ /usr/lib/ruby/site_ruby/1.8/puppet/reports/foreman.rb 2014-06-24 10:28:54.497406986 -0500
      @@ -5,7 +5,8 @@
       # URL of your Foreman installation
       $foreman_url='https://puppet.tld.blah'
       # if CA is specified, remote Foreman host will be verified
      -$foreman_ssl_ca = "/var/lib/puppet/ssl/certs/ca.pem"
      +#$foreman_ssl_ca = "/var/lib/puppet/ssl/certs/ca.pem"
      +$foreman_ssl_ca = "/var/lib/puppet/ssl/certs/puppet.tld.blah-2048-incommon-interm.crt"
       # ssl_cert and key are required if require_ssl_puppetmasters is enabled in Foreman
       $foreman_ssl_cert = "/var/lib/puppet/ssl/certs/puppet.tld.blah.pem"
       $foreman_ssl_key = "/var/lib/puppet/ssl/private_keys/puppet.tld.blah.pem"
    • /etc/httpd/conf.d/05-foreman-ssl.conf: Change three lines SSLCertificateFile, SSLCertificateKeyFile and SSLCertificateChainFile to use the new cert, key and CA bundle respectively
      --- /etc/httpd/conf.d/05-foreman-ssl.conf.orig 2014-06-24 10:30:59.917531640 -0500
      +++ /etc/httpd/conf.d/05-foreman-ssl.conf 2014-06-24 10:32:36.318164714 -0500
      @@ -35,11 +35,18 @@
       
         ## SSL directives
         SSLEngine on
      -  SSLCertificateFile      /var/lib/puppet/ssl/certs/puppet.tld.blah.pem
      -  SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppet.tld.blah.pem
      -  SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
      +  SSLCertificateFile      "/var/lib/puppet/ssl/certs/puppet.tld.blah-2048-incommon-cert.crt"
      +  SSLCertificateKeyFile   "/var/lib/puppet/ssl/private_keys/puppet.tld.blah-2048.key"
      +  SSLCertificateChainFile "/var/lib/puppet/ssl/certs/incommon-ssl.ca-bundle"
         SSLCACertificatePath    /etc/pki/tls/certs
         SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem
      +
      +#  SSLCertificateFile      /var/lib/puppet/ssl/certs/puppet.tld.blah.pem
      +#  SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppet.tld.blah.pem
      +#  SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
      +#  SSLCACertificatePath    /etc/pki/tls/certs
      +#  SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem
      +
         SSLVerifyClient         optional
         SSLVerifyDepth          3
         SSLOptions +StdEnvVars
  9. Restart the services (foreman-proxy restart probably isn't necessary but may as well)
    service httpd restart
    service foreman-proxy restart