Had an itch I’ve been meaning to scratch for a while. I build my Puppet environment using Terraform, which makes it nice and easy to tear things down and rebuild them. That is great, but it does leave me with an issue when it comes to the console SSL certificates.

Puppet will generate self-signed certs for the console, which work fine, but it was always a niggle that the certs couldn’t be automagically coaxed into being valid.

Since moving over to kubernetes for my home lab, I’ve come to expect managed SSL certificates for any public facing services, without me having to do anything.

Finally set some time aside to look at the options and thought I’d publish the details of the journey as well as where I ended up.

Step 1 - Let’s Encrypt

Obviously I wasn’t going to reinvent the wheel. If I wanted to manage and use Let’s Encrypt on a Puppet Enterprise server, I’d be using the Let’s Encrypt module. The module makes it very simple to get the relevant packages installed and configured.

1
2
3
4
5
6
class { 'letsencrypt':
  config     => {
    email  => 'certs@albatrossflavour.com',
  },
  config_dir => '/etc/letsencrypt',
}

One of the beauties of this module is that it also sets up a cron job to renew the generated certs, so you don’t need to keep an eye on it.

Classify your server with that and you’ll end up with certbot and it’s dependencies. Great start and feeling confident about the future!

Then we need to generate a certificates:

1
2
3
4
5
6
letsencrypt::certonly { 'puppet.gcp.albatrossflavour.com':
  domains       => ['puppet.gcp.albatrossflavour.com'],
  manage_cron   => true,
  plugin        => 'webroot',
  webroot_paths => ['/var/www],
}

Annnnnnnd that’s where things start to go south.

When you request a cert, the most simple method of validating you are who you say you are is to have a web server on the host respond to a query sent to port 80. Sure we say, easy, couldn’t take much to do!

By default, the Puppet Enterprise nginx config does a 301 redirect for http requests to https. This means any queries from Let’s Encrypt to validate the requests will end up as https requests and the cert request will fail.

Step 2 - nginx

Let me introduce you to puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect. This parameter controls if that redirect is in place. So step …. 6(?) was to use hiera to disable the http redirect:

1
2
❯ cat data/role/role::pe::master.yaml
puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect: false

Once we run this through a puppet run, the redirect gets removed! Score!

Only problem is, without the redirect, the nginx server doesn’t listen on port 80. OK, we can fix that easily. We could use the pe_nginx::directive type, but I found it to be a bit of an overkill for what I needed. Instead I opted for a simple template:

1
2
3
4
5
6
7
8
  file { '/etc/puppetlabs/nginx/conf.d/certs.conf':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => template("${module_name}/cert_vhost.conf.erb"),
    notify  => Exec['pe_nginx'],
  }

and the template is just:

1
2
3
4
5
6
7
8
9
10
11
server {
  listen       80;
  server_name  <%= @fqdn %>;
  index        index.html;
  location /.well-known {
    root /var/www;
  }
  location / {
    return 301 https://$server_name$request_uri;
  }
}

(Sorry, still using erb, I will at some point rewrite everthing in epp)

The template keeps the 301 redirect in place for anything other than a request to /.well-known, which is where Let’s Encrypt looks for the validation info.

Run this through and the nginx vhost gets created. However the letsencrypt::certonly call still fails on the first run. The notify to the pe_nginx service, which is done when we create the cert_vhost.conf, doesn’t happen in the order we need. The Let’s Encrypt module is trying to get a response before we’ve setup the vhost. Now this would work on subsequent runs, but Golden Rule #1 is to make sure, whenever possible, that you get a clean puppet run in one pass. Plus this was a challenge.

Before I worked on fixing that, I wanted to make sure I could make the rest of it work. Let’s sum up where we are:

• I’ve got the Let’s Encrypt client installed and configured
• I’ve got a new nginx vhost running that allows the Let’s Encrypt web validation queries through and redirects any other http traffic.
• After a couple of puppet runs, I’ve got valid SSL certs in /etc/letsencrypt

Step 3 - The console certs

It’d been a while since I played with the Puppet console SSL certs, so I checked in with the source of truth, which outlines the steps we need to go through to use custom SSL certs with the console:

1. Retrieve the custom certificate and private key.
2. Move the certificate to /etc/puppetlabs/puppet/ssl/certs/console-cert.pem, replacing any existing file named console-cert.pem.
3. Move the private key to /etc/puppetlabs/puppet/ssl/private_keys/console-cert.pem, replacing any existing file named console-cert.pem.

So we can just create a file resource that takes the Let’s Encrypt cert/key and places them into the console SSL directory structure.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
  file { '/etc/puppetlabs/puppet/ssl/certs/console-cert.pem':
    ensure    => file,
    owner     => 'pe-puppet',
    group     => 'pe-puppet',
    links     => 'follow',
    mode      => '0640',
    source    => "/etc/letsencrypt/live/${facts['puppet_server']}/cert.pem",
    backup    => '.puppet_bak',
    notify    => Service['pe-nginx'],
    subscribe => Letsencrypt::Certonly[$facts['puppet_server']],
  }

  file { '/etc/puppetlabs/puppet/ssl/private_keys/console-cert.pem':
    ensure    => file,
    owner     => 'pe-puppet',
    group     => 'pe-puppet',
    links     => 'follow',
    mode      => '0644',
    source    => "/etc/letsencrypt/live/${facts['puppet_server']}/privkey.pem",
    backup    => '.puppet_bak',
    notify    => Service['pe-nginx'],
    subscribe => Letsencrypt::Certonly[$facts['puppet_server']],
  }

Sure enough, when I try this on the Puppet server, life is good and we have a console with valid SSL certs.

Time to crack open a bottle of red and tick an item off my to do list.

Damn it, Golden Rule #2 raises it’s head. It’s not finished until you know it works fine from scratch… WITHOUT breaking Golden Rule #1.

Step 4 - Rebuild

Nuked the Puppet server and rebuilt it. Lots of failures on the first run (we kinda expected that), however they didn’t go away. The nginx service never gets restarted as the various dependencies can be resolved, but not in the way we need.

This is the fun (?) of declarative configuration management. I can only manage things once, which includes only being able to notify the pe-nginx service once. Then the compiler will figure out when the service is restarted, based on all of the dependencies in the catalog.

I played around with a lot of options, some waaaaay hackier than I wanted to go with. Plus this was becoming a fun exercise.

All I needed to be able to do was to inject a restart of the pe-nginx service twice in a single run.

To get there, I bent a few of the future Golden Rules and came up with:

1
2
3
4
exec { 'restart_nginx':
  command     => '/bin/systemctl restart pe-nginx',
  refreshonly => true,
}

This exec resource will allow me to do a restart of pe-nginx outside of, and before, the notify => Service['pe-nginx'] parameters.

Yes, it’s a hack, but it’s a hack on the side of the angels. Not only will it solve the issue of the process not working at all, but it will also allow the certs to be generated, and installed, in a single run.

Step 5 - Quick Robin, to the pdk mobile

I did a fair bit of testing and found the end result to be far more reliable and useful than I thought it would be.

I first created a fully parameterised profile to manage the certs. Once I got that working, it was a no-brainer to create a module to share the love.

That’s how I ended up with the pe_console_letsencrypt module. You can also just check out the code.

I’ve done a fair bit of testing, but I’m certainly not saying it’s bulletproof.

Step 6 - Profit?

I’m still scratching my head to think of another way I could get the certs generated and in place in a single run without the exec hack. That being said, I’ve had hacks a lot worse go live in the past!

Note

While researching the links for this post, I came across a blog less frequently updated than mine which has a post from 2017 showing a similar way of achieving the same thing, but without the hack to make it work in a single run.