Dynamic Templating

Overview

When I switched companies a while ago, I came across some a pattern in several Salt formulas that the CTO at my new company had written. To this day, it’s still one of the more brilliant snippets I’ve seen for the use case of templating out several files in a state all at once.

Context

Some applications can require a great many configuration files. For example a webserver such as apache or nginx can have many configuration files to support various virtual hosts. In this case, the most straightforward way of writing a formula would dictate creating a declaration for each file or template. A time consuming, repetitive task that requires upkeep.

Oddly, though I’ve configured and worked with SaltStack across several organizations at this point, I had not come across this type of scenario before now. At most, I’ve only had a handful of config files for an application and so this problem wasn’t really a problem. At my new company though, the CTO had already come up against this problem, and then put the time in to figure out a very elegant solution.

Setup

Let’s say our application is nginx and we need to drop in a few configuration files for it. For arguments sake, some of these conf files need to be templated, and some do not.

The Conventional Method

Now, lets say that we need to manage a static nginx.conf, but also drop some templated config files, eggs.conf and bacon.conf, into the /etc/nginx/conf.d directory.

Conventionally, this would be accomplished with three separate declarations. On our Salt Master, we would need to create a state file along with the static and template files we will need to go with it. The directory structure in our file_roots may look something like:

nginx
├── files
│   ├── bacon.conf.jinja
│   ├── eggs.conf.jinja
│   └── nginx.conf
└── init.sls

And then our state file would look like:

{{ sls }} - manage the nginx config file:
  file.managed:
    - name: /etc/nginx/nginx.conf
    - source: salt://nginx/files/nginx.conf

{{ sls }} - managed the eggs config file:
  file.managed:
    - name: /etc/nginx/conf.d/eggs.conf
    - source: salt://nginx/files/eggs.conf.jinja

{{ sls }} - managed the bacon config file:
  file.managed:
    - name: /etc/nginx/conf.d/bacon.conf
    - source: salt://nginx/files/bacon.conf.jinja

Add Jinja Magic

Now, instead of being conventional, lets use jinja to make this dynamic. First, we’ll need to revise the layout of our states’s files directory to:

nginx
├── files
│   └── etc
│       └── nginx
│           ├── conf.d
│           │   ├── bacon.conf.jinja
│           │   └── eggs.conf.jinja
│           └── nginx.conf
└── init.sls

As can be seen, in our state’s files directory, we’ve added the directory path where the nginx config files will be when rendered to the minion. This is important because it provides the hint our jinja code will need to know where on the minion filesystem to put the files.

Now, we revise the state:

Given the files and state above, the output of a render is:

nginx/init.sls - manage the config file /etc/nginx/nginx.conf: 
  file.managed: 
    - source: salt://nginx/files/etc/nginx/nginx.conf 
    - name: /etc/nginx/nginx.conf 
    - makedirs: True 

nginx/init.sls - manage the template file /etc/nginx/conf.d/bacon.conf: 
  file.managed: 
    - source: salt://nginx/files/etc/nginx/conf.d/bacon.conf.jinja 
    - name: /etc/nginx/conf.d/bacon.conf 
    - makedirs: True 
    - template: jinja 

nginx/init.sls - manage the template file /etc/nginx/conf.d/eggs.conf: 
  file.managed: 
    - source: salt://nginx/files/etc/nginx/conf.d/eggs.conf.jinja 
    - name: /etc/nginx/conf.d/eggs.conf 
    - makedirs: True 
    - template: jinja

How It Works

The single declaration in our state uses the salt cp.list_master function to retrieve a list of full paths to all files on the master that match the prefix we specify. In our declaration, we specify the prefix as being the files directory of the running formula ( nginx/files). So now that we have a list of full paths to retrieve source files from the master with, we only need to know where to put the file on the minion. Since the directory layout once inside the SLS files directory fully matches the path for each config file on the minion, we are able to derive the target location by using a jinja replace function to substitute the prefix with nothing for each file.

As such, this code is completely portable between different states with no modifications. The only caveat is having the files directory contain the full directory paths of where the config files and templates need to be placed.

A Note On Security

Working with the salt cp.list_master function gives us a good reminder that ALL minions have access to everything in the salt master's file roots. Keep this in mind with static config files and make sure you never put secrets in anything in the file roots.

Originally published at https://smartaleksolutions.com on October 20, 2020.