Serializing Multiple Pillar Keys Into A File In SaltStack

Problem Overview

Recently, we wanted to have a file into which certain pieces of, potentially disparate, pillar data could be dropped into. The format of this data should remain as YAML, so using the file.serialize state function seemed a likely candidate to for use. I began testing with the serialize function setting the param merge_if_exists: True. And while this works with some caveats, overall it's not the solving the problem in the way that we wanted to.

This was tested with salt-minion and salt-master both on version 2019.2.

Setup

Pillar:

breakfast:  
  egg: scrambled  
  meats:
    - sausage
    - bacon
  sides:
    spam: gross
    beans: baked
dinner:
  egg: poached
  seafood: lobster
  other:
    - truffle pate
    - mornay sauce
  sides:
    spam: delicious
    egg: over easy

The contents of the created file are:

egg: poached
meats:
- sausage
- bacon
other:
- truffle pate
- mornay sauce
seafood: lobster
sides:
  beans: baked
  egg: over easy
  spam: delicious

Problems

1. The top level keys are gone. I would say that is not desirable or at least that we may want to be able to configure it.
2. This is not stateful. Meaning we would always show changes needing to be made because we are iterating data into the same file using a for loop.
3. As well, pillars are written in order so last key wins. this can be seen with the sides:spam key
4. Lists from pillars will overwrite values. In our example, adding to the dinner pillar meats: ['steak'] results in the breakfast meats being removed.
5. If the top level key is a list, then an error is thrown and that pillar will not be rendered into the file.

egg: poached
meats:
  - steak
other:
  - truffle pate
  - mornay sauce
seafood: lobster
sides:
  beans: baked
  egg: over easy
  spam: delicious

Added this pillar data and included lunch in application_pillar

lunch:
  - normal menu
  - specials

Caused this error:

----------
          ID: serialize key lunch
    Function: file.serialize
        Name: /tmp/serialize_test.txt
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/site-packages/salt/state.py", line 1933, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1951, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/site-packages/salt/states/file.py", line 6852, in serialize
                  merged_data = salt.utils.dictupdate.merge_recurse(existing_data, dataset)
                File "/usr/lib/python2.7/site-packages/salt/utils/dictupdate.py", line 92, in merge_recurse
                  return update(copied, obj_b, merge_lists=merge_lists)
                File "/usr/lib/python2.7/site-packages/salt/utils/dictupdate.py", line 43, in update
                  raise TypeError('Cannot update using non-dict types in dictupdate.update()')
              TypeError: Cannot update using non-dict types in dictupdate.update()
     Started: 18:31:19.807767
    Duration: 2.972 ms

Working The Problem

I came to the solution that was needed by first writing the Salt formula in Python. By doing this, I could more easily manipulate the data structures to build what I needed.

With this formula written in python, now we review the problems that were found before:

1. Top level keys can be kept or not. This is configurable.
2. This is stateful. The merge of all data happens before the call to file.serialize
3. Dictionary merging is handled through a merge strategy. See the code for available strategies.
4. Lists are merged by default. This can be made to be configurable if needed.
5. This will work so long as the top level keys are kept. By default this is the case.

Final Solution

With the logic completed in python, we can actually move our new function, _get_data_from_all_pillars_and_merge, into a custom salt module and then convert our state data in the run function back to a standard YAML + Jinja SLS file.

I moved the function to a custom helpers.py salt module.

Then I call the function from the final version of the Salt formula.

Originally published at https://smartaleksolutions.com on September 22, 2020.