Serializing Multiple Pillar Keys Into A File In SaltStack

Aleksandr Rain
3 min readSep 22, 2020

--

Problem Overview

Recently, we wanted to have a file into which certain pieces of, potentially disparate, pillar data could be dropped into. The format of this data should remain as YAML, so using the file.serialize state function seemed a likely candidate to for use. I began testing with the serialize function setting the param merge_if_exists: True. And while this works with some caveats, overall it's not the solving the problem in the way that we wanted to.

This was tested with salt-minion and salt-master both on version 2019.2.

Setup

Pillar:

breakfast:  
egg: scrambled
meats:
- sausage
- bacon
sides:
spam: gross
beans: baked
dinner:
egg: poached
seafood: lobster
other:
- truffle pate
- mornay sauce
sides:
spam: delicious
egg: over easy

The contents of the created file are:

egg: poached
meats:
- sausage
- bacon
other:
- truffle pate
- mornay sauce
seafood: lobster
sides:
beans: baked
egg: over easy
spam: delicious

Problems

  1. The top level keys are gone. I would say that is not desirable or at least that we may want to be able to configure it.
  2. This is not stateful. Meaning we would always show changes needing to be made because we are iterating data into the same file using a for loop.
  3. As well, pillars are written in order so last key wins. this can be seen with the sides:spam key
  4. Lists from pillars will overwrite values. In our example, adding to the dinner pillar meats: ['steak'] results in the breakfast meats being removed.
  5. If the top level key is a list, then an error is thrown and that pillar will not be rendered into the file.
egg: poached
meats:
- steak
other:
- truffle pate
- mornay sauce
seafood: lobster
sides:
beans: baked
egg: over easy
spam: delicious

Added this pillar data and included lunch in application_pillar

lunch:
- normal menu
- specials

Caused this error:

----------
ID: serialize key lunch
Function: file.serialize
Name: /tmp/serialize_test.txt
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/state.py", line 1933, in call
**cdata['kwargs'])
File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1951, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/salt/states/file.py", line 6852, in serialize
merged_data = salt.utils.dictupdate.merge_recurse(existing_data, dataset)
File "/usr/lib/python2.7/site-packages/salt/utils/dictupdate.py", line 92, in merge_recurse
return update(copied, obj_b, merge_lists=merge_lists)
File "/usr/lib/python2.7/site-packages/salt/utils/dictupdate.py", line 43, in update
raise TypeError('Cannot update using non-dict types in dictupdate.update()')
TypeError: Cannot update using non-dict types in dictupdate.update()
Started: 18:31:19.807767
Duration: 2.972 ms

Working The Problem

I came to the solution that was needed by first writing the Salt formula in Python. By doing this, I could more easily manipulate the data structures to build what I needed.

With this formula written in python, now we review the problems that were found before:

  1. Top level keys can be kept or not. This is configurable.
  2. This is stateful. The merge of all data happens before the call to file.serialize
  3. Dictionary merging is handled through a merge strategy. See the code for available strategies.
  4. Lists are merged by default. This can be made to be configurable if needed.
  5. This will work so long as the top level keys are kept. By default this is the case.

Final Solution

With the logic completed in python, we can actually move our new function, _get_data_from_all_pillars_and_merge, into a custom salt module and then convert our state data in the run function back to a standard YAML + Jinja SLS file.

I moved the function to a custom helpers.py salt module.

Then I call the function from the final version of the Salt formula.

Originally published at https://smartaleksolutions.com on September 22, 2020.

--

--

Aleksandr Rain
Aleksandr Rain

Written by Aleksandr Rain

0 Followers

Climber, surfer, yogi, dad who does some IT on the side to get by.