Serializing Multiple Pillar Keys Into A File In SaltStack
Problem Overview
Recently, we wanted to have a file into which certain pieces of, potentially disparate, pillar data could be dropped into. The format of this data should remain as YAML, so using the file.serialize
state function seemed a likely candidate to for use. I began testing with the serialize function setting the param merge_if_exists: True
. And while this works with some caveats, overall it's not the solving the problem in the way that we wanted to.
This was tested with salt-minion and salt-master both on version 2019.2.
Setup
Pillar:
breakfast:
egg: scrambled
meats:
- sausage
- bacon
sides:
spam: gross
beans: bakeddinner:
egg: poached
seafood: lobster
other:
- truffle pate
- mornay sauce
sides:
spam: delicious
egg: over easy
The contents of the created file are:
egg: poached
meats:
- sausage
- bacon
other:
- truffle pate
- mornay sauce
seafood: lobster
sides:
beans: baked
egg: over easy
spam: delicious
Problems
- The top level keys are gone. I would say that is not desirable or at least that we may want to be able to configure it.
- This is not stateful. Meaning we would always show changes needing to be made because we are iterating data into the same file using a for loop.
- As well, pillars are written in order so last key wins. this can be seen with the
sides:spam
key - Lists from pillars will overwrite values. In our example, adding to the
dinner
pillarmeats: ['steak']
results in thebreakfast
meats being removed. - If the top level key is a list, then an error is thrown and that pillar will not be rendered into the file.
egg: poached
meats:
- steak
other:
- truffle pate
- mornay sauce
seafood: lobster
sides:
beans: baked
egg: over easy
spam: delicious
Added this pillar data and included lunch
in application_pillar
lunch:
- normal menu
- specials
Caused this error:
----------
ID: serialize key lunch
Function: file.serialize
Name: /tmp/serialize_test.txt
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/state.py", line 1933, in call
**cdata['kwargs'])
File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1951, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/salt/states/file.py", line 6852, in serialize
merged_data = salt.utils.dictupdate.merge_recurse(existing_data, dataset)
File "/usr/lib/python2.7/site-packages/salt/utils/dictupdate.py", line 92, in merge_recurse
return update(copied, obj_b, merge_lists=merge_lists)
File "/usr/lib/python2.7/site-packages/salt/utils/dictupdate.py", line 43, in update
raise TypeError('Cannot update using non-dict types in dictupdate.update()')
TypeError: Cannot update using non-dict types in dictupdate.update()
Started: 18:31:19.807767
Duration: 2.972 ms
Working The Problem
I came to the solution that was needed by first writing the Salt formula in Python. By doing this, I could more easily manipulate the data structures to build what I needed.
With this formula written in python, now we review the problems that were found before:
- Top level keys can be kept or not. This is configurable.
- This is stateful. The merge of all data happens before the call to
file.serialize
- Dictionary merging is handled through a merge strategy. See the code for available strategies.
- Lists are merged by default. This can be made to be configurable if needed.
- This will work so long as the top level keys are kept. By default this is the case.
Final Solution
With the logic completed in python, we can actually move our new function, _get_data_from_all_pillars_and_merge
, into a custom salt module and then convert our state data in the run function back to a standard YAML + Jinja SLS file.
I moved the function to a custom helpers.py
salt module.
Then I call the function from the final version of the Salt formula.
Originally published at https://smartaleksolutions.com on September 22, 2020.