Serializing Multiple Pillar Keys Into A File In SaltStack

Problem Overview

This was tested with salt-minion and salt-master both on version 2019.2.

Setup

breakfast:  
egg: scrambled
meats:
- sausage
- bacon
sides:
spam: gross
beans: baked
dinner:
egg: poached
seafood: lobster
other:
- truffle pate
- mornay sauce
sides:
spam: delicious
egg: over easy

The contents of the created file are:

egg: poached
meats:
- sausage
- bacon
other:
- truffle pate
- mornay sauce
seafood: lobster
sides:
beans: baked
egg: over easy
spam: delicious

Problems

  1. The top level keys are gone. I would say that is not desirable or at least that we may want to be able to configure it.
  2. This is not stateful. Meaning we would always show changes needing to be made because we are iterating data into the same file using a for loop.
  3. As well, pillars are written in order so last key wins. this can be seen with the sides:spam key
  4. Lists from pillars will overwrite values. In our example, adding to the dinner pillar meats: ['steak'] results in the breakfast meats being removed.
  5. If the top level key is a list, then an error is thrown and that pillar will not be rendered into the file.
egg: poached
meats:
- steak
other:
- truffle pate
- mornay sauce
seafood: lobster
sides:
beans: baked
egg: over easy
spam: delicious

Added this pillar data and included lunch in application_pillar

lunch:
- normal menu
- specials

Caused this error:

----------
ID: serialize key lunch
Function: file.serialize
Name: /tmp/serialize_test.txt
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/state.py", line 1933, in call
**cdata['kwargs'])
File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1951, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/salt/states/file.py", line 6852, in serialize
merged_data = salt.utils.dictupdate.merge_recurse(existing_data, dataset)
File "/usr/lib/python2.7/site-packages/salt/utils/dictupdate.py", line 92, in merge_recurse
return update(copied, obj_b, merge_lists=merge_lists)
File "/usr/lib/python2.7/site-packages/salt/utils/dictupdate.py", line 43, in update
raise TypeError('Cannot update using non-dict types in dictupdate.update()')
TypeError: Cannot update using non-dict types in dictupdate.update()
Started: 18:31:19.807767
Duration: 2.972 ms

Working The Problem

With this formula written in python, now we review the problems that were found before:

  1. Top level keys can be kept or not. This is configurable.
  2. This is stateful. The merge of all data happens before the call to file.serialize
  3. Dictionary merging is handled through a merge strategy. See the code for available strategies.
  4. Lists are merged by default. This can be made to be configurable if needed.
  5. This will work so long as the top level keys are kept. By default this is the case.

Final Solution

I moved the function to a custom helpers.py salt module.

Then I call the function from the final version of the Salt formula.

Originally published at https://smartaleksolutions.com on September 22, 2020.

--

--

Climber, surfer, yogi, dad who does some IT on the side to get by.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aleksandr Rain

Climber, surfer, yogi, dad who does some IT on the side to get by.