Some tips and tricks to debug salt (states).

Use salt-call on minion

Useful to:

  • Trigger the deployment with direct and verbose feedback on terminal
  • Enables identification of slow state functions, i.e. repository refreshs

Steps:

Log into client machine, as root / sudo:

salt-call -l debug state.highstate

To apply a single state file (not recommended on production):

salt-call -l debug state.sls elasticsearch

Also useful to apply special housekeeping states (not in highstate), here a state that purges unnecessary kernels which is found in env stable:

salt-call state.sls kernelpurge stable

Dry Run

When fixing sth. on a machine an adapting the configuration management, one often want’s to double check that it is applied correctly before applying a state or a highstate again. This can be easily accomplished by running with test set to True:

salt-call state.sls elasticsearch test=True

or

salt-call state.highstate test=True

A diff of what would have been change is shown at the end but not applied, which can be checked for differences, i.e. in config files.

Use vagrant for development of states

Useful to:

  • Get quick feedback on salt state editing

Steps:

  • Create Vagrantfile according to vagrant salt provisioner documentation.
  • Setup master and minion as close to your production setup
  • Map salt sources into vm (/vagrant)

    vagrant up salt-master vagrant up salt-minion

  • Use salt-call in conjunction with state.highstate or state.sls as described above.

Start master / minion in debug mode

Useful to debug:

  • Connection problems
  • Salt State / Pillar rendering

Steps:

##### Minion:

service salt-minion stop

or

/etc/init.d/salt-minion stop

then

salt-minion -l debug
Master:
service salt-master stop

or

/etc/init.d/salt-master stop

then

salt-master -l debug

In both cases, the log information is written to the terminal.

Further Debugging with pdb:

pdb $(which salt-master)

Check your YAML state files

Common Pitfalls:

Error: Too many Functions...

/usr/local:
  file.directory:
    - recurse
    - mode: 755
  • Solution: recurse is not formulated as dict.

Error: Duplicate ID

nginx:
  pkg.installed
  
nginx:
  service.running
Solution 1: Use one state id
nginx:
  pkg.installed: []
    service.running: []
Solution 2: Use dedicated names for the state (e.g. if distributed across multiple files)
nginx_package:
  pkg.installed:
    - name: nginx
  
nginx_service:
  service.running:
    - name: nginx