YAML preprocessor / macro processor

Question

Is there a simple way to use a preproccesor / macro-processor with YAML files? (I.e. I\'m thinking of something along the lines of the C preprocessor)?

We have a lo

Answer 1

# Yamp - YAML Macro-Processor
# https://github.com/birchb1024/yamp

# in master.yaml
defmacro:
  name: country
  args: [$COUNTRY$]
  value:
    name: $COUNTRY$
    file: C:\data\{{$COUNTRY$}}
---
# in some file
- include: [master.yaml]

# Call with wherever needed:
{ country: USA }

Answer 2

You are trying to change things on the level of the string representation of YAML, and I think you shouldn't. YAML can load objects, and those objects can influence later elements loaded, by hooking into the parser. That way you can replace complete nodes with data, change values within scalars, etc.

Let's assume you have this YAML file main.yml:

- !YAMLPreProcessor
  verbose: '3'
  escape: ♦
- ♦replace(verbose)
- abcd
- ♦include(xyz.yml)
- xyz

and that xyz.yml is:

k: 9
l: 8
m: [7. 6]   # can be either

and you have ♦ as special character (it could be anything as long as YAMLPreProcessor value for special matches the start of the action keyword (replace and include). You want this to be round-tripped (loaded into data in memory and then dumped to the following YAML:

- !YAMLPreProcessor
  verbose: '3'
  escape: ♦
- '3'
- abcd
- k: 9
  l: 8
  m: [7. 6] # can be either
- xyz

You can do that by overloading the scalar constructor that gets called for each scalar and an appropriate YAMLPreProcessor class:

# coding: utf-8

from __future__ import print_function

import ruamel.yaml as yaml

def construct_scalar(loader, node):
    self = getattr(loader, '_yaml_preprocessor', None)
    if self and self.d.get('escape'):
        if node.value and node.value.startswith(self.d['escape']):
            key_word, rest = node.value[1:].split('(', 1)
            args, rest = rest.split(')', 1)
            if key_word == 'replace':
                res = u''
                for arg in args.split(','):
                    res += str(self.d[arg])
                node.value = res + rest
            elif key_word == 'include':
                inc_yml = yaml.load(
                    open(args),
                    Loader=yaml.RoundTripLoader
                )
                # this needs ruamel.yaml>=0.9.6
                return inc_yml
            else:
                print('keyword not found:', key_word)
    ret_val = loader._org_construct_scalar(node)
    # print('ret_val', type(ret_val), ret_val)
    return ret_val

class YAMLPreProcessor:
    def __init__(self, escape=None, verbose=0):
        self.d = dict(escape=escape, verbose=verbose)

    def __repr__(self):
        return "YAMLPreProcessor({escape!r}, {verbose})".format(**self.d)

    @staticmethod
    def __yaml_out__(dumper, self):
        return dumper.represent_mapping('!YAMLPreProcessor', self.d)

    @staticmethod
    def __yaml_in__(loader, data):
        from ruamel.yaml.comments import CommentedMap
        result = YAMLPreProcessor()
        loader._yaml_preprocessor = result
        z = dict()
        loader.construct_mapping(data, z)
        result.d = z
        yield result

    def __delete__(self):
        loader._yaml_preprocessor = None



def construct_yaml_str(self, node):
    value = self.construct_scalar(node)
    if isinstance(value, ScalarString):
        return value
    if PY3:
        return value
    try:
        return value.encode('ascii')
    except AttributeError:
        # in case you replace the node dynamically e.g. with a dict
        return value
    except UnicodeEncodeError:
        return value


loader = yaml.RoundTripLoader

loader.add_constructor('!YAMLPreProcessor', YAMLPreProcessor.__yaml_in__)
loader._org_construct_scalar = loader.construct_scalar
loader.construct_scalar = construct_scalar

data_from_yaml = yaml.load(open('main.yml'), Loader=loader)

#print ('out', data_from_yaml)

dumper = yaml.RoundTripDumper
# need to be able to represent '!YAMLPreProcessor'
# but you can of course also remove the first element
# from data_from_yaml if you don't want the preprocessor in your output
dumper.add_representer(YAMLPreProcessor, YAMLPreProcessor.__yaml_out__)

print(yaml.dump(data_from_yaml, Dumper=dumper, allow_unicode=True))

The above needs a recent version of ruamel.yaml (0.9.6) as older versions choke if construct_scalar returns a non-string object.

Please note that the position of the comment behind the line with the m key is relative to the start of the line, and in the example there is no compensation for the indent level of the node where the xyz.yml file is inserted.