cannot get boost::spirit parser&lexer working for token types other than std::string or int or double

Question

This does not compile (code below).

There was another question here with the same error. But I don\'t understand the answer. I already tried inserting qi::eps in pla

Answer 1

Has anybody gotten this working for a lexer returning something else than double or int or string?

Sure. Simple examples might be found on this site

And for the parser also returning non-trivial objects?

Here's your real problem. Spirit is nice for a subset of parsers that are expressed easily in a eDSL, and has the huge benefit of "magically" mapping to a selection of attributes.

Some of the realities are:

• attributes are expected to have value-semantic; using polymorphic attributes is hard (How can I use polymorphic attributes with boost::spirit::qi parsers?, e.g.)

• using Lex makes most of the sweet-spot disappear since all "highlevel" parsers (like real_parser, [u]int_parser) are out the window. The Spirit devs are on record they prefer not to use Lex. Moreover, Spirit X3 doesn't have Lex support anymore.

---

Background Information:

I'd very much consider parsing the source as-is, into direct value-typed AST nodes. I know, this is probably what you consider "trivial objects", but don't be deceived by apparent simplicity: recursive variant trees have some expressive power.

Examples

• Here's a trivial AST to represent JSON in <20 LoC: Boost Karma generator for composition of classes¹
• Here we represent the Graphviz source format with full fidelity: How to use boost spirit list operator with mandatory minimum amount of elements?

I've since created the code to transform that AST into a domain representation with fully correct ownership, cascading lexically scoped node/edge attributes and cross references. I have just recovered that work and put it up on github if you're interested, mainly because the task is pretty similar in many respects, like the overriding/inheriting of properties and resolving identifiers within scopes: https://github.com/sehe/spirit-graphviz/blob/master/spirit-graphviz.cpp#L660

Suggestions, Ideas

In your case I'd take similar approach to retain simplicity. The code shown doesn't (yet) cover the trickiest ingredients (like nature attribute overrides within a discipline).

Once you start implementing use-cases like resolving compatible disciplines and the absolute tolerances at a given node, you want a domain model with full fidelity. Preferrably, there would be no loss of source information, and immutable AST information².

As a middle ground, you could probably avoid building an entire source-AST in memory only to transform it in one big go, at the top-level you could have:

file = qi::skip(skipper) [
        *(m_sDiscipline | m_sNature | m_sModule) [process_ast(_1)]
    ];

Where process_ast would apply the "trivial" AST representation into the domain types, one at a time. That way you keep only small bits of temporary AST representation around.

The domain representation can be arbitrarily sophisticated to support all your logic and use-cases.

Let's "Show, Don't Tell"

Baking the simplest AST that comes to mind matching the grammar³:

namespace frank { namespace ast {
    struct nature {
        struct empty{};

        std::string name;
        std::string inherits;

        enum class Attribute { units, access, idt, ddt, abstol };
        using Value = boost::variant<int, double, std::string>;
        std::map<Attribute, Value> attributes;
    };

    struct discipline {
        enum enumDomain { eUnspecified, eDiscrete, eContinuous };
        struct properties_t {
            enumDomain domain = eUnspecified;
            boost::optional<std::string> flow, potential;
        };

        std::string name;
        properties_t properties;
    };

    // TODO
    using module = qi::unused_type;
    using file   = std::vector<boost::variant<nature, discipline, module> >;

    enum class type { real, integer, string };
} }

This is trivial and maps 1:1 onto the grammar productions, which means we have very little impedance.

Tokens? We Don't Need Lex For That

You can have common token parsers without requiring the complexities of Lex

Yes, Lex (especially statically generated) can potentially improve performance, but

• if you need that, I wager Spirit Qi is not your best option anyways
• premature optimization...

What I did:

struct tokens {
    // implicit lexemes
    qi::rule<It, std::string()> string, identifier;
    qi::rule<It, double()> real;
    qi::rule<It, int()> integer;
    qi::rule<It, ast::nature::Value()> value;
    qi::rule<It, ast::nature::Attribute()> attribute;
    qi::rule<It, ast::discipline::enumDomain()> domain;

    struct attribute_sym_t : qi::symbols<char, ast::nature::Attribute> {
        attribute_sym_t() {
            this->add
               ("units", ast::nature::Attribute::units)
               ("access", ast::nature::Attribute::access)
               ("idt_nature", ast::nature::Attribute::idt)
               ("ddt_nature", ast::nature::Attribute::ddt)
               ("abstol", ast::nature::Attribute::abstol);
        }
    } attribute_sym;

    struct domain_sym_t : qi::symbols<char, ast::discipline::enumDomain> {
        domain_sym_t() {
            this->add
               ("discrete", ast::discipline::eDiscrete)
               ("continuous", ast::discipline::eContinuous);
        }
    } domain_sym;

    tokens() {
        using namespace qi;
        auto kw = qr::distinct(copy(char_("a-zA-Z0-9_")));

        string     = '"' >> *("\\" >> char_ | ~char_('"')) >> '"';
        identifier = char_("a-zA-Z_") >> *char_("a-zA-Z0-9_");
        real       = double_;
        integer    = int_;
        attribute  = kw[attribute_sym];
        domain     = kw[domain_sym];

        value = string | identifier | real | integer;

        BOOST_SPIRIT_DEBUG_NODES((string)(identifier)(real)(integer)(value)(domain)(attribute))
    }
};

Liberating, isn't it? Note how

• all attributes are automatically propagated
• strings handle escapes (this bit was commented out in your Lex approach). We don't even need semantic actions to (badly) pry out the unquoted/unescaped value

• we used distinct to ensure keyword parsing matches only full identifiers. (See How to parse reserved words correctly in boost spirit).

  This is actually where you notice the lack of separate lexer.

  On the flipside, this makes context-sensitive keywords a breeze (lex can easily prioritizes keywords over identifiers that occur in places where keywords cannot occur.⁴)

What About Skipping Space/Comments?

We could have added a token, but for reasons of convention I made it a parser:

struct skipParser : qi::grammar<It> {
    skipParser() : skipParser::base_type(spaceOrComment) {
        using namespace qi;
        spaceOrComment = space
            | ("//" >> *(char_ - eol) >> (eoi|eol))
            | ("/*" >> *(char_ - "*/") >> "*/");

        BOOST_SPIRIT_DEBUG_NODES((spaceOrComment))
    }
  private:
    qi::rule<It> spaceOrComment;
};

natureParser

We inherit our AST parsers from tokens:

struct natureParser : tokens, qi::grammar<It, ast::nature(), skipParser> {

And from there it is plain sailing:

property = attribute >> '=' >> value >> ';';

nature
    = kw["nature"] >> identifier >> -(':' >> identifier) >> ';'
    >> *property
    >> kw["endnature"];

disciplineParser

discipline = kw["discipline"] >> identifier >> ';' 
    >> properties 
    >> kw["enddiscipline"]
    ;

properties
    = kw["domain"] >> domain >> ';'
    ^ kw["flow"] >> identifier >> ';'
    ^ kw["potential"] >> identifier >> ';'
    ;

This shows a competing approach that uses the permutation operator (^) to parse optional alternatives in any order into a fixed frank::ast::discipline properties struct. Of course, you might elect to have a more generic representation here, like we had with ast::nature.

Module AST is left as an exercise for the reader, though the parser rules are implemented below.

Top Level, Encapsulating The Skipper

I hate having to specify the skipper from the calling code (it's more complex than required, and changing the skipper changes the grammar). So, I encapsulate it in the top-level parser:

struct fileParser : qi::grammar<It, ast::file()> {
    fileParser() : fileParser::base_type(file) {
        file = qi::skip(qi::copy(m_sSkip)) [
                *(m_sDiscipline | m_sNature | m_sModule)
            ];

        BOOST_SPIRIT_DEBUG_NODES((file))
    }
  private:
    disciplineParser m_sDiscipline;
    natureParser     m_sNature;
    moduleParser     m_sModule;
    skipParser       m_sSkip;

    qi::rule<It, ast::file()> file;
};

Demo Time

This demo adds operator<< for the enums, and a variant visitor to print some AST details for debug/demonstrational purposes (print_em).

Then we have a test driver:

int main() {
    using iterator_type = std::string::const_iterator;

    iterator_type iter = sInput.begin(), last = sInput.end();

    frank::Parsers<iterator_type>::fileParser parser;
    print_em print;

    frank::ast::file file;
    bool ok = qi::parse(iter, last, parser, file);

    if (ok) {
        for (auto& symbol : file)
            print(symbol);
    }
    else {
        std::cout << "Parse failed\n";
    }

    if (iter != last) {
        std::cout << "Remaining unparsed: '" << std::string(iter,last) << "'\n";
    }
}

With the sample input from your question we get the following output:

Live On Coliru

-- Nature
name: Current
inherits: 
attribute: units = A
attribute: access = I
attribute: idt = Charge
attribute: abstol = 1e-12
-- Nature
name: Charge
inherits: 
attribute: units = coul
attribute: access = Q
attribute: ddt = Current
attribute: abstol = 1e-14
-- Nature
name: Voltage
inherits: 
attribute: units = V
attribute: access = V
attribute: idt = Flux
attribute: abstol = 1e-06
-- Discipline
name: electrical
domain: (unspecified)
flow:  Current
potential:  Voltage
Remaining unparsed: '
'

With BOOST_SPIRIT_DEBUG defined, you get rich debug information: Live On Coliru

Full Listing

Live On Coliru

//#define BOOST_SPIRIT_DEBUG
#include <map>
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/include/adapted.hpp>
#include <boost/spirit/repository/include/qi_distinct.hpp>

namespace qi = boost::spirit::qi;

namespace frank { namespace ast {
    struct nature {
        struct empty{};

        std::string name;
        std::string inherits;

        enum class Attribute { units, access, idt, ddt, abstol };
        using Value = boost::variant<int, double, std::string>;
        std::map<Attribute, Value> attributes;
    };

    struct discipline {
        enum enumDomain { eUnspecified, eDiscrete, eContinuous };
        struct properties_t {
            enumDomain domain = eUnspecified;
            boost::optional<std::string> flow, potential;
        };

        std::string name;
        properties_t properties;
    };

    // TODO
    using module = qi::unused_type;
    using file   = std::vector<boost::variant<nature, discipline, module> >;

    enum class type { real, integer, string };
} }

BOOST_FUSION_ADAPT_STRUCT(frank::ast::nature, name, inherits, attributes)
BOOST_FUSION_ADAPT_STRUCT(frank::ast::discipline, name, properties)
BOOST_FUSION_ADAPT_STRUCT(frank::ast::discipline::properties_t, domain, flow, potential)

namespace frank {
    namespace qr = boost::spirit::repository::qi;

    template <typename It> struct Parsers {

        struct tokens {
            // implicit lexemes
            qi::rule<It, std::string()> string, identifier;
            qi::rule<It, double()> real;
            qi::rule<It, int()> integer;
            qi::rule<It, ast::nature::Value()> value;
            qi::rule<It, ast::nature::Attribute()> attribute;
            qi::rule<It, ast::discipline::enumDomain()> domain;

            struct attribute_sym_t : qi::symbols<char, ast::nature::Attribute> {
                attribute_sym_t() {
                    this->add
                       ("units", ast::nature::Attribute::units)
                       ("access", ast::nature::Attribute::access)
                       ("idt_nature", ast::nature::Attribute::idt)
                       ("ddt_nature", ast::nature::Attribute::ddt)
                       ("abstol", ast::nature::Attribute::abstol);
                }
            } attribute_sym;

            struct domain_sym_t : qi::symbols<char, ast::discipline::enumDomain> {
                domain_sym_t() {
                    this->add
                       ("discrete", ast::discipline::eDiscrete)
                       ("continuous", ast::discipline::eContinuous);
                }
            } domain_sym;

            tokens() {
                using namespace qi;
                auto kw = qr::distinct(copy(char_("a-zA-Z0-9_")));

                string     = '"' >> *("\\" >> char_ | ~char_('"')) >> '"';
                identifier = char_("a-zA-Z_") >> *char_("a-zA-Z0-9_");
                real       = double_;
                integer    = int_;
                attribute  = kw[attribute_sym];
                domain     = kw[domain_sym];

                value = string | identifier | real | integer;

                BOOST_SPIRIT_DEBUG_NODES((string)(identifier)(real)(integer)(value)(domain)(attribute))
            }
        };

        struct skipParser : qi::grammar<It> {
            skipParser() : skipParser::base_type(spaceOrComment) {
                using namespace qi;
                spaceOrComment = space
                    | ("//" >> *(char_ - eol) >> (eoi|eol))
                    | ("/*" >> *(char_ - "*/") >> "*/");

                BOOST_SPIRIT_DEBUG_NODES((spaceOrComment))
            }
          private:
            qi::rule<It> spaceOrComment;
        };

        struct natureParser : tokens, qi::grammar<It, ast::nature(), skipParser> {
            natureParser() : natureParser::base_type(nature) {
                using namespace qi;
                auto kw = qr::distinct(copy(char_("a-zA-Z0-9_")));

                property = attribute >> '=' >> value >> ';';

                nature
                    = kw["nature"] >> identifier >> -(':' >> identifier) >> ';'
                    >> *property
                    >> kw["endnature"];

                BOOST_SPIRIT_DEBUG_NODES((nature)(property))
            }
          private:
            using Attribute = std::pair<ast::nature::Attribute, ast::nature::Value>;

            qi::rule<It, ast::nature(), skipParser> nature;
            qi::rule<It, Attribute(), skipParser> property;

            using tokens::attribute;
            using tokens::value;
            using tokens::identifier;
        };

        struct disciplineParser : tokens, qi::grammar<It, ast::discipline(), skipParser> {
            disciplineParser() : disciplineParser::base_type(discipline) {

                auto kw = qr::distinct(qi::copy(qi::char_("a-zA-Z0-9_")));

                discipline = kw["discipline"] >> identifier >> ';' 
                    >> properties 
                    >> kw["enddiscipline"]
                    ;

                properties
                    = kw["domain"] >> domain >> ';'
                    ^ kw["flow"] >> identifier >> ';'
                    ^ kw["potential"] >> identifier >> ';'
                    ;

                BOOST_SPIRIT_DEBUG_NODES((discipline)(properties))
            }
          private:
            qi::rule<It, ast::discipline(), skipParser> discipline;
            qi::rule<It, ast::discipline::properties_t(), skipParser> properties;

            using tokens::domain;
            using tokens::identifier;
        };

        struct moduleParser : tokens, qi::grammar<It, ast::module(), skipParser> {
            moduleParser() : moduleParser::base_type(module) {
                auto kw = qr::distinct(qi::copy(qi::char_("a-zA-Z0-9_")));

                m_sPort           = identifier;
                m_sPortList       = m_sPort % ',';
                m_sModulePortList = '(' >> m_sPortList >> ')';
                m_sModule         = kw["module"];
                m_sType           = kw["real"] | kw["integer"] | kw["string"];
                m_sParameter      = kw["parameter"] >> m_sType >> identifier;
                m_sModuleItem     = m_sParameter;
                m_sModuleItemList = *m_sModuleItem;
                module =
                    (m_sModule >> identifier >> m_sModulePortList >> m_sModuleItemList >> kw["endmodule"]);
            }
          private:
            qi::rule<It, ast::module(), skipParser> module;
            qi::rule<It, skipParser> m_sModulePortList;
            qi::rule<It, skipParser> m_sPortList;
            qi::rule<It, skipParser> m_sPort;
            qi::rule<It, skipParser> m_sModule;

            qi::rule<It, skipParser> m_sModuleItemList;
            qi::rule<It, skipParser> m_sParameter;
            qi::rule<It, skipParser> m_sModuleItem;
            qi::rule<It, skipParser> m_sType;

            using tokens::identifier;
        };

        struct fileParser : qi::grammar<It, ast::file()> {
            fileParser() : fileParser::base_type(file) {
                file = qi::skip(qi::copy(m_sSkip)) [
                        *(m_sDiscipline | m_sNature | m_sModule)
                    ];

                BOOST_SPIRIT_DEBUG_NODES((file))
            }
          private:
            disciplineParser m_sDiscipline;
            natureParser     m_sNature;
            moduleParser     m_sModule;
            skipParser       m_sSkip;

            qi::rule<It, ast::file()> file;
        };
    };

}

extern std::string const sInput;

// just for demo
#include <boost/optional/optional_io.hpp>

namespace frank { namespace ast {
    //static inline std::ostream &operator<<(std::ostream &os, const nature::empty &) { return os; }
    static inline std::ostream &operator<<(std::ostream &os, nature::Attribute a) {
        switch(a) {
            case nature::Attribute::units:  return os << "units";
            case nature::Attribute::access: return os << "access";
            case nature::Attribute::idt:    return os << "idt";
            case nature::Attribute::ddt:    return os << "ddt";
            case nature::Attribute::abstol: return os << "abstol";
        };
        return os << "?";
    }
    static inline std::ostream &operator<<(std::ostream &os, discipline::enumDomain d) {
        switch(d) {
            case discipline::eDiscrete:    return os << "discrete";
            case discipline::eContinuous:  return os << "continuous";
            case discipline::eUnspecified: return os << "(unspecified)";
        };
        return os << "?";
    }
} }

struct print_em {
    using result_type = void;
    template <typename V>
    void operator()(V const& variant) const {
        boost::apply_visitor(*this, variant);
    }
    void operator()(frank::ast::nature const& nature) const {
        std::cout << "-- Nature\n";
        std::cout << "name: " << nature.name << "\n";
        std::cout << "inherits: " << nature.inherits << "\n";
        for (auto& a : nature.attributes) {
            std::cout << "attribute: " << a.first << " = " << a.second << "\n";
        }
    }
    void operator()(frank::ast::discipline const& discipline) const {
        std::cout << "-- Discipline\n";
        std::cout << "name: " << discipline.name << "\n";
        std::cout << "domain: " << discipline.properties.domain << "\n";
        std::cout << "flow: " << discipline.properties.flow << "\n";
        std::cout << "potential: " << discipline.properties.potential << "\n";
    }
    void operator()(frank::ast::module const&) const {
        std::cout << "-- Module (TODO)\n";
    }
};

int main() {
    using iterator_type = std::string::const_iterator;

    iterator_type iter = sInput.begin(), last = sInput.end();

    frank::Parsers<iterator_type>::fileParser parser;
    print_em print;

    frank::ast::file file;
    bool ok = parse(iter, last, parser, file);

    if (ok) {
        for (auto& symbol : file)
            print(symbol);
    }
    else {
        std::cout << "Parse failed\n";
    }

    if (iter != last) {
        std::cout << "Remaining unparsed: '" << std::string(iter,last) << "'\n";
    }
}

std::string const sInput = R"(
nature Current;
  units        = "A";
  access       = I;
  idt_nature   = Charge;
  abstol       = 1e-12;
endnature

// Charge in coulombs
nature Charge;
  units      = "coul";
  access     = Q;
  ddt_nature = Current;
  abstol     = 1e-14;
endnature

// Potential in volts
nature Voltage;
  units      = "V";
  access     = V;
  idt_nature = Flux;
  abstol     = 1e-6;
endnature

discipline electrical;
  potential    Voltage;
  flow         Current;
enddiscipline
)";

---

¹ incidentally, the other answer there demonstrates the "impedance mismatch" with polymorphic attributes and Spirit - this time on the Karma side of it

² (to prevent subtle bugs that depend on evaluation order or things like that, e.g.)

³ (gleaning some from here but not importing too much complexity that wasn't reflected in your Lex approach)

⁴ (In fact, this is where you'd need state-switching inside the grammar, an area notoriously underdeveloped and practically unusable in Spirit Lex: e.g. when it works how to avoid defining token which matchs everything in boost::spirit::lex or when it goes badly: Boost.Spirit SQL grammar/lexer failure)

Answer 2

One solution would be to use a std::string everywhere and define a boost::variant with everything needed but not use it anywhere in the parser or lexer directly but only serialize & deserialize it into/from the string.

Is this what the originators of boost::spirit intended?