boost::spirit recursive imperative c++ grammar: BOOST_FUSION_ADAPT_STRUCT fails

Question

I've been working on a grammar to parse imperative statements (if/else/do/while/for/switch etc.) in c++. All other statements are kept as strings. I'm currently only testing with if/else (though other statements should work analogous in a variant). Unfortunately I get a compile time error:

Error 1 error C2440: 'return' : cannot convert from 'std::vector< someSeqNode,std::allocator< Ty>>' to 'boost::fusion::vector< someSeqNode, boost::fusion::void , boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_, boost::fusion::void_> &'

on line 80 (closing bracket of BOOST_FUSION_ADAPT_STRUCT for ifElseStruct)

The only other questions I found regarding compile issues with BOOST_FUSION_ADAPT_STRUCT and attribute propagation were on structs with only one member or otherwise incompatible attributes between adapted struct and a rule.

It seems to fail on adapting ifElseStruct.ifContent but I don't get why. Resolving the typedef it really is just vector< variant< ifElseStruct, string>>.

Is the recursion the problem? If so, how do I solve this?

#define BOOST_SPIRIT_DEBUG
#define BOOST_SPIRIT_USE_PHOENIX_V3

#pragma region INCLUDE_STUFF
#include <vector>
#include <string>
#include <iostream>
//boost includes for parser and some collection types (e.g. tuple)
#include <boost/spirit/include/qi.hpp>
#include <boost/phoenix/fusion.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/fusion/include/boost_tuple.hpp>
#include <boost/optional.hpp>
#include <boost/optional/optional_io.hpp>
#include <boost/spirit/include/qi_raw.hpp>
#include <boost/variant.hpp>
#include "vectorStreamOp.h"//overload stream operator << for std::vector -> for BOOST_SPIRIT_DEBUG
#pragma endregion INCLUDE_STUFF

#pragma region NAMESPACE_STUFF
//to shorten calls fruther down
namespace phx = boost::phoenix;
namespace qi = boost::spirit::qi;
using std::string;
using std::cout;
using std::endl;
using std::vector;
using boost::spirit::qi::parse;
using boost::optional;
using phx::ref;
using phx::at_c;
using qi::char_;
using qi::lexeme;
using qi::_1;
using qi::lit;
using qi::alnum;
using qi::alpha;
using qi::space;
using qi::raw;
using qi::as_string;
#pragma endregion NAMESPACE_STUFF

#pragma region STRUCT_STUFF
/*later make this variant with all impStatementVariants -> either make this a vector to have sequences on all levels or make imperativeCpp derive from this
-> typedef variant<
recursive_wrapper<ifElseStruct>,
recursive_wrapper<switchStruct>,
recursive_wrapper<forStruct>,
recursive_wrapper<whileStruct>,
recursive_wrapper<doWhileStruct>
*/
struct ifElseStruct;
typedef boost::variant<ifElseStruct, string> someSeqNode;

struct ifElseStruct
{
    string ifCond;
    vector<someSeqNode> ifContent;
    optional<vector<someSeqNode>> elseContent;

    //for BOOST DEBUG
    friend std::ostream& operator<< (std::ostream& stream, const ifElseStruct& var) {
        stream << "ifCond: " << var.ifCond << "   ifContent: " << var.ifContent << endl << "elseContent:" << var.elseContent;
        return stream;
    }
};

BOOST_FUSION_ADAPT_STRUCT(
    ifElseStruct,
    (string, ifCond)
    (vector<someSeqNode>, ifContent)
    (optional<vector<someSeqNode>>, elseContent)
    )
#pragma endregion STRUCT_STUFF

#pragma region GRAMMAR_STUFF
    //GRAMMAR for flowcontrol (branching and looping)
    template<typename Iterator, typename Skipper> struct imperativeGrammar :qi::grammar<Iterator, vector<someSeqNode>(), Skipper>
{
    imperativeGrammar() : imperativeGrammar::base_type(startRule)
    {
        startRule = *(recursiveImpCpp | nestedSomething); //vector<variant<ifElseStruct(), string>>
        recursiveImpCpp = ifElseNormalRule.alias() /*| switchRule | whileRule | forRule ...*/;

        //attr: ifElseStruct containing-> string, vector<someSeqNode>, optional<vector<someSeqNode>>
        ifElseNormalRule = lit("if")>> '(' >> condition >> ')' >> ifElseContent >> -(lit("else") >> ifElseContent);

        condition = *~char_(")");//TODO: replace with nestedSomething rule
        ifElseContent = ('{' >> startRule >> '}') /*| singleStatement*/;

        singleStatement = !recursiveImpCpp >> (qi::as_string[*~char_(";")] >> qi::as_string[char_(';')]);
        nestedSomething = !recursiveImpCpp >> qi::as_string[*~char_("(){}")]
            >> -(raw['(' >> nestedSomething >> ')']) >> -(raw['{' >> nestedSomething >> '}'])
            >> !recursiveImpCpp >> qi::as_string[*~char_("(){}")];

        BOOST_SPIRIT_DEBUG_NODES((startRule)(ifElseNormalRule)(ifElseContent))
    }

    qi::rule<Iterator, vector<someSeqNode>(), Skipper> startRule;
    qi::rule<Iterator, ifElseStruct(), Skipper> recursiveImpCpp;
    qi::rule<Iterator, ifElseStruct(), Skipper> ifElseNormalRule;

    qi::rule<Iterator, string(), Skipper> condition;
    qi::rule<Iterator, vector<someSeqNode>(), Skipper> ifElseContent;
    qi::rule<Iterator, std::string(), Skipper> nestedSomething;
    qi::rule<Iterator, std::string(), Skipper> singleStatement;

    /*qi::rule<Iterator, Skipper> forRule;
    qi::rule<Iterator, Skipper> switchCaseBreakRule;
    qi::rule<Iterator, Skipper> whileRule;
    qi::rule<Iterator, Skipper> doWhileRule;*/
};
#pragma endregion GRAMMAR_STUFF

Answer 1

There's a number of issues.

1. Like the comment said, don't use using-directives ; they land you in trouble

2. if all you want to concatenate all source strings in nestedSomething then, just wrap it all in a raw[] (or as_string[raw[...]] but that's not even necessary), eg

    nestedSomething = !recursiveImpCpp >> qi::raw[*~char_("(){}") >> -('(' >> nestedSomething >> ')') >> -('{' >> nestedSomething >> '}') >> !recursiveImpCpp >> *~char_("(){}")]; 

3. that rule is broken in the sense that it will match an empty string. This makes the grammar never end (it will match an "infinite" amount of empty nestedSomething ). You will have to decide on some non-optional part. Here's a brute-force fix:

    qi::raw[...] [ qi::_pass = px::size(qi::_1) > 0 ]; 

   Even with that the grammar gets stuck in an infinite loop for non-trivial programs (try it with its own source). The following should clear things up a little:

    identifier_soup = !recursiveImpCpp >> +~qi::char_("(){}"); parenthesized = '(' >> -nestedSomething >> ')'; bracketed = '{' >> -nestedSomething >> '}'; nestedSomething %= qi::raw[ -identifier_soup >> -parenthesized >> -bracketed >> -identifier_soup] [ qi::_pass = px::size(qi::_1) > 0 ]; 

   But that will still not parse eg int main() { if(true) { std::cout << "Yes\\n"; } else { std::cout << "No\\n"; } } int main() { if(true) { std::cout << "Yes\\n"; } else { std::cout << "No\\n"; } } int main() { if(true) { std::cout << "Yes\\n"; } else { std::cout << "No\\n"; } } ). The reason is that main(<parenthesized>){<bracketed>} does only accept nestedSomething inside the brackets; that expressly prohibits the if-else construct...

   Let's rename ifElseContent to something proper (like statement )

    block = '{' >> startRule >> '}'; statement = block | singleStatement; 

   and use it instead of the bracketed :

    nestedSomething %= qi::raw[ -identifier_soup >> -parenthesized >> -block >> -identifier_soup] [ qi::_pass = boost::phoenix::size(qi::_1) > 0 ]; 

General various notes

• you can simplify the includes, instead of wrapping them in regions to hide them
• you can simplify the rest too (see my demo)
• the grammar is going to be quite inefficient with all the negative look-aheads. Consider tokenizing or using a much more finegrained keyword detection scheme (using expectation points and or distinct()[] from the Qi Repository)

Partial Demo

Applying the notes above:

Live On Coliru

#define BOOST_SPIRIT_DEBUG
#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/fusion/include/adapted.hpp>
#include <boost/optional/optional_io.hpp>

#ifdef BOOST_SPIRIT_DEBUG
    namespace std {
        template <typename... T, typename... V>
        auto& operator<<(basic_ostream<T...>& os, vector<V...> const& v) {
            os << "{";
            for (auto& el : v) os << el << " ";
            return os;
        }
    }
#endif

namespace qi  = boost::spirit::qi;

/* later make this variant with all impStatementVariants -> either make this a
 * vector to have sequences on all levels or make imperativeCpp derive from
 * this

    -> typedef variant<
        recursive_wrapper<ifElseStruct>,
        recursive_wrapper<switchStruct>,
        recursive_wrapper<forStruct>,
        recursive_wrapper<whileStruct>,
        recursive_wrapper<doWhileStruct>

 *
 */
struct ifElseStruct;
typedef boost::variant<ifElseStruct, std::string> someSeqNode;

struct ifElseStruct
{
    std::string ifCond;
    std::vector<someSeqNode> ifContent;
    boost::optional<std::vector<someSeqNode>> elseContent;

    friend std::ostream& operator<< (std::ostream& stream, const ifElseStruct& var) {
        stream << "ifCond: " << var.ifCond << " ifContent: " << var.ifContent << std::endl << "elseContent:" << var.elseContent;
        return stream;
    }
};

BOOST_FUSION_ADAPT_STRUCT(ifElseStruct, ifCond, ifContent, elseContent)

//GRAMMAR for flowcontrol (branching and looping)
template<typename Iterator, typename Skipper> struct imperativeGrammar :qi::grammar<Iterator, std::vector<someSeqNode>(), Skipper>
{
    imperativeGrammar() : imperativeGrammar::base_type(startRule)
    {
        startRule = *(recursiveImpCpp | nestedSomething); //vector<variant<ifElseStruct(), string>>
        recursiveImpCpp = ifElseNormalRule.alias() /*| switchRule | whileRule | forRule ...*/;

        //attr: ifElseStruct containing-> string, vector<someSeqNode>, optional<vector<someSeqNode>>
        ifElseNormalRule = qi::lit("if") >> '(' >> condition >> ')' >> statement >> -("else" >> statement);

        condition = *~qi::char_(")");//TODO: replace with nestedSomething rule
        block     = '{' >> startRule >> '}';
        statement = block | singleStatement;

        identifier_soup = !recursiveImpCpp >> +~qi::char_("(){}");
        parenthesized = '(' >> -nestedSomething >> ')';
        bracketed     = '{' >> -nestedSomething >> '}';
        nestedSomething %= qi::raw[ -identifier_soup
            >> -parenthesized
            >> -block
            >> -identifier_soup] [ qi::_pass = boost::phoenix::size(qi::_1) > 0 ];
        singleStatement = !recursiveImpCpp >> qi::raw[*~qi::char_(';') >> ';'];

        BOOST_SPIRIT_DEBUG_NODES((startRule)(ifElseNormalRule)(statement)(block)(nestedSomething)(identifier_soup)(parenthesized)(bracketed)(singleStatement))
    }

    qi::rule<Iterator, std::vector<someSeqNode>(), Skipper> startRule;
    qi::rule<Iterator, ifElseStruct(), Skipper> recursiveImpCpp;
    qi::rule<Iterator, ifElseStruct(), Skipper> ifElseNormalRule;

    qi::rule<Iterator, std::string(), Skipper> condition;
    qi::rule<Iterator, std::vector<someSeqNode>(), Skipper> block, statement;
    qi::rule<Iterator, std::string(), Skipper> nestedSomething;
    qi::rule<Iterator, std::string(), Skipper> singleStatement;
    qi::rule<Iterator, Skipper> identifier_soup, parenthesized, bracketed;

    /*qi::rule<Iterator, Skipper> forRule;
    qi::rule<Iterator, Skipper> switchCaseBreakRule;
    qi::rule<Iterator, Skipper> whileRule;
    qi::rule<Iterator, Skipper> doWhileRule;*/
};

#include <fstream>

int main() {
    //std::string const input = { std::istreambuf_iterator<char>(std::ifstream("main.cpp").rdbuf()), {} };
    std::string const input = "int main() { if(true) { std::cout << \"Yes\\n\"; } else { std::cout << \"No\\n\"; } }";

    using It = std::string::const_iterator;
    It f(input.begin()), l(input.end());

    imperativeGrammar<It, qi::space_type> p;
    std::vector<someSeqNode> rep;
    bool ok = phrase_parse(f, l, p, qi::space, rep);
    if (ok) {
        std::cout << "Parse success: " << rep << "\n";
    }
    else
        std::cout << "Parse failure\n";

    if (f!=l)
        std::cout << "Remaining unparsed input: '" << std::string(f,l) << "'\n";
}

Prints:

Parse success: {int main() { if(true) { std::cout << "Yes\n"; } else { std::cout << "No\n"; } } 

With debug info

<startRule>
  <try>int main() { if(true</try>
  <ifElseNormalRule>
    <try>int main() { if(true</try>
    <fail/>
  </ifElseNormalRule>
  <nestedSomething>
    <try>int main() { if(true</try>
    <identifier_soup>
      <try>int main() { if(true</try>
      <ifElseNormalRule>
        <try>int main() { if(true</try>
        <fail/>
      </ifElseNormalRule>
      <success>() { if(true) { std:</success>
      <attributes>[]</attributes>
    </identifier_soup>
    <parenthesized>
      <try>() { if(true) { std:</try>
      <nestedSomething>
        <try>) { if(true) { std::</try>
        <identifier_soup>
          <try>) { if(true) { std::</try>
          <ifElseNormalRule>
            <try>) { if(true) { std::</try>
            <fail/>
          </ifElseNormalRule>
          <fail/>
        </identifier_soup>
        <parenthesized>
          <try>) { if(true) { std::</try>
          <fail/>
        </parenthesized>
        <block>
          <try>) { if(true) { std::</try>
          <fail/>
        </block>
        <identifier_soup>
          <try>) { if(true) { std::</try>
          <ifElseNormalRule>
            <try>) { if(true) { std::</try>
            <fail/>
          </ifElseNormalRule>
          <fail/>
        </identifier_soup>
        <fail/>
      </nestedSomething>
      <success> { if(true) { std::c</success>
      <attributes>[]</attributes>
    </parenthesized>
    <block>
      <try> { if(true) { std::c</try>
      <startRule>
        <try> if(true) { std::cou</try>
        <ifElseNormalRule>
          <try> if(true) { std::cou</try>
          <statement>
            <try> { std::cout << "Yes</try>
            <block>
              <try> { std::cout << "Yes</try>
              <startRule>
                <try> std::cout << "Yes\n</try>
                <ifElseNormalRule>
                  <try> std::cout << "Yes\n</try>
                  <fail/>
                </ifElseNormalRule>
                <nestedSomething>
                  <try> std::cout << "Yes\n</try>
                  <identifier_soup>
                    <try>std::cout << "Yes\n"</try>
                    <ifElseNormalRule>
                      <try>std::cout << "Yes\n"</try>
                      <fail/>
                    </ifElseNormalRule>
                    <success>} else { std::cout <</success>
                    <attributes>[]</attributes>
                  </identifier_soup>
                  <parenthesized>
                    <try>} else { std::cout <</try>
                    <fail/>
                  </parenthesized>
                  <block>
                    <try>} else { std::cout <</try>
                    <fail/>
                  </block>
                  <identifier_soup>
                    <try>} else { std::cout <</try>
                    <ifElseNormalRule>
                      <try>} else { std::cout <</try>
                      <fail/>
                    </ifElseNormalRule>
                    <fail/>
                  </identifier_soup>
                  <success>} else { std::cout <</success>
                  <attributes>[[s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  ]]</attributes>
                </nestedSomething>
                <ifElseNormalRule>
                  <try>} else { std::cout <</try>
                  <fail/>
                </ifElseNormalRule>
                <nestedSomething>
                  <try>} else { std::cout <</try>
                  <identifier_soup>
                    <try>} else { std::cout <</try>
                    <ifElseNormalRule>
                      <try>} else { std::cout <</try>
                      <fail/>
                    </ifElseNormalRule>
                    <fail/>
                  </identifier_soup>
                  <parenthesized>
                    <try>} else { std::cout <</try>
                    <fail/>
                  </parenthesized>
                  <block>
                    <try>} else { std::cout <</try>
                    <fail/>
                  </block>
                  <identifier_soup>
                    <try>} else { std::cout <</try>
                    <ifElseNormalRule>
                      <try>} else { std::cout <</try>
                      <fail/>
                    </ifElseNormalRule>
                    <fail/>
                  </identifier_soup>
                  <fail/>
                </nestedSomething>
                <success>} else { std::cout <</success>
                <attributes>[[[s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  ]]]</attributes>
              </startRule>
              <success> else { std::cout <<</success>
              <attributes>[[[s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  ]]]</attributes>
            </block>
            <success> else { std::cout <<</success>
            <attributes>[[[s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  ]]]</attributes>
          </statement>
          <statement>
            <try> { std::cout << "No\</try>
            <block>
              <try> { std::cout << "No\</try>
              <startRule>
                <try> std::cout << "No\n"</try>
                <ifElseNormalRule>
                  <try> std::cout << "No\n"</try>
                  <fail/>
                </ifElseNormalRule>
                <nestedSomething>
                  <try> std::cout << "No\n"</try>
                  <identifier_soup>
                    <try>std::cout << "No\n";</try>
                    <ifElseNormalRule>
                      <try>std::cout << "No\n";</try>
                      <fail/>
                    </ifElseNormalRule>
                    <success>} }</success>
                    <attributes>[]</attributes>
                  </identifier_soup>
                  <parenthesized>
                    <try>} }</try>
                    <fail/>
                  </parenthesized>
                  <block>
                    <try>} }</try>
                    <fail/>
                  </block>
                  <identifier_soup>
                    <try>} }</try>
                    <ifElseNormalRule>
                      <try>} }</try>
                      <fail/>
                    </ifElseNormalRule>
                    <fail/>
                  </identifier_soup>
                  <success>} }</success>
                  <attributes>[[s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  ]]</attributes>
                </nestedSomething>
                <ifElseNormalRule>
                  <try>} }</try>
                  <fail/>
                </ifElseNormalRule>
                <nestedSomething>
                  <try>} }</try>
                  <identifier_soup>
                    <try>} }</try>
                    <ifElseNormalRule>
                      <try>} }</try>
                      <fail/>
                    </ifElseNormalRule>
                    <fail/>
                  </identifier_soup>
                  <parenthesized>
                    <try>} }</try>
                    <fail/>
                  </parenthesized>
                  <block>
                    <try>} }</try>
                    <fail/>
                  </block>
                  <identifier_soup>
                    <try>} }</try>
                    <ifElseNormalRule>
                      <try>} }</try>
                      <fail/>
                    </ifElseNormalRule>
                    <fail/>
                  </identifier_soup>
                  <fail/>
                </nestedSomething>
                <success>} }</success>
                <attributes>[[[s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  ]]]</attributes>
              </startRule>
              <success> }</success>
              <attributes>[[[s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  ]]]</attributes>
            </block>
            <success> }</success>
            <attributes>[[[s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  ]]]</attributes>
          </statement>
          <success> }</success>
          <attributes>[[[t, r, u, e], [[s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  ]], [[s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  ]]]]</attributes>
        </ifElseNormalRule>
        <ifElseNormalRule>
          <try> }</try>
          <fail/>
        </ifElseNormalRule>
        <nestedSomething>
          <try> }</try>
          <identifier_soup>
            <try>}</try>
            <ifElseNormalRule>
              <try>}</try>
              <fail/>
            </ifElseNormalRule>
            <fail/>
          </identifier_soup>
          <parenthesized>
            <try>}</try>
            <fail/>
          </parenthesized>
          <block>
            <try>}</try>
            <fail/>
          </block>
          <identifier_soup>
            <try>}</try>
            <ifElseNormalRule>
              <try>}</try>
              <fail/>
            </ifElseNormalRule>
            <fail/>
          </identifier_soup>
          <fail/>
        </nestedSomething>
        <success> }</success>
        <attributes>[[[[t, r, u, e], [[s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  ]], [[s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  ]]]]]</attributes>
      </startRule>
      <success></success>
      <attributes>[[[[t, r, u, e], [[s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  ]], [[s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  ]]]]]</attributes>
    </block>
    <identifier_soup>
      <try></try>
      <ifElseNormalRule>
        <try></try>
        <fail/>
      </ifElseNormalRule>
      <fail/>
    </identifier_soup>
    <success></success>
    <attributes>[[i, n, t,  , m, a, i, n, (, ),  , {,  , i, f, (, t, r, u, e, ),  , {,  , s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  , },  , e, l, s, e,  , {,  , s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  , },  , }]]</attributes>
  </nestedSomething>
  <ifElseNormalRule>
    <try></try>
    <fail/>
  </ifElseNormalRule>
  <nestedSomething>
    <try></try>
    <identifier_soup>
      <try></try>
      <ifElseNormalRule>
        <try></try>
        <fail/>
      </ifElseNormalRule>
      <fail/>
    </identifier_soup>
    <parenthesized>
      <try></try>
      <fail/>
    </parenthesized>
    <block>
      <try></try>
      <fail/>
    </block>
    <identifier_soup>
      <try></try>
      <ifElseNormalRule>
        <try></try>
        <fail/>
      </ifElseNormalRule>
      <fail/>
    </identifier_soup>
    <fail/>
  </nestedSomething>
  <success></success>
  <attributes>[[[i, n, t,  , m, a, i, n, (, ),  , {,  , i, f, (, t, r, u, e, ),  , {,  , s, t, d, :, :, c, o, u, t,  , <, <,  , ", Y, e, s, \, n, ", ;,  , },  , e, l, s, e,  , {,  , s, t, d, :, :, c, o, u, t,  , <, <,  , ", N, o, \, n, ", ;,  , },  , }]]]</attributes>
</startRule>

Summary/advice

If you ran the above sample on its own source you'll notice it bails on the namespace in line 9. Though it's nice that the program completes, rather than crashing, that's not encouraging.

You need a proper grammar . You need to THINK about what is a statement, what is a block, what is a keyword and what is the relationship between them. Part of your confusion seems to come from conflating expressions with statements .

I'd very much consider coming up with a grammar that you 1. can work 2. you know why it will work instead of ... seemingly just trying some things to cheat your way through parsing a programming language.

Have you even considered how you'd parse std::cout << "int main() { return 0; }"; yet?

Have you thought about macros with line-continuations?