Boost Spirit-使用-O1构建项目，但不使用-O2构建项目

Question

I have a small project (similar to their example - https://github.com/cierelabs/x3_fun/ ) using Boost Spirit X3 and it builds in debug but not in release.

X3 program structure recommendation + another example: https://www.boost.org/doc/libs/1_69_0/libs/spirit/doc/x3/html/spirit_x3/tutorials/minimal.html

compiler: GCC 8.2.0

OS: Windows 10

After git bisect and some project configuration experiments I narrowed the problem down to compiler optimization flag.

this builds:

g++ -std=c++17 -O1 -pedantic -Wall -Wextra -c -fmessage-length=0

this does not build:

g++ -std=c++17 -O2 -pedantic -Wall -Wextra -c -fmessage-length=0

The error I get:

C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ./src/main.o:main.cpp:(.text+0x2156): undefined reference to `bool fs::parser::parse_rule<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::spirit::x3::unused_type, boost::spirit::x3::unused_type const>(boost::spirit::x3::rule<fs::parser::whitespace_class, boost::spirit::x3::unused_type, false>, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::spirit::x3::unused_type const&, boost::spirit::x3::unused_type const&)'

It looks like the optimizer is removing parse_rule definition (which is a core function generated from Spirit) which is actually needed.

nm -C main.o dump (left is -O2 , right -O1 ): https://www.diffchecker.com/YAFmJkbN

nm -C grammar.o dump (left is -O2 , right -O1 ): https://www.diffchecker.com/6dUgMiEK

nm -C utility.o dump (left is -O2 , right -O1 ): https://www.diffchecker.com/mXGf2Hx7 (no difference except addresses)

Initially I thought the problem is similar to c++ linker error: undefined references only on optimized build but I have no templates in source files, apart from what Boost might define in their macros.

What can be the cause of the optimizer erroneously removing function definition?

Code:

grammar.hpp

#pragma once
#include <boost/spirit/home/x3.hpp>
#include <tuple>
#include <utility>

namespace fs::parser
{

namespace x3 = boost::spirit::x3;
// symbols that denote filter's language constants
struct booleans_ : x3::symbols<bool>
{
    booleans_()
    {
        add
            ("True", true)
            ("False", false)
        ;
    }

};
const booleans_ booleans;

// some constants to aovid code duplication
constexpr auto keyword_boolean = "Boolean";
constexpr auto keyword_number  = "Number";
constexpr auto keyword_true    = "True";
constexpr auto keyword_false   = "False";
constexpr auto assignment_operator = '=';
constexpr auto newline_character   = '\n';

auto set_first = [](auto& ctx){ _val(ctx).first += _attr(ctx); };
auto set_second = [](auto& ctx){ _val(ctx).second += _attr(ctx); };

using x3::raw;
using x3::lexeme;
using x3::alpha;
using x3::alnum;
using x3::char_;
using x3::lit;

// comment - a line that starts with #
using comment_type = x3::rule<class comment_class>;
BOOST_SPIRIT_DECLARE(comment_type)

// identifier
using identifier_type = x3::rule<class identifier_class, std::string>;
BOOST_SPIRIT_DECLARE(identifier_type)

// whitespace
// Filter Spirit grammar skips any whitespace except newline character
using whitespace_type = x3::rule<class whitespace_class>;
BOOST_SPIRIT_DECLARE(whitespace_type)

// string
using string_type = x3::rule<class string_class, std::string>;
BOOST_SPIRIT_DECLARE(string_type)

// boolean definition: Boolean b = True
using constant_boolean_definition_type = x3::rule<class constant_boolean_definition_class, std::pair<std::string, bool>>;
BOOST_SPIRIT_DECLARE(constant_boolean_definition_type)

// number definition: Number n = 3
using constant_number_definition_type = x3::rule<class constant_number_definition_class, std::pair<std::string, int>>;
BOOST_SPIRIT_DECLARE(constant_number_definition_type)

// constants
using constant_definition_type = x3::rule<class constant_definition_class, boost::variant<constant_boolean_definition_type::attribute_type, constant_number_definition_type::attribute_type>>;
BOOST_SPIRIT_DECLARE(constant_definition_type)

// filter language consists of lines, of which every is a comment or some code
using code_line_type = x3::rule<class code_line_class, boost::optional<constant_definition_type::attribute_type>>;
BOOST_SPIRIT_DECLARE(code_line_type)

// the entire language grammar
using grammar_type = x3::rule<class grammar_class, std::vector<code_line_type::attribute_type>>;
BOOST_SPIRIT_DECLARE(grammar_type)

using skipper_type = whitespace_type;

// Boost Spirit recommends that this should be in a separate config.hpp file but
// in our case we need skipper type to be visible so we place configuration here
using iterator_type = std::string::const_iterator;
using context_type = x3::phrase_parse_context<skipper_type>::type;


}

namespace fs
{

parser::grammar_type grammar();
parser::skipper_type skipper();

}

grammar_def.hpp

#pragma once
#include "grammar.hpp"

namespace fs::parser
{

const comment_type comment = "comment";
const auto comment_def = lexeme['#' >> *(char_ - newline_character)];
BOOST_SPIRIT_DEFINE(comment)

const identifier_type identifier = "identifier";
const auto identifier_def = lexeme[(alpha | '_') >> *(alnum | '_')];
BOOST_SPIRIT_DEFINE(identifier)

const whitespace_type whitespace = "whitespace";
const auto whitespace_def = x3::space - newline_character;
BOOST_SPIRIT_DEFINE(whitespace)

const string_type string = "string";
const auto string_def = lexeme['"' >> +(char_ - '"') >> '"'];
BOOST_SPIRIT_DEFINE(string)

const constant_boolean_definition_type constant_boolean_definition = "Boolean definition";
const auto constant_boolean_definition_def = lit(keyword_boolean) >> identifier[set_first] >> lit(assignment_operator) >> booleans[set_second];
BOOST_SPIRIT_DEFINE(constant_boolean_definition)

const constant_number_definition_type constant_number_definition = "Number definition";
const auto constant_number_definition_def = lit(keyword_number) >> identifier[set_first] >> lit(assignment_operator) >> x3::int_[set_second];
BOOST_SPIRIT_DEFINE(constant_number_definition)

const constant_definition_type constant_definition = "value definition";
const auto constant_definition_def = constant_boolean_definition | constant_number_definition;
BOOST_SPIRIT_DEFINE(constant_definition)

const code_line_type code_line = "line";
const auto code_line_def = newline_character | comment | constant_definition;
BOOST_SPIRIT_DEFINE(code_line)

const grammar_type grammar = "code";
const auto grammar_def = *code_line;
BOOST_SPIRIT_DEFINE(grammar)

}

namespace fs
{

inline
parser::grammar_type grammar()
{
    return parser::grammar;
}

inline
parser::skipper_type skipper()
{
    return parser::whitespace;
}

}

grammar.cpp

#include "grammar_def.hpp"

namespace fs::parser
{

BOOST_SPIRIT_INSTANTIATE(grammar_type, iterator_type, context_type)

}

utility.hpp

#pragma once

#include <string>
#include <optional>

namespace fs::utility
{

bool file_exists(const char* path);
inline
bool file_exists(const std::string& path) { return file_exists(path.c_str()); }

std::optional<std::string> load_file(const char* file_path);
inline
std::optional<std::string> load_file(const std::string& file_path) { return load_file(file_path.c_str()); }

}

utility.cpp

#include "utility.hpp"
#include <fstream>
#include <sys/types.h>
#include <sys/stat.h>

namespace fs::utility
{

bool file_exists(const char* path)
{
    struct stat buffer;
    return stat(path, &buffer) == 0;
}

std::optional<std::string> load_file(const char* file_path)
{
    std::ifstream file(file_path);

    if (!file.good())
        return {};

    return std::string(std::istreambuf_iterator<char>{file}, {});
}

}

main.cpp

#include "utility.hpp"
#include "grammar.hpp"

#include <iostream>

bool parse(std::string const& source)
{
    namespace x3 = boost::spirit::x3;

    fs::parser::iterator_type it = source.begin();
    const fs::parser::iterator_type end = source.end();
    fs::parser::grammar_type::attribute_type data;

    const bool result = x3::phrase_parse(
        it,
        end,
        fs::grammar(),
        fs::skipper(),
        data
    );

    if (it != end) // fail if we did not get a full match
    {
        std::cout << "parse failure at pos: " << it - source.begin() << "\n";
        return false;
    }

    return result;
}


int main(int argc, char* argv[])
{
    if (argc < 2)
    {
        std::cout << "usage: " << "./program filepath\n";
        return -1;
    }

    if (parse(fs::utility::load_file(argv[1]).value()))
    {
        std::cout << "parse successful\n";
    }
    else
    {
        std::cout << "parse failure\n";
    }
}

Answer 1

I have managed to fix it myself.

Solution: add #include "grammar_def.hpp" to main.cpp .

What was happenning: BOOST_SPIRIT_DEFINE declares inline function templates. All are stripped away if they are not immediately used in any given translation unit. They need to be visible in the same TU where x3::phrase_parse is used or otherwise they will be considered unneeded by the compiler - hence undefined reference error.

Edit: The source of the problem is a bug in Spirit. This answer is just a workaround of TU mechanism. See other answer for the proper solution.

Answer 2

1. The parser::grammar_type grammar() and parser::skipper_type skipper() are not exported (I got linker error about this, but you have not mention it in your question). Move them to grammar.cpp (and remove inline on them of course) to solve the issue.

2. Missing skipper instantiation (the only error message you have provided is about that). Placing BOOST_SPIRIT_INSTANTIATE(whitespace_type, iterator_type, boost::spirit::x3::unused_type) into grammar.cpp should solve the problem, but...

   But the error message will not gone, and that's most likely because of a bug in Spirit. It instantiates parse_rule with an attribute type of the rule, and default is non-const unused_type , but it looks like somewhere at skipping it takes the attribute by const ref. To workaround this specify the skipper rule with a const attribute type x3::rule<class whitespace_class, boost::spirit::x3::unused_type const> .

The result https://gist.github.com/Kojoley/f7766422a5b6fd9e34b4b9d434325336 .

Update : The PR https://github.com/boostorg/spirit/pull/437 will fix the issue for everyone.