Issues with using python metaclasses and inheritence

Question

I've been working on a metaclass layout for a project whereby all classes use a custom metaclass that loads config they define, along with that of parents. Basically each class defines a nested Config class which is loaded to a dict, then children can also define one and the class uses all parent config with any new values overwritten.

It works nicely when I don't remove the Config class after loading it to a dict, but now I am trying to refactor and clear up the namespace and it causes issues. The new (broken) code is as follows:

class AbstractConfigMeta(ABCMeta):
    """Parse nested Config classes and fill a new classes config dict."""

    def __new__(mcs, name, bases, namespace):
        """Traverse the MRO backwards from object to load all Config classes.

        Any config declared in sub classes overwrites base classes.
        """
        # get any config defined in parent classes first
        config = {}
        for parent in reversed(bases):
            if hasattr(parent, "config"):
                config.update(parent.config)
        # pop Config class and add values if defined
        config_class = namespace.pop("Config", None)
        if config_class:
            # get all non-magic (i.e. user-defined) attributes
            attributes = {
                key: value
                for key, value in config_class.__dict__.items()
                if not key.startswith("__")
            }
            config.update(attributes)

        namespace["config"] = config
        return super().__new__(mcs, name, bases, namespace)

Which parses the Config class when used but now does not use any config from parents. The old code which worked but kept the nested classes after instantiation is:

class AbstractConfigMeta(ABCMeta):
    """Parse nested Config classes and fill a new classes config dict."""

    def __new__(mcs, name, bases, namespace):
        """Traverse the MRO backwards from object to load all Config classes.

        Any config declared in sub classes overwrites base classes.
        """
        new_class = super().__new__(mcs, name, bases, namespace)
        new_class.config = {}  # type: ignore

        for parent in reversed(new_class.__mro__):
            config_class = getattr(parent, "Config", None)
            if config_class:
                # get all non-magic attributes from each Config class
                values = {
                    key: value
                    for key, value in config_class.__dict__.items()
                    if not key.startswith("__")
                }
                new_class.config.update(values)  # type: ignore
        return new_class

It seems by now trying to access the config using the dict created by the metaclass, parent config is discarded. Any help would be much appreciated.

Update

The issue turned out to be caused by some Mixins that use nested Config classes but don't use the metaclass. This was fine in the old code block, but when changing to getting parent config from the config dict instead of the nested class, anything not using the metaclass will not have this defined, so instead has a Config class whose values are not used.

Final working code including fixes and covering edge cases suggested by jsbueno:

class AbstractConfigMeta(ABCMeta):
    """Parse nested Config classes and fill a new classes config dict."""

    def __new__(mcs, name, bases, namespace):
        """Traverse the MRO backwards from object to load any config dicts.

        Any Config class declared in sub classes overwrites parent classes.
        """
        # pop Config class and add its attributes if defined
        config_class = namespace.pop("Config", None)
        if config_class:
            # get all non-magic (i.e. user-defined) attributes
            attributes = {
                key: value
                for key, value in config_class.__dict__.items()
                if not key.startswith("__")
            }
            if namespace.get("config"):
                warnings.warn(
                    f"A config dict and a config class are defined for {name}."
                    + " Any values in the config dict will be overwritten."
                )
            namespace["config"] = attributes

        new_class = super().__new__(mcs, name, bases, namespace)
        # get any config dicts defined in the MRO (including the current class)
        config = {}
        for parent in reversed(new_class.__mro__):
            if hasattr(parent, "config"):
                config.update(parent.config)  # type: ignore

        new_class.config = config  # type: ignore
        return new_class

Answer 1

The problem is that in the new code you are interacting over the class explicit bases , while the old (working) code iterates over __mro__ .

bases will yield only the explicit declared ancestors, and any "grandparents" or classes in a more complex hierarchy will not be visited.

The way to go is to allow Python generate the __mro__ by actually creating your new class, and iterating to retrieve your config keys on the new class. The config attribute can just be set on the newly created class - no need to do that in the namespace.

What is not recommended is to try to replicate Python's __mro__ - it is a rather complex algorithm, and even if you follow step by step to get it right, you will be just reinventing the wheel.

So, something along:

class AbstractConfigMeta(ABCMeta):
    """Parse nested Config classes and fill a new classes config dict."""

    def __new__(mcs, name, bases, namespace):
        """Traverse the MRO backwards from object to load all Config classes.

        Any config declared in sub classes overwrites base classes.
        """


        config_class = namespace.pop("Config", None)

        cls = super().__new__(mcs, name, bases, namespace)
        # get any config defined in parent classes first
        config = {}

        for parent in reversed(cls.__mro__):
            # Keep in mind this also runs for `cls` itself, so "config" can
            # also be specced as a dictionary. If you don't want that
            # to be possible, place a condition here to raise if `parent is cls and hasattr...`
            if hasattr(parent, "config"):
                config.update(parent.config)
        # pop Config class and add values if defined

        if config_class:
            # get all non-magic (i.e. user-defined) attributes
            attributes = {
                key: value
                for key, value in config_class.__dict__.items()
                if not key.startswith("__")
            }
            config.update(attributes)

        cls.config = config
        return cls