Python: How to serialize objects for multiplayer game?

Question

We are working on a Top-Down-RPG-like Multiplayer game for learning purposes (and fun!) with some friends. We already have some Entities in the Game and Inputs are working, but the gives us headache :D 让我们头疼：D

The Issues

Already functional

• Client-Server connection
• Transfering JSON objects in both directions
• Async networking and synchronized putting into a Queue

Situation

A new player connects to the server and wants to get the current game state with all objects.

Data-Structure

We use a "Entity-Component" based architecture, so we separated the game logic very strictly into "systems", while the data is stored in the "components" of each Entity. The Entity is a very simple container and has nothing more than a ID and a list of components.

Example Entity (shorten for better readability):

Entity
      |-- Component (Moveable)
      |-- Component (Graphic)
      |         |- complex datatypes like pygame.SURFACE
      |         `- (...)
       `- Component (Inventory)

We tried different approaches, but all seems not to fit very well or feel "hacky".

pickle

Very Python near, so not easy to implement other clients in future. And I´ve read about some security risks when creating items from network in this dynamic way how pickle it offers. It does not even solve the Surface/Rectangle issue.

__dict__

Still contains the reference to the old objects, so a "cleanup" or "filter" for unwanted datatypes happens also in the origin. A deepcopy throws Exception.

 ...\\Python\\Python36\\lib\\copy.py", line 169, in deepcopy rv = reductor(4) TypeError: can't pickle pygame.Surface objects 

Show some code

The method of the "EnitityManager" Class which should generate the Snapshot of all Entities, including their components. This Snapshot should be converted to JSON without any errors - and if possible without much configuration in this core-class.

  class EnitityManager: def generate_world_snapshot(self): """ Returns a dictionary with all Entities and their components to send this to the client. This function will probably generate a lot of data, but, its to send the whole current game state when a new player connects or when a complete refresh is required """ # It should be possible to add more objects to the snapshot, so we # create our own Snapshot-Datastructure result = {'entities': {}} entities = self.get_all_entities() for e in entities: result['entities'][e.id] = deepcopy(e.__dict__) # Components are Objects, but dictionary is required for transfer cmp_obj_list = result['entities'][e.id]['components'] # Empty the current list of components, its going to be filled with # dictionaries of each cmp which are cleaned for the dump, because # of the errors directly coverting the whole datastructure to JSON result['entities'][e.id]['components'] = {} for cmp in cmp_obj_list: cmp_copy = deepcopy(cmp) cmp_dict = cmp_copy.__dict__ # Only list, dict, int, str, float and None will stay, while # other Types are being simply deleted including their key # Lists and directories will be cleaned ob recursive as well cmp_dict = self.clean_complex_recursive(cmp_dict) result['entities'][e.id]['components'][type(cmp_copy).__name__] \\ = cmp_dict logging.debug("EntityMgr: Entity#3: %s" % result['entities'][3]) return result 

Expectation and actual results

We can find a way to manually override elements which we dont want. But as the list of components will increase we have to put all the filter logic into this core class, which should not contain any components specializations.

Do we really have to put all the logic into the EntityManager for filtering the right objects? This does not feel good, as I would like to have all convertion to JSON done without any hardcoded configuration.



Thanks for reading so far and thank you very much for your help in advance!

Interesting articles which we were already working threw and maybe helpful for others with similar issues

• https://gafferongames.com/post/what_every_programmer_needs_to_know_about_game_networking/
• http://code.activestate.com/recipes/408859/
• https://docs.python.org/3/library/pickle.html

UPDATE: Solution - thx 2 sloth

We used a combination of the following architecture, which works really great so far and is also good to maintain!

Entity Manager now calls the get_state() function of the entity.

class Component:
    def __init__(self):
        logging.debug('generic component created')

    def get_state(self):
        state = {}
        for attr, value in self.__dict__.items():
            if value is None or isinstance(value, (str, int, float, bool)):
                state[attr] = value
            elif isinstance(value, (list, dict)):
                # logging.warn("Generating state: not supporting lists yet")
                pass
        return state

class GraphicComponent(Component):
   # (...)

The Entity has only some basic attributes to add to the state and forwards the get_state() call to all the Components:

 class Entity: def get_state(self): state = {'name': self.name, 'id': self.id, 'components': {}} for cmp in self.components: state['components'][type(cmp).__name__] = cmp.get_state() return state 

The components itself now inherit their get_state() method from their new superclass components, which simply cares about all simple datatypes:

 class Component: def __init__(self): logging.debug('generic component created') def get_state(self): state = {} for attr, value in self.__dict__.items(): if value is None or isinstance(value, (str, int, float, bool)): state[attr] = value elif isinstance(value, (list, dict)): # logging.warn("Generating state: not supporting lists yet") pass return state class GraphicComponent(Component): # (...) 

Now every developer has the opportunity to overlay this function to create a more detailed get_state() function for complex types directly in the Component Classes (like Graphic, Movement, Inventory, etc.) if it is required to safe the state in a more accurate way - which is a huge thing for maintaining the code in future, to have these code pieces in one Class.

Next step is to implement the static method for creating the items from the state in the same Class. This makes this working really smooth.
Thank you so much sloth for your help.

Answer 1

Do we really have to put all the logic into the EntityManager for filtering the right objects?

No, you should use polymorphism .

You need a way to represent your game state in a form that can be shared between different systems; so maybe give your components a method that will return all of their state, and a factory method that allows you create the component instances out of that very state.

(Python already has the __repr__ magic method, but you don't have to use it)

So instead of doing all the filtering in the entity manager, just let him call this new method on all components and let each component decide that the result will look like.

Something like this:

...
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
    result['entities'][e.id] = {'components': {}}
    for cmp in e.components:
         result['entities'][e.id]['components'][type(cmp).__name__] = cmp.get_state()
...

And a component could implement it like this:

class GraphicComponent:
    def __init__(self, pos=...):
        self.image = ...
        self.rect = ...
        self.whatever = ...

    def get_state(self):
        return { 'pos_x': self.rect.x, 'pos_y': self.rect.y, 'image': 'name_of_image.jpg' }

    @staticmethod
    def from_state(state):
        return GraphicComponent(pos=(state.pos_x, state.pos_y), ...)

And a client's EntityManager that recieves the state from the server would iterate for the component list of each entity and call from_state to create the instances.