Unable to insert nested object in mongodb using pymongo

Question

I am coming today following an issue that doesn't make sense to me using python and mongodb. I am a Go/C# developer so maybe I am missing something but I have the following case:

from datetime import datetime
from bson import ObjectId 

class DailyActivity:
    user_ids = []
    date = None

    def __init__(self, user_ids : [ObjectId] = [], date : datetime = None):
        self.user_ids = user_ids
        self.date = date


class ActivitiesThroughDays:
    daily_activies = []

    def add_daily_activity(self, daily_activity : DailyActivity = None):
        daily_activies.append(daily_activity)

I then have these 2 classes but also another file containing some helper to use mongodb:

from pymongo import MongoClient

def get_client():
    return MongoClient('localhost', 27017)


def get_database(database_name: str = None):
    if database_name is None:
        raise AttributeError("database name is None.")
    return get_client().get_database(database_name)


def get_X_database():
    return get_database("X")

And here we get to the issue.. I am now building a simple ActivitiesThroughDays object which has only one DailyActivity containing X user ids (as ObjectId array/list).

However, when I try to insert_one, I get the following:

TypeError: document must be an instance of dict, bson.son.SON, bson.raw_bson.RawBSONDocument, or a type that inherits from collections.MutableMapping

this is the piece of code that raise the exception:

def insert_activities_though_days(activities_through_days: ActivitiesThroughDays = None):
    if activities_through_days is None:
        raise AttributeError("activities_through_days is None.")
    col = get_EM_column("activities_through_days")
    col.insert_one(activities_through_days)

---

Based on the above issue, I then tried to convert my ActivitiesThroughDays into dic/json:

col.insert_one(activities_through_days.__dict__)

bson.errors.InvalidDocument: cannot encode object: models. DailyActivity. DailyActivity object at 0x10eea0320, of type: class 'models. DailyActivity. DailyActivity'

col.insert_one(json.dumps(activities_through_days))

TypeError: Object of type ActivitiesThroughDays is not JSON serializable

So based on this, I began to search for different solutions over google and found out solutions such as :

def to_dict(obj):
    if not hasattr(obj,"__dict__"):
        return obj
    result = {}
    for key, val in obj.__dict__.items():
        if key.startswith("_"):
            continue
        element = []
        if isinstance(val, list):
            for item in val:
                element.append(to_dict(item))
        else:
            element = to_dict(val)
        result[key] = element
    return result

But I got :

bson.errors.InvalidDocument: cannot encode object: property object at 0x10229aa98, of type: class 'property'

---

For each step I move forward, another issue comes up... To me, all of this doesn't make sense at all because.. there should be a generic serializer/deserializer somewhere that would, from 1 line, convert any nested objects/arrays to be inserted in mongodb..

Also, from one of the solution I tried, I found out that ObjectId were ignored while mapping to json/dict (I don't remember which one)

I am not at all a Python developer so please, feel free to give any tips :)

Thanks

Answer 1

pymongo's interface expects dict and .__dict__ is a very low level attribute.

I'm afraid you'll spend a lot of energy if you try to build an ORM/ODM for mongodb from scratch.

There are existing ORM/ODM libraries that exist for mongodb in python ( mongoengine , pymodm which are quite similar) and that could help you to get something working quickly.

Here are a few lines that shows how the models would look with mongoengine and how to save them:

import datetime as dt
from mongoengine import *

connect(host='mongodb://localhost:27017/testdb')

class User(Document):
    email = EmailField(required=True)

class DailyActivity(Document):
    users = ListField(ReferenceField(User))
    date = DateTimeField(default=dt.datetime.utcnow)

user = User(email='test@garbage.com').save()
user2 = User(email='test2@garbage.com').save()

activity = DailyActivity(users=[user, user2]).save()

I hope this helps