How to properly use sqlalchemy declarative_base with multiple model files

Question

.。 Answers to this question are eligible for a +50 reputation bounty. Learn more . Corne wants to to this question: I would like a clean design pattern that allows a single instance of declarative_base to be used across different model files. ：我想要一个简洁的设计模式，它允许在不同的 model 文件中使用 declarative_base 的单个实例。 So that a database_manager type class can create the entire database and its relations in such a way that the importing of models does not trigger PEP F401 unused import.

I want to use sqlalchemy declarative_base to create all tables and relations for my database, however, I have separated my models across multiple files and now fail to find a suitable architecture to use it.

Calling declarative_base returns a object which the models need to use as base class. However, it returns a new object every time it is called and the object needs to be the same for all models for it to work.

I have defined a small method for a database manager class which should construct the database. Naturally, for this to work the Base needs to be the same instance as used in all the models:

        engine = sqlalchemy.create_engine("sqlite:///%s" % file)
        create_database(engine.url) #sqlalchemy-utils
        Base.metadata.create_all(engine) # declarative_base

The problem arises at all my solutions either violating PEP F401 due to unused imports or causing circular dependencies. If I do not import the models without using them the class will not be defined using declarative_base and it won't be added to the database. While importing the models without using them will violate F401.

I fail to see how to create a clean construct with a shared declarative_base for all the models and the database manager without having unused imports. How can an instance of declarative_base be shared across models and database manager in such a way all tables will be created without unused imports?

Answer 1

After thoroughly read through your question I finally understand what you meant. The answer to your question worked out to be a lot simpler than I thought. I updated the answer to actually answer what you wanted. However, I still keep the previous examples as I thought you will benefit from them later on when you want to have a single instance of database manager.

For your programming problem, whether it is a single instance of Base or a single instance of database manager sharing across multiple files, it will all come down to a singleton design.

All 3 examples consist of 3 files, main.py, db.py, and main2.py. Example 1 is what you need for a single instance of the declarative Base. Example 1 will create test1.sqlite and test2.sqlite. Where example 2 and 3 will create "test1.sql" and "test2.sql". The previous example 1 and 2 now become 2 and 3.

These examples were not built package style but you can adapt them to your project easily. Also, another solution is to import with __init__.py.

For information on __init__.py, you can check here https://www.reddit.com/r/Python/comments/1bbbwk/whats_your_opinion_on_what_to_include_in_init_py/ and read a few answers here What is __init__.py for? .

For the examples, just run main.py.

---

Example 1

db.py:

# Don't mind the import, remix to your usage case.
from sqlalchemy.ext.declarative import declared_attr, declarative_base
from sqlalchemy import * 
import sqlalchemy_utils

# Declare attribute go here
class Base(object):
    @declared_attr
    def __tablename__(cls):
        return cls.__name__.lower()

    __table_args__ = {'mysql_engine': 'InnoDB'}

    id = Column(Integer, primary_key=True)

# The only single instance of Base from here that can be share 
# across all files.
Base = declarative_base(cls=Base)

# Other Schema for the design go here
# The first table will take on the name "MyModel"
# All this class derived from the single instance of Base above.
class MyModel(Base):
    name = Column(String(1000))

class MyModel2(Base):
    title = Column(String(1000))

class MyModel3(Base):
    lastname = Column(String(1000))

main2.py:

import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from db import *
import sqlalchemy_utils
import sqlalchemy

# Another Model from here that use the only single instance of Base from db.
class MyModel7(Base):
    surname = sqlalchemy.Column(sqlalchemy.types.String(1000))

def createDB():
    fileName = "test2.sqlite"
    engine = sqlalchemy.create_engine("sqlite:///%s" % fileName)
    sqlalchemy_utils.create_database(engine.url)
    tables_obj = [ MyModel.__table__,
                   MyModel2.__table__,
                   MyModel3.__table__,
                   MyModel7.__table__]
    Base.metadata.create_all(bind=engine, tables=tables_obj)

main.py:

import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from db import *
from main2 import *
import sqlalchemy_utils
import sqlalchemy

fileName = "test1.sqlite"

# Another Model from here that use the only single instance of Base from db.
class MyModel4(Base):
    lastname = sqlalchemy.Column(sqlalchemy.types.String(1000))

# Create a db file with multiple models "test1.sqlite" with the single 
# declarative Base from here.
engine = sqlalchemy.create_engine("sqlite:///%s" % fileName)
sqlalchemy_utils.create_database(engine.url)

# Creating tables
tables_obj = [ MyModel.__table__,
               MyModel2.__table__,
               MyModel3.__table__,
               MyModel4.__table__]
Base.metadata.create_all(bind=engine, tables=tables_obj)

# Create a db file "test2.sqlite" with 
# the single instance of dclarative Base from main2
createDB()

---

Example 2

db.py:

import sqlalchemy
import sqlalchemy_utils

class db():
    # These variable can be convert to protected, etc...
    engine = None
    fileName = None
    Base = None

    # Not __init__ because manual init
    def init(self, fileName):
        if self.fileName is not None: 
            raise Exception("Already initialize. Use setConfig() instead.")
        self.fileName = fileName
        self.Base = sqlalchemy.ext.declarative.declarative_base()
        '''
        The engine will also initialize on creation.
        '''
        self.initEngine()

    # Initialize the engine
    def initEngine(self):
        self.engine = sqlalchemy.create_engine("sqlite:///%s" % self.fileName)
        sqlalchemy_utils.create_database(self.engine.url) #sqlalchemy-utils

        # I don't even know why I called base here
        # I am just copying the question
        self.Base.metadata.create_all(bind=self.engine) # declarative_base

    # Set config. Note that more can be set here. This is just an example.
    def setConfig(self, fileName):
        self.fileName = fileName
        '''
        One the config is changed, it is probably preferable to reset the engine.
        Should garbage collector be run here? Does this consume too much memory 
        so that it benificial to call the garbage collector?
        '''
        self.initEngine()

   # These methods below are to demonstrate the singleton instance
    def createTable(self):
        conn = self.engine.connect()
        conn.execute("create table if not exists x (a integer, b integer)")
        conn.close()

    def insertData(self):
        conn = self.engine.connect()
        conn.execute("insert into x (a, b) values (1, 1)")
        conn.execute("insert into x (a, b) values (2, 2)")
        conn.execute("insert into x (a, b) values (3, 3)")
        conn.execute("insert into x (a, b) values (5, 5)")
        conn.close()

    def getData(self):
        conn = self.engine.connect()
        '''
        Convert result into list otherwise can't work with result on a closed
        connection
        '''
        result = [conn.execute('''
          select * from x;
          ''').fetchall()]
        conn.close()
        return result

    def getEngineUrl(self):
        return self.engine.url


db = db()

main2.py

import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from db import db

database = db

# Testing insert
def justTesting():
    print("Currently working with engine " + str(database.getEngineUrl()))
    database.insertData()

# Testing changing file
def changeFile():
    database.setConfig("test2.sql")
    database.createTable()

main.py

import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from db import db
from main2 import *

# Creating the db object. It need to be initialize 1.
database = db
database.init("test1.sql")

# Create the database table from here
database.createTable()

# Insert data into the table with main2
# Printing engine url with main2.
justTesting()

# get the result
results = database.getData()

for result in results:
    print(result)

# Change the database from main2.
changeFile()
print("Currently working with engine " + str(database.getEngineUrl()))

# Test the error on reinitialization
try:
    database.init("123.sql")
except Exception as e:
    print("\n" * 5)
    print("Error working fine. Just printing the message.")
    print(e)

---

Example 3

• Note that example 2 is updated to use __init__.py. For previous version prefer to the prior version of the answer.

db.py

import sqlalchemy
import sqlalchemy_utils

class db(object):
    # These variable can be convert to protected, etc...
    _instance = None
    engine = None
    fileName = None
    Base = None

    def __new__(cls, *args, **kwargs):
        if not cls._instance:
            cls._instance = object.__new__(cls)
        return cls._instance

    # __init__ any later object creation have to be the __new__(class)
    # or error will occur.
    def __init__(self, fileName):
        self.fileName = fileName
        self.Base = sqlalchemy.ext.declarative.declarative_base()
        '''
        The engine will also initialize on creation.
        '''
        self.initEngine()

    # Initialize the engine
    def initEngine(self):
        self.engine = sqlalchemy.create_engine("sqlite:///%s" % self.fileName)
        sqlalchemy_utils.create_database(self.engine.url) #sqlalchemy-utils

        # I don't even know why I called base here
        # I am just copying the question
        self.Base.metadata.create_all(bind=self.engine) # declarative_base

    # Set config. Note that more can be set here. This is just an example.
    def setConfig(self, fileName):
        self.fileName = fileName
        '''
        One the config is changed, it is probably preferable to reset the engine.
        Should garbage collector be run here? Does this consume too much memory 
        so that it benificial to call the garbage collector?
        '''
        self.initEngine()

   # These methods below are to demonstrate the singleton instance
    def createTable(self):
        conn = self.engine.connect()
        conn.execute("create table if not exists x (a integer, b integer)")
        conn.close()

    def insertData(self):
        conn = self.engine.connect()
        conn.execute("insert into x (a, b) values (1, 1)")
        conn.execute("insert into x (a, b) values (2, 2)")
        conn.execute("insert into x (a, b) values (3, 3)")
        conn.execute("insert into x (a, b) values (5, 5)")
        conn.close()

    def getData(self):
        conn = self.engine.connect()
        '''
        Convert result into list otherwise can't work with result on a closed
        connection
        '''
        result = [conn.execute('''
          select * from x;
          ''').fetchall()]
        conn.close()
        return result

    def getEngineUrl(self):
        return self.engine.url

main2.py

import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from db import db

# NOTE THAT THIS  IS THE BIGGEST DIFFERENT BETWEEN THE TWO SINGLETON
# EXAMPLE
database = db.__new__(db)

# Testing insert
def justTesting():
    print("Currently working with engine " + str(database.getEngineUrl()))
    database.insertData()

# Testing changing file
def changeFile():
    database.setConfig("test2.sql")
    database.createTable()

main.py:

import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from db import db
from main2 import *

# Creating the db object. It need to be initialize 1.
database = db("test1.sql")

# Create the database table from here
database.createTable()

# Insert data into the table from main2
# Using main2 to print the engine.url
justTesting()

# get the result
results = database.getData()

for result in results:
    print(result)

# Change the database from main2
changeFile()
print("Currently working with engine " + str(database.getEngineUrl()))