Multiple databases

[ph55]

Hi,

Does beanie supports working with multiple databases ?
I see that following example working, but is it considered "good practice" ?

        await init_beanie(database=client.db_one, document_models=[DbOneModelX, DbOneModelY])
        await init_beanie(database=client.db_two, document_models=[DbTwoModelX])

[roman-right]

Hi,
There is no "good practice" option for this now. I'm working on a new initializer atm - plan to add db swaps there.

[ph55]

Hi, thanks for the answer.

But what happens behind the scenes in following case:

await init_beanie(database=client.db_one, document_models=[DbOneModelX, DbOneModelY])
await init_beanie(database=client.db_two, document_models=[DbTwoModelX])

Because it does work and I am able to operate with all the models on different databases.

[roman-right]

Yes. This will work correctly. It will not face any internal conflict

[mg3146]

Thanks for this - I'd be keen for db swaps as well (a test, and prod)

[rollebolle123]

Hi!
But this would not work with async calls?
Consider a fastapi async endpoint, which in turn does other calls, for example HTTP calls (Which may or may not be executed, take different time to run etc).
I tried to simulate this with an endpoint with configurable sleep:

# concept
client = AsyncIOMotorClient(database_url)
@a_router.get("/", response_model=SomethingCool)
async def get_something_cool(sleep_time: int, db: str):
await init_beanie(db)
await asyncio.sleep(sleep_time)
await SomethingCool.find()

If the endpoint is being called multiple times from different consumers, the global DB instance would be updated, fetching the wrong documents for the wrong consumer.

Even if init_beanie runs precisely before every call to example fetch documents, it could still end up fetching the wrong documents? :
@a_router.get("/", response_model=SomethingCool)
async def get_something_cool(sleep_time: int, db: str):
await init_beanie(db)
await SomethingCool.find()
await init_beanie(db)
await SomethingElse.find()

[treylade]

Is there any update on the db swap feature you talked about? I would like to build a multi-tenant architecture using Beanie (actually Bunnet) using separate databases per tenant but the same Document classes.

At the moment I call init_bunnet on the current tenant db for every new request made in a Flask environment. This works. But I suppose there might be a better off the shelf solution.

[Vwake04]

In the provided snippet, all instances of XX share the same document class, causing the connection session to be reinitialized each time init_beanie is called.

import asyncio
from motor.motor_asyncio import AsyncIOMotorClient
from beanie import init_beanie, Document


class XX(Document):
    v: int
    db_name: str = "test"


    class Settings:
        collection_name = "XX"


async def init_db(db_name):
    client = AsyncIOMotorClient("mongodb://localhost:27017")
    await init_beanie(database=client[db_name], document_models=[XX])
    
    xs: Document = XX(v=1, db_name=db_name)
    await xs.insert()


async def main():
    f = [
        init_db("abc#1"),
        init_db("abc#2"),
        init_db("abc#3"),
        init_db("abc#4"),
        init_db("abc#5"),
        init_db("abc#6"),
    ]
    await asyncio.gather(*f)


if __name__ == "__main__":
    asyncio.run(main())

[Yalchin403]

@roman-right I'm using multitenant-architecture and this setup you have provided is working fine, unless we have a concurrent requests, to reproduce it, you can set up two simple endpoints (FastAPI) that deal with 2 different databases, then you can try to set up postman runner to make concurrent requests. In this case we endpoint with the following error:

raise InvalidOperation("Can only use session with the MongoClient that started it")

Additionally calling init_beanie for every switch causes performance issues

[ajordanb]

@roman-right any updates on this?

[frenzyk]

I suggest Django approach could be good for adoption. I mean DB mapping and use .using('db_map_name') in queries.