将多个一对多关系合并到 json - oracle - python

Question

(I know this can be done using SQL Alchemy, but this is also an exercise in learning so I understand how it is done without tools like SQL Alchemy) My company is using a 10.1 oracle DB.

DB Structure Primary table (primary key) Second Table (foreign key with primary key) Third Table (foreign key with primary key)

Second table could have from 0 - 4 linked entries Third table could have 0 - infinite (though not likely to have more than 20)

My SQL query is running fine and getting me the data fine. If we get multiple entries in the other tables it will just return multiple rows with the same primary key. Seems standard (new to SQL so I assume this is how it should be).

The problem is when I try and process the data into a singular JSON entry looping over everything. Here is what I have currently as my script:

cursor = conn.cursor()

    sql = """
        SELECT *
        FROM RC_CAL 
               join CALENDAR_EVENT 
                 ON RC_CAL.rc_id = CALENDAR_EVENT.rc_id
               left outer join CALENDAR_EVENT CALENDAR_EVENT_1 
                            ON RC_CAL.rc_id = CALENDAR_EVENT_1.rc_id 
               left outer join RC_HISTORY RC_HISTORY_1 
                            ON RC_CAL.rc_id = RC_HISTORY_1.rc_id
    """

    has_where = False
    params = {}
    query_conditions = []
    if is_active:
        has_where = True
        query_conditions.append('WHERE CALENDAR_EVENT.ACTIVE_YN = :ACTIVE')
        params['ACTIVE'] = is_active
    if env:
        env_string = 'AND CALENDAR_EVENT.env = :ENV' if has_where else 'WHERE CALENDAR_EVENT.env = :ENV'
        has_where = True
        query_conditions.append(env_string)
        params['ENV'] = env
    if from_date:
        from_string = 'AND CALENDAR_EVENT.end_dt >= :FROM_DATE' if has_where else 'WHERE CALENDAR_EVENT.end_dt >= :FROM_DATE'
        has_where = True
        query_conditions.append(from_string)
        params['FROM_DATE'] = from_date
    if to_date:
        to_string = 'AND CALENDAR_EVENT.end_dt <= :TO_DATE' if has_where else 'WHERE CALENDAR_EVENT.end_dt <= :TO_DATE'
        query_conditions.append(to_string)
        params['TO_DATE'] = to_date

    if len(query_conditions) > 0:
        sql += " ".join(query_conditions)

    cursor.execute(sql, **params)

    rows = cursor.fetchall()

    events = []
    for rcId, data in groupby(rows, key=lambda t: t[0]):
        rc_item = None
        generic_list = []
        history_list = []
        for item in data:
            if not rc_item:
                rc_item = EventModalV2(*item[0:11])
            generic_id = item[11]
            history_id = item[17]
            generic_data = item[11:17]
            history_data = item[17:]
            if not any(generic_id in generic_item for generic_item in generic_list) and generic_id:
                rc_item.update_with_generic(generic_data)
                generic_list.append(generic_data)
            if not any(history_id in history_item for history_item in history_list) and history_id:
                rc_item.update_with_history(history_data)
                history_list.append(history_data)
                
        events.append(rc_item)

    cursor.close()
    conn.close()

    return events

an example dataset if I had multiple table 3 entries, though there could be more if table 2 gives more entries (where it says PROD, that is part of table 2. This is a table where we store dates depending on environment PROD:

(170079, 3, 1, None, None, None, 'Test Demo Event - DS', 'N', 'N', None, datetime.datetime(2020, 11, 5, 19, 16, 22), 237247, datetime.datetime(2020, 11, 1, 4, 0), 'PROD', datetime.datetime(2020, 11, 1, 4, 0), 91, 'Y', 5051, <cx_Orac
le.LOB object at 0x00000237AE8954B8>, 'rc_create', datetime.datetime(2020, 11, 5, 19, 16, 22))


(170079, 3, 1, None, None, None, 'Test Demo Event - DS', 'N', 'N', None, datetime.datetime(2020, 11, 5, 19, 16, 22), 237247, datetime.datetime(2020, 11, 1, 4, 0), 'PROD', datetime.datetime(2020, 11, 1, 4, 0), 91, 'Y', 5052, <cx_Orac
le.LOB object at 0x00000237AE895580>, 'rc_patch', datetime.datetime(2020, 11, 5, 19, 17, 7))


(170079, 3, 1, None, None, None, 'Test Demo Event - DS', 'N', 'N', None, datetime.datetime(2020, 11, 5, 19, 16, 22), 237247, datetime.datetime(2020, 11, 1, 4, 0), 'PROD', datetime.datetime(2020, 11, 1, 4, 0), 91, 'Y', 5053, <cx_Orac
le.LOB object at 0x00000237AE895648>, 'rc_patch', datetime.datetime(2020, 11, 5, 19, 17, 59))


(170079, 3, 1, None, None, None, 'Test Demo Event - DS', 'N', 'N', None, datetime.datetime(2020, 11, 5, 19, 16, 22), 237247, datetime.datetime(2020, 11, 1, 4, 0), 'PROD', datetime.datetime(2020, 11, 1, 4, 0), 91, 'Y', 5054, <cx_Orac
le.LOB object at 0x00000237AE895710>, 'rc_patch', datetime.datetime(2020, 11, 5, 19, 19, 9))

This is running extremely slow on larger queries. I am not sure how I can make this any better.

Answer 1

First, as Chris suggested, find out where the slowdown is occurring. Depending on what you find I have a couple suggestions:

(1) Avoid fetching LOB values as these require an additional round-trip to fetch the data, slowing down the process. You can see an example here .

(2) If the previous suggestion is insufficient, you can also consider simply fetching the three (four?) queries independently and putting them together in Python (by using dicts). This prevents the need for fetching the top-level rows multiple times and then removing the duplicates.