How to iterate over a `groupBy` returned object in Python?

Question

I have the following data input as a single string:

data = ['historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '343727655', '2019.11.14 21:24:01', '2019.11.14 21:24:01', '6', '0.01', '', '0.00000', '0.00000', '0.00000', '0.00000', '0.00', '0.00', '100000.00', '< >', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377716', '2019.11.27 21:07:22', '2019.12.06 17:14:03', '1', '0.20', 'GBPUSD', '1.29118', '1.31069', '0.00000', '0.00000', '0.00', '4.24', '-353.20', '<>', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377723', '2019.11.27 21:07:56', '2019.12.06 17:14:02', '1', '0.20', 'GBPUSD', '1.29117', '1.31070', '0.00000', '0.00000', '0.00', '4.24', '-353.57', '<>', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377724', '2019.11.27 21:08:00', '2019.12.02 18:38:14', '1', '0.20', 'GBPUSD', '1.29118', '1.29444', '0.00000', '0.00000', '0.00', '2.36', '-58.87', '<>', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '345200110', '2019.12.06 17:14:08', '2019.12.06 17:22:43', '0', '0.20', 'EURUSD', '1.10474', '1.10479', '0.00000', '0.00000', '0.00', '0.00', '0.91', '<>', 'historical_trades', '1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '345200125', '2019.12.06 17:14:21', '2019.12.06 17:22:45', '0', '4.00', 'EURUSD', '1.10483', '1.10479', '0.00000', '0.00000', '0.00', '0.00', '-14.48', '<>', '']

I use the Python itertool groupBy to extract the repeating strings/objects. I then want to push these extracted objects to the SQL database while each object represents one row.

            for key, group in groupby(data, key=lambda s: s != "historical_trades"):
                if key:

                    print(list(group))

                    # print output:

                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '343727655', '2019.11.14 21:24:01', '2019.11.14 21:24:01', '6', '0.01', '', '0.00000', '0.00000', '0.00000', '0.00000', '0.00', '0.00', '100000.00', '< >']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377716', '2019.11.27 21:07:22', '2019.12.06 17:14:03', '1', '0.20', 'GBPUSD', '1.29118', '1.31069', '0.00000', '0.00000', '0.00', '4.24', '-353.20', '<>']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377723', '2019.11.27 21:07:56', '2019.12.06 17:14:02', '1', '0.20', 'GBPUSD', '1.29117', '1.31070', '0.00000', '0.00000', '0.00', '4.24', '-353.57', '<>']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '344377724', '2019.11.27 21:08:00', '2019.12.02 18:38:14', '1', '0.20', 'GBPUSD', '1.29118', '1.29444', '0.00000', '0.00000', '0.00', '2.36', '-58.87', '<>']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '345200110', '2019.12.06 17:14:08', '2019.12.06 17:22:43', '0', '0.20', 'EURUSD', '1.10474', '1.10479', '0.00000', '0.00000', '0.00', '0.00', '0.91', '<>']
                    ['1.00', '30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f', '1925323', '345200125', '2019.12.06 17:14:21', '2019.12.06 17:22:45', '0', '4.00', 'EURUSD', '1.10483', '1.10479', '0.00000', '0.00000', '0.00', '0.00', '-14.48', '<>', '']

                    # Insert iteration logic here, to iterate returned (printed) object to execute the following statements.

                    [...]
                    [...]
                    s.th. like: "for every row in list(group):" ??

                    version = data[1]
                    DID = uuid.UUID(data[2])
                    accountNumber = int(data[3])
                    orderTicket = data[4]
                    orderOpenTime = data[5]
                    orderCloseTime = data[6]
                    orderType = float(data[7])
                    orderLots = float(data[8])
                    orderSymbol = data[9]
                    orderOpenPrice = float(data[10])
                    orderClosePrice = float(data[11])
                    orderStopLoss = float(data[12])
                    orderTakeProfit = float(data[13])
                    orderCommission = float(data[14])
                    orderSwap = float(data[15])
                    orderProfit = float(data[16])
                    orderComment = data[17]


                    # push the manipulated data to the PostgreSQL DB using `Trades` model
                    # If orderTicket exists, update values, if not create new entry

                    obj, created = Trades.objects.update_or_create(
                        orderTicket=orderTicket,
                        defaults={
                            'version': version,
                            'DID': DID,
                            'accountNumber': accountNumber,
                            'orderTicket': orderTicket,
                            'orderOpenTime': orderOpenTime,
                            'orderCloseTime': orderCloseTime,
                            'orderType': orderType,
                            'orderLots': orderLots,
                            'orderSymbol': orderSymbol,
                            'orderOpenPrice': orderOpenPrice,
                            'orderClosePrice': orderClosePrice,
                            'orderStopLoss': orderStopLoss,
                            'orderTakeProfit': orderTakeProfit,
                            'orderCommission': orderCommission,
                            'orderSwap': orderSwap,
                            'orderProfit': orderProfit,
                            'orderComment': orderComment
                        }
                    )

Now how can I access the groupby returned object and iterate over it to populate the database?

Working solution:

I tried to adapt @Tobi's solution into my code and it works properly now.

            for key, group in groupby(data, key=lambda s: s != "live_trades"):
                if key:

                    trades = list(group)
                    print(trades)

                    for data in trades:

                        version = trades[0]
                        print(version)
                        DID = uuid.UUID(trades[1])
                        accountNumber = int(trades[2])
                        print(accountNumber)
                        orderTicket = trades[3]
                        print(orderTicket)
                        orderOpenTime = trades[4]
                        orderCloseTime = trades[5]
                        orderType = float(trades[6])
                        orderLots = float(trades[7])
                        orderSymbol = trades[8]
                        orderOpenPrice = float(trades[9])
                        orderClosePrice = float(trades[10])
                        orderStopLoss = float(trades[11])
                        orderTakeProfit = float(trades[12])
                        orderCommission = float(trades[13])
                        orderSwap = float(trades[14])
                        orderProfit = float(trades[15])
                        orderComment = trades[16]
                        print(orderTicket)


                        # convert dateTime strings into Python comfort dateTime

                        orderOpenTime = datetime.strptime(orderOpenTime, '%Y.%m.%d %H:%M:%S')
                        orderCloseTime = datetime.strptime(orderCloseTime, '%Y.%m.%d %H:%M:%S')

                        # push the manipulated data to the PostgreSQL DB using `Trades` model
                        # If DID exists, update values, if not create new entry

                        obj, created = Trades.objects.update_or_create(
                            orderTicket=orderTicket,
                            defaults={
                                'version': version,
                                'DID': DID,
                                'accountNumber': accountNumber,
                                'orderTicket': orderTicket,
                                'orderOpenTime': orderOpenTime,
                                'orderCloseTime': orderCloseTime,
                                'orderType': orderType,
                                'orderLots': orderLots,
                                'orderSymbol': orderSymbol,
                                'orderOpenPrice': orderOpenPrice,
                                'orderClosePrice': orderClosePrice,
                                'orderStopLoss': orderStopLoss,
                                'orderTakeProfit': orderTakeProfit,
                                'orderCommission': orderCommission,
                                'orderSwap': orderSwap,
                                'orderProfit': orderProfit,
                                'orderComment': orderComment
                            }
                        )

Answer 1

The following should work just fine:

for data in group:
    version = data[0]
    ...

Two things to note:

• indexing starts with 0 , so the first element in data is [0] , the second is [1] and so on

• not only the result from groupby , but also the indivisual groups within it are iterators, so once you consume them, eg with print(list(group)) , they are exhausted (I guess that was the problem when you tried your loop)

   >>> k, g = next(groupby([1,1,1,2,2,2])) >>> list(g) # prints fine at first... [1, 1, 1] >>> list(g) # ... nothing happens in the loop afterwards? []

  Instead, you could print the current group in your inner loop, or collect all the groups in a list, ie group = list(group)

Also, instead of storing all the fields in data in 17 different variables and then writing them back to a dictionary, you could use zip to combine them with the correct names and types directly from data . (Only works if you need all the fields from data , but this seems to be the case here.)

fields = ('version','DID','accountNumber','orderTicket','orderOpenTime',
        'orderCloseTime','orderType','orderLots','orderSymbol','orderOpenPrice',
        'orderClosePrice','orderStopLoss','orderTakeProfit','orderCommission',
        'orderSwap','orderProfit','orderComment')

types = (str,uuid.UUID,int,str,str,str,float,float,str,float,float,float,
        float,float,float,float,str)

defaults = {f: t(d) for f,t,d in zip(fields, types, data)}
#{'orderSwap': 0.0, 'orderCloseTime': '2019.12.06 17:22:45', 'orderComment': '<>', 
# 'orderTakeProfit': 0.0, 'accountNumber': 1925323, 'DID': UUID('30e61aec-0f6e-4fa0-8c1b-eb07f9347c1f'), 
# 'orderType': 0.0, 'orderCommission': 0.0, 'orderLots': 4.0, 'orderOpenTime': '2019.12.06 17:14:21', 
# 'orderSymbol': 'EURUSD', 'orderOpenPrice': 1.10483, 'version': '1.00', 'orderProfit': -14.48, 
# 'orderStopLoss': 0.0, 'orderClosePrice': 1.10479, 'orderTicket': '345200125'}