基于列名的加法在一个巨大的 dataframe pandas

Question

I have thousands of columns in the alldata dataframe. The column names consist of the elements like A_B_C or A_B_D and so on. I already have A , B , C in the same dataframe and in other dataframes separately like df_A , df_B , df_C , df_D

Iterating for getting the sum of A, B and C and check them if the list of sum of A,B,C or A,B,D is less than 0 on any row, doesn't look like a good idea as it taking unlimited time. Not sure where is the issue.

Here's my code. How should it be optimized?

res1 is the list of combinations for A_B_C and more

    for i in res1:
        x = i.split("_")
        alldata['sum'] = alldata[x[0]]+alldata[x[1]]+alldata[x[2]]
        if sum(n < 0 for n in alldata['sum']) >0:
            c=""
            print("nah")
        else:
            nice = [x[0],x[1],x[2]]
            good = good.append(nice)
            print(nice)
        alldata = alldata.drop([i], axis=1)
        print("dropped," + str(len(alldata.columns)) + "columns remaining")

Answer 1

Let's look at what your code is doing, line by line:

res1 = ['A_B_C', 'A_B_D']
for i in res1:

How long is your actual res1 ? If it is very long (many thousands), this for loop is always going to take a while. By the way, i is a terrible variable name for a multipart string (how about grouped_names or something?).

    x = i.split("_")

That won't take much time at all, assuming each string is short as in your example.

    alldata['sum'] = alldata[x[0]] + alldata[x[1]] + alldata[x[2]]

The good news is the above is vectorized, so it will run at native speed (as Pandas Series addition and assignment are implemented in compiled code). But how many rows does alldata have? That's the main factor in performance for the above line.

    if sum(n < 0 for n in alldata['sum']) > 0:
        c=""
        print("nah")

You don't use c anywhere else, remove it. And what's the print statement about? I'd remove that too.

    else:
        nice = [x[0],x[1],x[2]]
        good = good.append(nice)
        print(nice)

Just say nice = x[:3] to take the first three elements instead of creating a new list.

    alldata = alldata.drop([i], axis=1)

This doesn't seem great. You're creating a new DataFrame every time, Instead: how about this:

    drop_cols.append(i)

And then do alldata.drop(drop_cols, axis=1, inplace=True) at the very end, just once.

    print("dropped," + str(len(alldata.columns)) + "columns remaining")

You can simplify that a little:

    print("dropped,", len(alldata.columns), "columns remaining")

Once you've tried the above, let us know how long the code takes on some reasonably-sized DataFrame (one which contains real data, but perhaps not all the rows). Then tell us how much more speedup you need to make the solution acceptable.

Answer 2

I think @John Zwinck covered all the right points...

Suggest using df.apply() to sum and then issue raise Exception to stop when find 1st failure so should speed up on failures. If all successes, then sum of df's is better approach, IMO.

def testColumns(row, colName):
    s_ = sum(r[c] for c in colName.split('_')) 
    if s_ < 0 : 
        raise Exception 'nah'
    return True

for col in columnSet:
    sName = 's_'%str(col)
    try: 
        df[sName] = df.apply(lambda r : testColumns(r, col), axis=1 )
        msg = "col Kept  "
        nice = colName.split('_')
        good.append(nice)
    except Exception as err:
        msg = "col dropped"
        alldata.drop([col, sName], axis=1, inplace=True)
    print(msg, str(len(alldata.columns)) + "columns remaining")