如何解决这个特定问题，使循环更快/更好？

Question

Python beginner here.

Here is my problem: I have aa csv file with roughly 3200 rows and 660 columns. The rows are filled with either 0s, 1s, or 50s.

I need to update the newly created column 'answer' by these requirements:

1. It should be the sum of 1s in that row that happen before a '50' occurs.
2. If there is no '50' in that row, just update the last column to a zero.

so, for example, the row [1, 0, 0, 0, 1, 1, 50, 0, 0, 0, 1] should have a new value at the end of it as '3' because we found three 1s before finding a 50.

Here's my code:

df_numRows = len(df.values)
df_numCols = len(df.columns)

for row in range(df_numRows):
    df_sum = 0
    for col in range(df_numCols):
        if '50' not in df.values[row]:
            df.at[row, 'answer'] = '0'
        elif df.values[row][col] == '0':
            continue
        elif df.values[row][col] == '1':
            df_sum += 1
            df.at[row, 'answer'] = df_sum
        elif df.values[row][col] == '50':
            break

I wrote this nested for loop to iterate through my Pandas dataframe but it seems to take a VERY long time to run.

I ran this piece of code on the same dataset but with only 100 rows x 660 columns and it took about 1.5 mins, however, when I try to run it on the entire thing, it ran for about 2.5 hours and I just shut it down because I thought it had taken too long.

How can I make my code more efficient/faster/better? I would love any help at all from you guys, and I apologize in advance if this is an easy question but I am just getting started in Python!

Thanks guys!

Answer 1

Just do cumprod after we find the 50, if it is 50 we all values below will become 0 , then we using this Boolean dataframe filter the original df , and do sum

df=pd.DataFrame({'A':[1, 0, 0, 0, 1, 1, 50, 0, 0, 0, 1] })
df.mul(df.ne(50).cumprod()).sum()
Out[35]: 
A    3
dtype: int64

Answer 2

Setup

df = pd.DataFrame([
    [1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1],    # No 50s
    [1, 0, 0, 0, 1, 1, 50, 0, 0, 0, 1],   # One 50
    [1, 50, 0, 0, 1, 50, 50, 0, 0, 0, 1], # Three 50s but 2 are consecutive
    [1, 50, 0, 0, 1, 1, 50, 0, 0, 0, 1],  # Two 50s
])

df

   0   1   2   3   4   5   6   7   8   9   10
0   1   0   0   0   1   1   0   0   0   0   1
1   1   0   0   0   1   1  50   0   0   0   1
2   1  50   0   0   1  50  50   0   0   0   1
3   1  50   0   0   1   1  50   0   0   0   1

---

Use logical_and with its accumulate method

np.logical_and will take the and operator and apply it to a group of booleans. The accumulate part says to keep applying it and as we go keep track of the most recent and of all prior booleans. By specifying axis=1 I'm saying to do this for each row. This returns an array of booleans where the rows are true until we hit the value of 50 . I then check to see of any are fifty withe all(1) . The proper multiplication gives the sums of all values not 50 prior to the first 50... for each row.

d = np.logical_and.accumulate(df.ne(50), axis=1)

df.mul(d).mul(~d.all(1), 0).sum(1)

0    0
1    3
2    1
3    1
dtype: int64

Combine to get new column

d = np.logical_and.accumulate(df.ne(50), axis=1)

df.assign(answer=df.mul(d).mul(~d.all(1), 0).sum(1))

   0   1  2  3  4   5   6  7  8  9  10  asnswer
0  1   0  0  0  1   1   0  0  0  0   1        0
1  1   0  0  0  1   1  50  0  0  0   1        3
2  1  50  0  0  1  50  50  0  0  0   1        1
3  1  50  0  0  1   1  50  0  0  0   1        1

---

If you want to go full blown Numpy

v = df.values
a = np.logical_and.accumulate(v != 50, axis=1)
df.assign(answer=(v * (a & ~a.all(1, keepdims=True))).sum(1))

   0   1  2  3  4   5   6  7  8  9  10  asnswer
0  1   0  0  0  1   1   0  0  0  0   1        0
1  1   0  0  0  1   1  50  0  0  0   1        3
2  1  50  0  0  1  50  50  0  0  0   1        1
3  1  50  0  0  1   1  50  0  0  0   1        1

Answer 3

Please try this logic and let me know if this helps.

df_numRows = len(df.values)
df_numCols = len(df.columns)

for row in range(df_numRows):
    df_sum = 0

    try:
        indexOf50 = np.argwhere(df.loc[row]==50)[0][0]
        colArrayTill50 = df.loc[row][:indexOf50].values
        numberOfOne = colArrayTill50.sum()
    except:
        numberOfOne = 0

    print(numberOfOne)

Answer 4

This solves it, though bit robust:

import pandas as pd
import numpy as np

np.random.seed(1)

df = pd.DataFrame(np.random.choice([0, 1, 50], (3200,660)))

data = df.values
idxs = [np.where(d == 50) for d in data]
sums = [sum(d[:i[0][0]]) if i[0].size else 0 for d, i in zip(data, idxs)]

data = np.column_stack((data, sums))

df = df.assign(answer=sums)

df.head()

#    0   1   2   3   4   5  6   7   8   9   ...    651  652  653  654  655  \
#0   1   0   0   1   1   0  0   1   0   1   ...     50   50    1    1    0   
#1   1   0  50   1  50  50  0   1   1  50   ...      1    0    1    0    0   
#2  50   0   1   0   1  50  1  50   0  50   ...      0   50    1   50   50   
#3   0   1   0  50   1   0  0  50   1   0   ...      1    1    0    1    1   
#4   1  50   1   1   1   1  0  50  50   1   ...      0    1    0    1    0   
#
#   656  657  658  659  answer  
#0    0    0    1    0       5  
#1    1   50    0   50       1  
#2   50    1    1   50       0  
#3    0   50    1   50       1  
#4    0   50    0   50       1