how to eliminate rows with continuous values for a column using python/pandas

Question

I have a data frame like this with continuous zeros in column 1:

col1    col2    col3
  1       2       3
  0       4       5
  0       1       4
  2       7       8
  0       1       2
  4       4       4
  0       1       3
  0       4       2
  0       1       9
  4       6       2

I want to skip those rows where is continuous zeros for atleast 2 times.

for Example the output will look like:

col1    col2    col3
  1       2       3
  2       7       8
  0       1       2
  4       4       4
  4       6       2

Answer 1

Use:

m = df['col1'].ne(0)
s = m.cumsum() * (~m)
df = df[s.groupby(s).transform('size').lt(2) | m]

Or:

df = df[s.map(s.value_counts()).lt(2) | m]

---

print (df)
   col1  col2  col3
0     1     2     3
3     2     7     8
4     0     1     2
5     4     4     4
9     4     6     2

Explanation :

First compare for not equal 0 by Series.ne :

print (df['col1'].ne(0))
0     True
1    False
2    False
3     True
4    False
5     True
6    False
7    False
8    False
9     True
Name: col1, dtype: bool

Then use cumsum for groups - values with 0 have same groups:

print (m.cumsum())
0    1
1    1
2    1
3    2
4    2
5    3
6    3
7    3
8    3
9    4
Name: col1, dtype: int32

Multiple by inversed boolean mask for remove non 0 values:

print (m.cumsum() * (~m))
0    0
1    1
2    1
3    0
4    2
5    0
6    3
7    3
8    3
9    0
Name: col1, dtype: int32

Then get count of groups by GroupBy.transform :

print (s.groupby(s).transform('size'))
0    4
1    2
2    2
3    4
4    1
5    4
6    3
7    3
8    3
9    4
Name: col1, dtype: int64

And compare by lt < :

print (s.groupby(s).transform('size').lt(2))
0    False
1    False
2    False
3    False
4     True
5    False
6    False
7    False
8    False
9    False
Name: col1, dtype: bool

Last chain by original mask m by | for bitwise OR :

print (s.groupby(s).transform('size').lt(2) | m)
0     True
1    False
2    False
3     True
4     True
5     True
6    False
7    False
8    False
9     True
Name: col1, dtype: bool

And last filter by boolean indexing :

print (df[s.groupby(s).transform('size').lt(2) | m])

   col1  col2  col3
0     1     2     3
3     2     7     8
4     0     1     2
5     4     4     4
9     4     6     2