在大熊猫数据框中，算出某一列中某条件发生的次数？

Question

背景

我在CSV文件中有五年的NO2测量数据，每个位置和年份都一个文件。 我已将所有文件以相同格式加载到pandas数据框中：

Date    Hour    Location    NO2_Level
0   01/01/2016  00  Street  18
1   01/01/2016  01  Street  39
2   01/01/2016  02  Street  129
3   01/01/2016  03  Street  76
4   01/01/2016  04  Street  40

目标

对于每个数据帧计数，NO2_Level大于150的次数并输出。

因此，我编写了一个循环，该循环从正确的目录创建所有数据帧，并适当地清理它们。

问题

无论我已经试过产生的结果我知道检查是不正确的，例如：对于给定年份的每个位置-the计数值是相同的（可能的，但不太可能） -对于一年当我知道应该有任何的正数计数，每个位置返回0

我尝试过的

我尝试了很多方法来为每个数据框获取此值，例如将列设为一系列：

NO2_Level = pd.Series(df['NO2_Level'])
count = (NO2_Level > 150).sum()'''

使用pd.count（）：

count = df[df['NO2_Level'] >= 150].count()

这两种方法最接近我要输出的内容

测试示例

data = {'Date': ['01/01/2016','01/02/2016',' 01/03/2016', '01/04/2016', '01/05/2016'], 'Hour': ['00', '01', '02', '03', '04'], 'Location':  ['Street','Street','Street','Street','Street',], 'NO2_Level': [18, 39, 129, 76, 40]}
df = pd.DataFrame(data=d)
NO2_Level = pd.Series(df['NO2_Level'])
count = (NO2_Level > 150).sum()
count

预期产出

因此，我试图通过它为位置，年份，计数（条件）格式的每个数据框输出一行：

Kirkstall Road,2013,47
Haslewood Close,2013,97
...
Jack Lane Hunslet,2015,158

所以上面的例子会产生

Street, 2016, 1

实际每年在每个位置产生的结果都是相同的，在某些年份（2014年）中，在检查时似乎根本不起作用，应该有：

Kirkstall Road,2013,47
Haslewood Close,2013,47
Tilbury Terrace,2013,47
Corn Exchange,2013,47
Temple Newsam,2014,0
Queen Street Morley,2014,0
Corn Exchange,2014,0
Tilbury Terrace,2014,0
Haslewood Close,2015,43
Tilbury Terrace,2015,43
Corn Exchange,2015,43
Jack Lane Hunslet,2015,43
Norman Rows,2015,43

Answer 1

这是一个带有随机生成的样本的解决方案：

def random_dates(start, end, n):
    start_u = start.value // 10 ** 9
    end_u = end.value // 10 ** 9
    return pd.to_datetime(np.random.randint(start_u, end_u, n), unit='s')

location = ['street', 'avenue', 'road', 'town', 'campaign']

df = pd.DataFrame({'Date' : random_dates(pd.to_datetime('2015-01-01'), pd.to_datetime('2018-12-31'), 20),
                   'Location' : np.random.choice(location, 20),
                   'NOE_level' : np.random.randint(low=130, high= 200, size=20)})

---

#Keep only year for Date
df['Date'] = df['Date'].dt.strftime("%Y")

print(df)

df = df.groupby(['Location', 'Date'])['NOE_level'].apply(lambda x: (x>150).sum()).reset_index(name='count')
print(df)

生成的示例df：

        Date  Location  NOE_level
0       2018      town        191
1       2017  campaign        187
2       2017      town        137
3       2016    avenue        148
4       2017  campaign        195
5       2018      town        181
6       2018      road        187
7       2018      town        184
8       2016      town        155
9       2016    street        183
10      2018      road        136
11      2017      road        171
12      2018    street        165
13      2015    avenue        193
14      2016  campaign        170
15      2016    street        132
16      2016  campaign        165
17      2015      road        161
18      2018      road        161
19      2015      road        140 

输出：

    Location       Date  count
0     avenue       2015      1
1     avenue       2016      0
2   campaign       2016      2
3   campaign       2017      2
4       road       2015      1
5       road       2017      1
6       road       2018      2
7     street       2016      1
8     street       2018      1
9       town       2016      1
10      town       2017      0
11      town       2018      3

Answer 2

希望这会有所帮助。

import pandas as pd

ddict = {
    'Date':['2016-01-01','2016-01-01','2016-01-01','2016-01-01','2016-01-01','2016-01-02',],
    'Hour':['00','01','02','03','04','02'],
    'Location':['Street','Street','Street','Street','Street','Street',],
    'N02_Level':[19,39,129,76,40, 151],
}

df = pd.DataFrame(ddict)

# Convert dates to datetime
df['Date'] = pd.to_datetime(df['Date'])

# Make a Year column
df['Year'] = df['Date'].apply(lambda x: x.strftime('%Y'))

# Group by lcoation and year, count by M02_Level > 150
df1 = df[df['N02_Level'] > 150].groupby(['Location','Year']).size().reset_index(name='Count')

# Interate the results
for i in range(len(df1)):
    loc = df1['Location'][i]
    yr = df1['Year'][i]
    cnt = df1['Count'][i]
    print(f'{loc},{yr},{cnt}')


### To not use f-strings
for i in range(len(df1)):
    print('{loc},{yr},{cnt}'.format(loc=df1['Location'][i], yr=df1['Year'][i], cnt=df1['Count'][i]))

样本数据：

        Date Hour Location  N02_Level
0 2016-01-01   00   Street         19
1 2016-01-01   01   Street         39
2 2016-01-01   02   Street        129
3 2016-01-01   03   Street         76
4 2016-01-01   04   Street         40
5 2016-01-02   02   Street        151

输出：

Street,2016,1