堆栈内存溢出
  简体   繁体   English

Python Pandas DataFrame：按Timestamp列进行过滤，并带有字符串时间戳列表

[英]Python Pandas DataFrame: filter by a Timestamp column with a list of string timestamps

 原文  2019-09-06 04:30:16  6   5    python/ pandas/ dataframe

问题描述

Example setup: 设置示例： 

import pandas as pd
df = pd.DataFrame(
    data={'ts':
          [
                '2008-11-05 07:45:23.100',
                '2008-11-17 06:53:25.150',
                '2008-12-02 07:36:18.643',
                '2008-12-15 07:36:24.837',
                '2009-01-06 07:03:47.387',
          ], 
          'val': range(5)})

df.ts = pd.to_datetime(df.ts)

df.set_index('ts', drop=False, inplace=True)

df


                        | ts                      | val
2008-11-05 07:45:23.100 | 2008-11-05 07:45:23.100 | 0
2008-11-17 06:53:25.150 | 2008-11-17 06:53:25.150 | 1
2008-12-02 07:36:18.643 | 2008-12-02 07:36:18.643 | 2
2008-12-15 07:36:24.837 | 2008-12-15 07:36:24.837 | 3
2009-01-06 07:03:47.387 | 2009-01-06 07:03:47.387 | 4

Although the index is a pd.Timestamp type, I can use a string representation of a timestamp to filter it. 尽管索引是pd.Timestamp类型，但我可以使用时间戳的字符串表示形式对其进行过滤。 For example: 例如： 

df.loc['2008-11-05']

                        | ts                      | val
2008-11-05 07:45:23.100 | 2008-11-05 07:45:23.100 | 0

Moreover, pandas comes with a very convenient feature that when my filter is vague it returns the desirable result. 而且，熊猫具有非常方便的功能，当我的过滤器模糊时，它会返回理想的结果。 For example: 例如： 

df.loc['2008-12']
                        | ts                      | val
2008-12-02 07:36:18.643 | 2008-12-02 07:36:18.643 | 2
2008-12-15 07:36:24.837 | 2008-12-15 07:36:24.837 | 3

My first question is , how can I filter the df with a list of string timestamps? 我的第一个问题是 ，如何使用字符串时间戳列表过滤df？ For example if I run the code below 例如，如果我运行下面的代码 

df.loc[['2008-11-05','2008-12']]

, the result I want to get is ，我想要得到的结果是 

                        | ts                      | val
2008-11-05 07:45:23.100 | 2008-11-05 07:45:23.100 | 0
2008-12-02 07:36:18.643 | 2008-12-02 07:36:18.643 | 2
2008-12-15 07:36:24.837 | 2008-12-15 07:36:24.837 | 3

, but in fact I get the following error: ，但实际上我得到以下错误： 

KeyError: "None of [Index(['2008-11-05', '2008-12'], dtype='object', name='ts')] are in the [index]"

My second question is , can I do the similar filtering logic for a regular column? 我的第二个问题是 ，我可以对常规列执行类似的过滤逻辑吗？ Ie, if I don't set ts as the index but filter the ts column directly with a string filter. 即，如果我不将ts设置为索引，而是直接使用字符串过滤器过滤ts列。

-------------------- Follow up 2019-9-10 10:00 -------------------- --------------------跟进2019-9-10 10:00 --------------------

All the answers below are very much appreciated. 非常感谢以下所有答案。 I didn't know pd.Series.str.startswith can support the tuple input of multiple strings, or that pd.Series.str.contains can support the usage of '|' 我不知道pd.Series.str.startswith可以支持多个字符串的tuple输入，或者pd.Series.str.contains可以支持'|'的使用 . 。 New skills learned! 学习了新技能！

I think all the methods based on the use of astype(str) has one major shortcoming for me: In US people use all kinds of date time formats. 我认为所有基于astype(str)的使用方法对我来说都有一个主要缺点：在美国，人们使用各种日期时间格式。 Besides '2008-11-05', commonly used ones in my company are '2008-11-5', '11/05/2008', '11/5/2008', '20081105', '05nov2008', which would all fail if I used the string based method. 除了“ 2008-11-05”，我公司常用的还有“ 2008-11-5”，“ 11/05/2008”，“ 11/5/2008”，“ 20081105”，“ 05nov2008”如果我使用基于字符串的方法，所有操作都会失败。

For now I still have to stick with the following method, which requires the column to be the index and doesn't seem efficient (I haven't profiled), but should be sufficiently robust. 现在，我仍然必须遵循以下方法，该方法要求将列作为索引，并且似乎效率不高（我尚未进行分析），但是应该足够健壮。 I don't understand why it is not supported natively by pandas. 我不明白为什么熊猫本身不支持它。 

L = ['5nov2008','2008/12']
pd.concat([df.loc[val] for val in L]).drop_duplicates()

                        | ts                      | val
2008-11-05 07:45:23.100 | 2008-11-05 07:45:23.100 | 0
2008-12-02 07:36:18.643 | 2008-12-02 07:36:18.643 | 2
2008-12-15 07:36:24.837 | 2008-12-15 07:36:24.837 | 3

5 个解决方案

解决方案1
 1  2019-09-06 04:53:38

You can use .contains() by first converting them into str 您可以通过.contains()转换为str 

res = df.loc[(df.index.astype(str).str.contains("2008-12")) 
             | (df.index.astype(str).str.contains('2008-11-05'))]
print(res)

                                             ts  val
ts                                                  
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100    0
2008-12-02 07:36:18.643 2008-12-02 07:36:18.643    2
2008-12-15 07:36:24.837 2008-12-15 07:36:24.837    3

second question 第二个问题 

yes you can apply filter on normal column like

df.loc[(df.ts.astype(str).str.contains("2008-12"))
    |(df.ts.astype(str).str.contains('2008-11-05'))]

解决方案2
 1  2019-09-06 04:56:34

This should be get going for you.. 这应该为您服务。 

>>> df
                       ts  val
0 2008-11-05 07:45:23.100    0
1 2008-11-17 06:53:25.150    1
2 2008-12-02 07:36:18.643    2
3 2008-12-15 07:36:24.837    3
4 2009-01-06 07:03:47.387    4

Result: 结果： 

>>> df[df.apply(lambda row: row.astype(str).str.contains('2008-11-05')).any(axis=1)]
                       ts  val
0 2008-11-05 07:45:23.100    0

OR .. 要么 .. 

>>> df
                                             ts  val
ts
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100    0
2008-11-17 06:53:25.150 2008-11-17 06:53:25.150    1
2008-12-02 07:36:18.643 2008-12-02 07:36:18.643    2
2008-12-15 07:36:24.837 2008-12-15 07:36:24.837    3
2009-01-06 07:03:47.387 2009-01-06 07:03:47.387    4

Result: 结果： 

>>> df[df.apply(lambda row: row.astype(str).str.contains('2008-11-05')).any(axis=1)]
                                             ts  val
ts
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100    0

Looking for multiple values. 寻找多个值。 

>>> df[df.apply(lambda row: row.astype(str).str.contains('2008-11-05|2008-12')).any(axis=1)]
                                             ts  val
ts
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100    0
2008-12-02 07:36:18.643 2008-12-02 07:36:18.643    2
2008-12-15 07:36:24.837 2008-12-15 07:36:24.837    3

解决方案3
 1  2019-09-06 05:02:01

For your first question, you could use pd.DataFrame.append : 对于第一个问题，您可以使用pd.DataFrame.append ： 

df.loc['2008-11-05'].append(df.loc['2008-12'])

#                                              ts  val
# ts                                                  
# 2008-11-05 07:45:23.100 2008-11-05 07:45:23.100    0
# 2008-12-02 07:36:18.643 2008-12-02 07:36:18.643    2
# 2008-12-15 07:36:24.837 2008-12-15 07:36:24.837    3

For you second question, you could use pd.Series.str.match : 对于第二个问题，您可以使用pd.Series.str.match ： 

df.ts.astype(str).str.match('2008-11-05|2008-12')

# ts
# 2008-11-05 07:45:23.100     True
# 2008-11-17 06:53:25.150    False
# 2008-12-02 07:36:18.643     True
# 2008-12-15 07:36:24.837     True
# 2009-01-06 07:03:47.387    False
# Name: ts, dtype: bool

hence using this eg as a boolean index: 因此将其用作布尔索引： 

df[df.ts.astype(str).str.match('2008-11-05|2008-12')]

#                                              ts  val
# ts                                                  
# 2008-11-05 07:45:23.100 2008-11-05 07:45:23.100    0
# 2008-12-02 07:36:18.643 2008-12-02 07:36:18.643    2
# 2008-12-15 07:36:24.837 2008-12-15 07:36:24.837    3

Note that you can leave out the astype(str) part if your ts column is already of type string. 请注意，如果您的ts列已经是字符串类型，则可以astype(str)部分。

解决方案4
 1  2019-09-06 05:09:03

First idea is simply join together by concat : 第一个想法只是通过concat一起加入： 

df1 = pd.concat([df.loc['2008-11-05'], df.loc['2008-12']], sort=True)
print (df1)
                                             ts  val
ts                                                  
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100    0
2008-12-02 07:36:18.643 2008-12-02 07:36:18.643    2
2008-12-15 07:36:24.837 2008-12-15 07:36:24.837    3

Or filter by boolean indexing with mask by Series.str.contains with | 或按Series.str.contains 。的掩码按boolean indexing进行过滤，包含| for regex OR : 对于正则表达式OR ： 

df1 = df[df.index.astype(str).str.contains('2008-11-05|2008-12')]

Or with Series.str.startswith and tuple: 或使用Series.str.startswith和元组： 

df1 = df[df.index.astype(str).str.startswith(('2008-11-05', '2008-12'))]
print (df1)
                                             ts  val
ts                                                  
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100    0
2008-12-02 07:36:18.643 2008-12-02 07:36:18.643    2
2008-12-15 07:36:24.837 2008-12-15 07:36:24.837    3

If input is list of strings: 如果输入是字符串列表： 

L = ['2008-11-05','2008-12']

df2 = df[df.ts.astype(str).str.contains('|'.join(L))]

And similar: 和类似的： 

df2 = df[df.ts.astype(str).str.startswith(tuple(L))]
print (df2)
                       ts  val
0 2008-11-05 07:45:23.100    0
2 2008-12-02 07:36:18.643    2
3 2008-12-15 07:36:24.837    3

And for column only change index to ts : 对于列，仅将index更改为ts ： 

df2 = df[df.ts.astype(str).str.contains('2008-11-05|2008-12')]

Or: 要么： 

df2 = df[df.ts.astype(str).str.startswith(('2008-11-05', '2008-12'))]
print (df2)
                       ts  val
0 2008-11-05 07:45:23.100    0
2 2008-12-02 07:36:18.643    2
3 2008-12-15 07:36:24.837    3

解决方案5
 0  2019-09-06 04:53:59

You seem to have stumbled upon a bug! 您似乎偶然发现了一个错误！

This works 这有效 

df.loc['2008-11-05']

This works 这有效 

df.loc['2008-11-05':'2008-12-15']

but this doesn't, as you mentioned. 但这不是您所提到的。 

df.loc[['2008-11-05','2008-12-15']]

However, you can use as below to get the rows you want. 但是，您可以使用以下方法获取所需的行。 

df.iloc[[0,2,3]]

                                                 ts     val
ts      
2008-11-05 07:45:23.100     2008-11-05 07:45:23.100     0
2008-12-02 07:36:18.643     2008-12-02 07:36:18.643     2
2008-12-15 07:36:24.837     2008-12-15 07:36:24.837     3

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 使用 Pandas query() 过滤时间戳列上的数据框 - Using Pandas query() to filter dataframe on a timestamp column 按时间戳列过滤/选择熊猫数据帧的行 - filter/select rows of pandas dataframe by timestamp column 如果列不是所有时间戳，如何用字符串替换 dataframe 中的时间戳？ - How can I replace a timestamp in a dataframe with a string if the column is not all timestamps? 在 Python 中的 pandas 数据框的时间戳列中查找日期时间 - Find datetime in timestamp column of pandas dataframe in Python Pandas数据框，包含时间戳和时区列 - Pandas dataframe with column of timestamps and timezones 如果column是列表，是否可以过滤Pandas DataFrame列？ - Is it possible to filter Pandas DataFrame column if column is a list? 根据 Pandas Dataframe 中的时间戳列过滤给定的列（计数） - Filter a given Column(count) based on timestamp Column in Pandas Dataframe 将带有时间戳的pandas数据帧转换为String - Convert pandas dataframe with Timestamps to String 使用其他列表过滤 pandas DataFrame 列列表 - filter pandas DataFrame column list with other list 用每行中每个时间戳的列表替换pandas数据框中的时间戳列表 - Replace a list of timestamps in a pandas dataframe with a list of our for every timestamp in every row
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM