Pandas 如果列包含字符串，则从另一列获取唯一值并从 dataframe 中删除行

Question

I have a small problem. I have this dataframe with 7 columns. Two of them are 'IP' and 'URL'.

It is a web log data set, so I am trying to get unique IP of rows, where URL contains string "robots.txt" and then if this condition is applied drop the rows of the uniqueIP's from dataframe.

I had a hard time trying to solve this. I tried pandas groupby but cant solve it still. I am able to get unique ip's where url contains string "robots.txt" in this code:

robots = data2[data2.url.str.contains('robots.txt', regex=True)] 
len(robots[['ip']].drop_duplicates())

But after that I don't know how to drop these rows from dataframe. Does someone have some tips? Thanks.

---

Here is the sample: https://i.stack.imgur.com/t6q39.png

---

Dataframe has around 30k rows. So desired output is to drop all rows from dataframe if string "robots.txt" is in url column. I can do that but trick is to remember values from column 'ip' when column 'url' contains that particular string and drop rows that are accessed through that particular ip address

Answer 1

Just negate your condition

robots_condition = data2.url.str.contains('robots.txt')
no_crawl_ips = data2.loc[robots_condition, 'ip'].unique()
data2 = data2[~robots_condition]