Removing DataFrame rows by comparing dates

Question

I have a dataset and only wants to have the rows inside a time range. I put all the good rows in a Series object. But when I re-assign that object to the DataFrame object, I get NaT values:

code:

def get_tweets_from_range_in_csv():
    csvfile1 = "results_dataGOOGL050"
    df1 = temp(csvfile1)


def temp(csvfile):
    tweetdats = []
    d = pd.read_csv(csvfile + ".csv", encoding='latin-1')
    start = datetime.datetime.strptime("01-01-2018", "%d-%m-%Y")
    end = datetime.datetime.strptime("01-06-2018", "%d-%m-%Y")
    for index, current_tweet in d['Date'].iteritems():
        date_tw = datetime.datetime.strptime(current_tweet[:10], "%Y-%m-%d")
        if start <= date_tw <= end:
            tweetdats.append(date_tw)
        else:
            d.drop(index, inplace=True)
    d = d.drop("Likes", 1)
    d = d.drop("RTs", 1)
    d = d.drop("Sentiment", 1)
    d = d.drop("User", 1)
    d = d.drop("Followers", 1)
    df1['Date'] = pd.Series(tweetdats)
    return d

Output of tweetdats:

tweetdats
Out[340]: 
[datetime.datetime(2018, 1, 30, 0, 0),
 datetime.datetime(2018, 4, 1, 0, 0),
 datetime.datetime(2018, 4, 1, 0, 0),
 datetime.datetime(2018, 4, 1, 0, 0),
 datetime.datetime(2018, 1, 5, 0, 0),
 datetime.datetime(2018, 1, 5, 0, 0),
 datetime.datetime(2018, 1, 8, 0, 0),
 datetime.datetime(2018, 1, 20, 0, 0),
 datetime.datetime(2018, 1, 22, 0, 0),
 datetime.datetime(2018, 1, 5, 0, 0)]

Answer 1

You do not need to iterate through your dataframe with a for loop to select the rows inside the time range of interest.

Let us assume that your initial dataframe df has a 'Date' column containing the dates in datetime format; you can then simply create a new dataframe new_df :

new_df=df[(pd.to_datetime(df.time) > start) & (pd.to_datetime(self.df.time) < end)] 

This way you do not have to copy and paste the "good" rows in a Series and then reassign them to a dataframe.

Your temp function would look like:

def temp(csvfile):
    df = pd.read_csv(csvfile + ".csv", encoding='latin-1')
    start = datetime.datetime.strptime("01-01-2018", "%d-%m-%Y")
    end = datetime.datetime.strptime("01-06-2018", "%d-%m-%Y")
    new_df=df[(pd.to_datetime(df.time) > start) & (pd.to_datetime(self.df.time) < end)]

Hope this helps!