Python pandas 根据条件将数据附加到下一列

Question

​

I'm working on a dataframe with around 200k records that looks like this (information replaced with random text):

ID                Description         
1                 Eg.1
2                 Desc.2
3                 Desc.3
80                 
aaa
output
500                
c                   
d
e
f
input
100              Desc.100
200              Desc.200

I have set it up in a pandas dataframe and was thinking I could do something like:

for x in df['ID'] :
    if type(df['ID'][x]) == str:
        df['Description'][x-1] += ' ' + df['ID'][x].values       

To try and append the faulty text in ID (below is the desired outcome that I want to get)

ID                Description         
1                 Eg.1
2                 Desc.2
3                 Desc.3
80                aaa output
500               c d e f input         
100               Desc.100

Where only numerics are kept in ID column and all descriptions are appended to the previous correct ID. (another issue is that the number of faulty text under ids range from 1 to 10 in some cases)

I'm a bit stuck since x in the above code returns the string that was found in the df['ID'] section, any thoughts on how this could be accomplished in a relatively fast way across the 200k+ records?

Thanks!

Answer 1

Here's an idea on how to do it in pandas:

I read your example from the clipboard

import pandas as pd
import numpy as np
df = pd.read_clipboard()

First I copied the string indexes into the description where the ID was a string. Because it should go in the description field. I'm using the str(x).isnumeric() to treat each cell as string, even if it isn't. If some of the cells are imported as numbers and some are as strings, the .isnumeric part will cause an error on number typed fields.

df.loc[df['ID'].apply(lambda x: not str(x).isnumeric()), 'Description'] = df['ID']

Then I emptied the ID from those entries rows only

df.loc[df['ID'].apply(lambda x: not str(x).isnumeric()), 'ID'] = np.NaN

I filled the now empty ID with the previous line ID

df['ID'] = df['ID'].fillna(method='ffill')

As the first line of each of these groups is still empty, I drop it and group the rest

df_result = df.dropna().groupby('ID', sort=False).aggregate(lambda x: ' '.join(x))

print (df_result)

Something to consider: if the broken data is not in dataframe, but in a file, I'd probably write code that goes through the file line by line and writes the fixed lines into a correction file. This would not require the 200k lines to be in memory at the same time, which would make the process easier, because you only have to run the fix once.

Answer 2

You can try of keeping only numeric value in 'ID' by assigning non numeric id info to description. After forward fill the ID apply groupby and join the description.

df['Description'] = df.apply(lambda x : x['Description'] if x['ID'].isdigit() else x["ID"],1).fillna('')
df['ID'] = df.ID.apply(lambda x:x if x.isdigit() else np.nan).fillna(method='ffill')
df = pd.DataFrame(df.groupby(['ID'],sort=False)['Description'].apply(lambda x: ' '.join(x))).reset_index()

Out:

   ID   Description
0   1   Eg.1
1   2   Desc.2
2   3   Desc.3
3   80  aaa output
4   500 c d e f input
5   100 Desc.100
6   200 Desc.200

Answer 3

This uses numpy almost exclusively. It is faster than the pandas groupby methods even though the code is longer. Repeating numerical values in the ID column are OK (all numerical rows will be returned whether or not they are duplicated as the code stands now).

import numpy as np
import pandas as pd

df = pd.DataFrame({'ID': ['1', '2', '3', '80', 'aaa',
                           'output', '500', 'c', 'd',
                           'e', 'f', 'input', '100', '200'],
                   'Description': ['Eg.1', 'Desc.2', 'Desc.3',
                                   '', '', '', '', '', '', '',
                                   '', '', 'Desc.100', 'Desc.200']})

IDs = df.ID.values

# numeric test function for ID column
def isnumeric(s):
    try:
        float(s)
        return 1
    except ValueError:
        return 0

# find the rows which are numeric and mark with 1 (vs 0)
nums = np.frompyfunc(isnumeric, 1, 1)(IDs).astype(int)

# make another array, which marks
# str IDs with a 1 (opposite of nums)
strs = 1 - nums

# make arrays to hold shifted arrays of strs and nums
nums_copy = np.empty_like(nums)
strs_copy = np.empty_like(strs)

# make an array of nums shifted fwd 1
nums_copy[0] = 1
nums_copy[1:] = nums[:-1]

# make an array of strs shifted back 1
strs_copy[-1] = 0
strs_copy[:-1] = strs[1:]

# make arrays to detect where str and num
# ID segments begin and end
str_idx = strs + nums_copy
num_idx = nums + strs_copy

# find indexes of start and end of ID str segments
starts = np.where(str_idx == 2)[0]
ends = np.where(str_idx == 0)[0]

# make a continuous array of IDs which
# were marked as strings
txt = IDs[np.where(strs)[0]]
# split that array into string segments which will
# become a combined string row value
txt_arrs = np.split(txt, np.cumsum(ends - starts)[:-1])
# join the string segment arrays
txt_arrs = [' '.join(x) for x in txt_arrs]

# find the row indexes which will contain combined strings
combo_str_locs = np.where(num_idx == 2)[0][:len(txt_arrs)]
# put the combined strings into the Description column
# at the proper indexes
np.put(df.Description, combo_str_locs, txt_arrs)
# slice the original dataframe to retain only numeric
# ID rows
df = df.iloc[np.where(nums == 1)[0]]

# If a new index is desired >> df.reset_index(inplace=True, drop=True) 

Answer 4

Other approach could be as show below:

Input data:

df = pd.DataFrame({'ID': ['1', '2', '3', '80', 'aaa', 'output', '500', 'c', 'd', 'e', 'f', 'input', '100', '200'],
                   'Description': ['Eg.1', 'Desc.2', 'Desc.3', np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, 'Desc.100', 'Desc.200']})

Logic to process dataframe to get the desired result:

df['IsDigit'] = df['ID'].str.isdigit()
df['Group'] = df['IsDigit'].ne(df['IsDigit'].shift()).cumsum()
dfG = df[df['IsDigit'] == False].groupby(['Group'])['ID'].apply(lambda x: ' '.join(x))
df = df.drop(df[df['IsDigit'] == False].index)
df.loc[df['Description'].isna(), 'Description'] = df[df['Description'].isna()].apply(lambda x: dfG[x['Group'] + 1], axis=1)
df = df.drop(columns=['IsDigit', 'Group']).set_index('ID')

And it produces below output:

       Description
ID                
1             Eg.1
2           Desc.2
3           Desc.3
80      aaa output
500  c d e f input
100       Desc.100
200       Desc.200