在 Pandas 中基于列重复数据删除创建 unique_id

Question

I'm trying to generate an unique_id based using as a base some columns:

The current process has the following process:

• Indicates columns that will be used as unique;
• Create a bool column called is_duplicated ;
• Iterate over nonduplicates, the unique rows, and put an integer generated by enumerate in all equal rows.

optimal = ["date", "amount", "description", "tenant_id", "comment", "bank_account_id"]
data_normalization["is_duplicated"] = data_normalization.duplicated(subset=optimal)

for unique_id, row in enumerate(data_normalization.loc[data_normalization.is_duplicated == False].itertuples()):
    data_normalization.loc[
        (data_normalization.date == row.date,) &
        (data_normalization.amount == row.amount,) &
        (data_normalization.description == row.description,) &
        (data_normalization.tenant_id == row.tenant_id,) &
        (data_normalization.comment == row.comment,) &
        (data_normalization.bank_account_id == row.bank_account_id,),
        "unique_id"
    ] = unique_id
    

The way above works, but I'm wondering if there is no better way to do it, than using pandas features.

Example:

• Suppose that we have a table like below

| Row1     | row2           | Row3     | unique_id |
| -------- | -------------- | -------- | --------  |
| First    | row            | First    |    1      |
| First    | row            | First    |    1      |
| Second   | 22             |scondd    |    2      |
| Second   | 22             |scondd    |    2      |
| Second   | 22             |scondd    |    2      |
| Third    | 22             |scondd    |    3      |

• Basically, the unique_id is created through ["Row1", "row2"]
• Every time that the Row1 and row2 are equal the index remaining the same;
• When not the index is increased

--

• The idea is to create a unique integer id over the target columns.;
• The snippet code above works, but I want some more clear and performative, that uses pandas power

Answer 1

You can use duplicated and cumsum to get that done.

Starting with your sample data frame

     Row1 row2    Row3
0   First  row   First
1   First  row   First
2  Second   22  scondd
3  Second   22  scondd
4  Second   22  scondd
5   Third   22  scondd

Execute

df['unique_id']  = (~df.duplicated(['Row1','row2'])).cumsum()
print(df)

Result

     Row1 row2    Row3  unique_id
0   First  row   First          1
1   First  row   First          1
2  Second   22  scondd          2
3  Second   22  scondd          2
4  Second   22  scondd          2
5   Third   22  scondd          3