需要帮助对数据进行排序

Question

I'm trying to clean some data for a computational biology research project. However, an issue is coming up where some dogs born from the same litter on the same day have the same mother but multiple fathers. I need to find these data points and return them in some for so that I can manually go back to the documents and check them. Does anyone know a better way so that each set doesn't take 30+ minutes to finish?

I have been trying to use pandas to go through the data so far, and I'm no CS wizard. I basically used a for loop to check the data each individually and even the smaller sets have around 10k pieces of data.

data = raw_data.loc[:,['Order', 'Name', 'Sire', 'Dam', 'Registration', 'DOB']]
length = len(data.index)

for i in range(0,length,1):
    for j in range(i+1,length,1):
        if (data.iat[i,5]==data.iat[j,5]): #Same date of birth
            if (data.iat[i,3]==data.iat[j,3]): #Same mother
                if (data.iat[i,2]!= data.iat[j,2]): #Different father
                    print(data.iat[i,0]+data.iat[j,0])

Answer 1

You can group your data by date of birth and mother, then calculate the number of different values for the father columns. The result will be calculated for every group of DOB and Dam. You will be interested in all the groups with a result greater than 1.

import pandas as pd
data.groupby(by=['DOB','Dam']).\ # Group your data by 'DOB' and 'Dam'
aggregate({'Sire':pd.Series.nunique}).\ # Count distinct values for 'Sire' in each group
sort_values(by="Sire", ascending= False).\ # Descending order of the results
query("Sire > 1").\ # Take the 'DOB' and 'Dam' pairs with more than 1 'Sire'
to_excel("File_with_results.xlsx") # Write the results to an excel file

Answer 2

Welcome to Stackoverflow.

One additional suggestion beyond Miguel's.

For testing I would trim down your file to a small sample that includes the issue you are working on. You don't want to waste CPU time until you know the program is behaving.

BDS