使用Python从列中获取唯一值

Question

I'm trying to get unique values from the column 'name' for every distinct value in column 'gender'.

Here's sample data: sample input_file_data:

index,name,gender,alive
1,Adam,Male,Y
2,Bella,Female,N
3,Marc,Male,Y
1,Adam,Male,N

I could get it when I give a value corresponding to 'gender' like for example, gave "Male" in the code below:

filtered_data = filter(lambda person: person["gender"] == "Male", input_file_data)
reader = (dict((k, v.strip()) for k, v in row.items() if v) for row in filtered_data)
countt = [rec[gender] for rec in reader]
final1 = input_file_name + ".txt", "gender", "Male"
output1 = str(final1).replace("(", "").replace(")", "").replace("'","").replace(", [{", " -- [").replace("}", "")
final2 = set(re.findall(r"name': '(.*?)'", str(filtered_data)))
final_count = len(final2)
output = str(final_count) + " occurrences", str(final2)
output2 = output1, str(output)
output_final = str(output2).replace('\\', "").replace('"',"").replace(']"', "]").replace("set", "").replace("(", "").replace(")", "").replace("'","").replace(", [{", " -- [").replace("}", "")
output_final = output_final + "\n"

current output:

input_file_name.txt, gender, Male, 2 occurrences, [Adam,Marc]

Expected output:

input_file_name.txt, gender, Male, 2 occurrences, [Adam,Marc], Female, 1 occurrences [Bella]

which should show up all the unique occurrences of names, for every distinct gender value (without hardcoding). Also I do not want to use Pandas. Any help is highly appreciated.

PS- I have multiple files and not all files have the same columns. So I can't hardcode them. Also, all the files have a 'name' column, but not all files have a 'gender' column. And this script should work for any other column like 'index' or 'alive' or anything else for that matter and not just gender.

Answer 1

I would use the csv module along with the defaultdict from collections for this. Say this is stored in a file called test.csv:

>>> import csv
>>> from collections import defaultdict
>>> with open('test.csv', 'rb') as fin: data = list(csv.reader(fin))[1:]
>>> gender_dict = defaultdict(set)
>>> for idx, name, gender, alive in data:
    gender_dict[gender].add(name)

>>> gender_dict
defaultdict(<type 'set'>, {'Male': ['Adam', 'Marc'], 'Female': ['Bella']})

You now have a dictionary. Each key is a unique value from the gender column. Each value is a set, so you'll only get unique items. Notice that we added 'Adam' twice, but only see one in the resulting set.

You don't need defaultdict , but it allows you to use less boilerplate code to check if a key exists.

EDIT: It might help to have better visibility into the data itself. Given your code, I can make the following assumptions:

• input_file_data is an iterable (list, tuple, something like that) containing dictionaries.

• Each dictionary contains a 'gender' key. If it didn't include at least 'gender' , you would get a key error when trying to filter it.

• Each dictionary has a 'name' key, it looks like.

Rather than doing all of that regex, what about this?

>>> gender_dict = {'Male': set(), 'Female': set()}
>>> for item in input_file_data:
        gender_dict[item['gender']].add(item['name'])

You can use item.get('name') instead of item['name'] if not every entry will have a name.

Edit #2: Ok, the first thing you need to do is get your data into a consistent state. We can absolutely get to a point where you have a column name (gender, index, alive, whatever you want) and a set of unique names corresponding to those columns. Something like this:

data_dict = {'gender':
                 {'Male': ['Adam', 'Marc'],
                  'Female': ['Bella']}
             'alive':
                 {'Y': ['Adam', 'Marc'],
                  'N': ['Bella', 'Adam']}
             'index':
                 {1: ['Adam'],
                  2: ['Bella'],
                  3: ['Marc']}
              }

If that's what you want, you could try this:

>>> data_dict = defaultdict(lambda: defaultdict(lambda: defaultdict(set)))
>>> for element in input_file_data:
        for key, value in element.items():
            if key != 'name':
                data_dict[key][value].add(element[name])

That should get you what you want, I think? I can't test as I don't have your data, but give it a try.