处理多个文件时保存列表 python

Question

I have one python script to process several files in a folder, and the results are like this, there're two columns called user_agent and user_type

firefox, pc
IE, pc
iPhone, mobile
....

since the results from the same or different files may be duplicated, I use a list to keep track of the unique combination.

if (user_agent,user_type) not in lookuplist:
   lookuplist.append((user_agent,user_type))
   print user_agent,user_type

Now the problem is, since I have more than one raw data file to parse, how to "save" the lookuplist when one file is done, and when the second one starts, it still knows, for example, (firefox, pc) has already exist, then I won't have duplicated results

Many thanks

Answer 1

First, you should use a set and not a list for your lookuplist . Second, open all the files inside a loop and in the loop check for duplicates.