有没有办法更快地做到这一点？

Question

ladder have around 15000 elements, this code snippet performed in 5-8sec, is there any way to do it faster? I try do it without checking for duplicate and without creating accs list and time was down to 2-3sec, but I don't need duplicate in csv file.
I work in python 2.7.9

accs =[]
with codecs.open('test.csv','w', encoding="UTF-8") as out:
        row = ''
        for element in ladder:
                if element['account']['name'] not in accs:
                        accs.append(element['account']['name'])
                        row += element['account']['name']
                        if 'twitch' in element['account']:
                                row +=  "," + element['account']['twitch']['name'] + ","
                        else:
                                row += ",,"
                        row += str(element['account']['challenges']['total']) + "\n"
        out.write(row)

Answer 1

You can't do much about the loop, since you need to go through every element in ladder after all. However, you can improve this membership test:

if element['account']['name'] not in accs:

Since accs is a list, this will essentially loop through all items of accs and check if the name is in there. And you loop for every element in ladder , so this can easily become very inefficient.

Instead, use a set instead of a list for accs as this will give you a constant membership lookup. So you reduce your algorithm from a quadratic complexity to a linear complexity. For that, just use accs = set() and change your code to use accs.add() instead of append .

Another issue is that you are doing string concatenation. Every time you do someString + "something" you are throwing away that string object and create a new one. This can become inefficient for a high number of operations too. Instead, use a list here to collect all the elements you want to write, and then join them:

row = []
row.append(element['account']['name'])
if 'twitch' in element['account']:
    row.append(element['account']['twitch']['name'])
else:
    row.append('')
row.append(str(element['account']['challenges']['total']))

out.write(','.join(row))
out.write('\n')

Alternatively, since you are writing to a file anyway, you could just call out.write multiple times with each string part.

Finally, you could also look into the csv module if you are interested in writing out CSV data.

Answer 2

seen    = set()
results = []

for user in ladder:
    acc  = user['account']
    name = acc['name']
    if name not in seen:
        seen.add(name)
        twitch_name = acc['twitch']['name'] if "twitch" in acc else ''
        challenges  = acc['challenges']['total']
        results.append("%s,%s,%d" % (name, twitch_name, challenges))

with codecs.open('test.csv','w', encoding="UTF-8") as out:
    out.write("\n".join(results))