Calculating Precision, Recall and F-score in one pass - python

Question

Accuracy, precision, recall and f-score are measures of a system quality in machine-learning systems. It depends on a confusion matrix of True/False Positives/Negatives.

Given a binary classification task, I have tried the following to get a function that returns accuracy, precision, recall and f-score:

gold = [1] + [0] * 9
predicted = [1] * 10

def evaluation(gold, predicted):
  true_pos = sum(1 for p,g in zip(predicted, gold) if p==1 and g==1)
  true_neg = sum(1 for p,g in zip(predicted, gold) if p==0 and g==0)
  false_pos = sum(1 for p,g in zip(predicted, gold) if p==1 and g==0)
  false_neg = sum(1 for p,g in zip(predicted, gold) if p==0 and g==1)
  try:
    recall = true_pos / float(true_pos + false_neg)
  except:
    recall = 0
  try:
    precision = true_pos / float(true_pos + false_pos)
  except:
    precision = 0
  try:
    fscore = 2*precision*recall / (precision + recall)
  except:
    fscore = 0
  try:
    accuracy = (true_pos + true_neg) / float(len(gold))
  except:
    accuracy = 0
  return accuracy, precision, recall, fscore

But it seems like I have redundantly looped through the dataset 4 times to get the True/False Positives/Negatives.

Also the multiple try-excepts to catch the ZeroDivisionError is a little redundant.

So what is the pythonic way to get the counts of the True/False Positives/Negatives without multiple loops through the dataset?

How do I pythonically catch the ZeroDivisionError without the multiple try-excepts?

---

I could also do the following to count the True/False Positives/Negatives in one loop but is there an alternative way without the multiple if ? :

for p,g in zip(predicted, gold):
    if p==1 and g==1:
        true_pos+=1
    if p==0 and g==0:
        true_neg+=1
    if p==1 and g==0:
        false_pos+=1
    if p==0 and g==1:
        false_neg+=1

Answer 1

what is the pythonic way to get the counts of the True/False Positives/Negatives without multiple loops through the dataset?

I would use a collections.Counter , roughly what you're doing with all of the if s (you should be using elif s, as your conditions are mutually exclusive) at the end:

counts = Counter(zip(predicted, gold))

Then eg true_pos = counts[1, 1] .

How do I pythonically catch the ZeroDivisionError without the multiple try-excepts?

For a start, you should (almost) never use a bare except: . If you're catching ZeroDivisionError s, then write except ZeroDivisionError . You could also consider a "look before you leap" approach, checking whether the denominator is 0 before trying the division, eg

accuracy = (true_pos + true_neg) / float(len(gold)) if gold else 0

Answer 2

This is a pretty natural use case for the bitarray package.

import bitarray as bt

tp = (bt.bitarray(p) & bt.bitarray(g)).count()
tn = (~bt.bitarray(p) & ~bt.bitarray(g)).count()
fp = (bt.bitarray(p) & ~bt.bitarray(g)).count()
fn = (~bt.bitarray(p) & bt.bitarray(g)).count()

There's some type conversion overhead, but after that, the bitwise operations are much faster.

For 100 instances, timeit on my PC gives 0.036 for your method and 0.017 using bitarray at 1000 passes. For 1000 instances, it goes to 0.291 and 0.093. For 10000, 3.177 and 0.863. You get the idea.

It scales pretty well, using no loops, and doesn't have to store a large intermediate representation building a temporary list of tuples in zip .

Answer 3

Depending on your needs, there are several libraries that will calculate precision, recall, F-score, etc. One that I have used is scikit-learn . Assuming that you have aligned list s of actual and predicted values, then it is as simple as...

from sklearn.metrics import precision_recall_fscore_support as pr
bPrecis, bRecall, bFscore, bSupport = pr(gold, predicted, average='binary')

One of the advantages of using this library is that different flavors of metrics (such as micro-averaging, macro-averaging, weighted, binary, etc.) come free out of the box.