Building a TF-IDF Vectorizer from Scratch

Question

I am trying yo build a tf-idf vectorizer from scratch. I calculated the tf and the idf, but i am having trouble calculating the tf-idf. Here's the code:

from tqdm import tqdm
from scipy.sparse import csr_matrix
import math
import operator
from sklearn.preprocessing import normalize
import numpy

corpus = [
     'this is the first document',
     'this document is the second document',
     'and this is the third one',
     'is this the first document',
]

#splitting words of each document in the corpus
document = []
for doc in corpus:
    document.append(doc.split())

#calculating the word frequency of each ord inside a document
word_freq = {}   #calculate frequency of each word
for i in range(len(document)):
    tokens = document[i]
    for w in tokens:
        try:
            word_freq[w].add(i)  #add the word as key 
        except:
            word_freq[w] = {i}  #if it exists already, do not add.

for val in word_freq:
    word_freq[val] = len(word_freq[val])  #Counting the number of times a word(key)is in the whole corpus thus giving us the frequency of that word.

# Calculating term frequency
def tf(document):
    tf_dict = {}
    for word in document:
        if word in tf_dict:
            tf_dict[word] += 1
        else:
            tf_dict[word] = 1

    for word in  tf_dict:
        tf_dict[word] = tf_dict[word]/len(document)

tfDict = [tf(i) for i in document]

# Calculate inverse document frequency
def IDF():
    idfDict = {}
    for word in word_freq:
        idfDict[word] = 1 + math.log((1 + len(sentence)) / (1 + word_freq[word]))

    return idfDict

idfDict = IDF()

# Calculating TF-IDF
def TF_IDF():
    tfIdfDict = {}
    for i in tfDict:
        for j in i:
            tfIdfDict[j] = tfDict[i][j] * idfDict[j]

    return tfIdfDict

TF_IDF() 

The problem is with this line in the TF_IDF function - tfIdfDict[j] = tfDict[i][j] * idfDict[j]

The error occuring is this -

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-38-7b7d174d2ce3> in <module>()
      8     return tfIdfDict
      9 
---> 10 TF_IDF()

<ipython-input-38-7b7d174d2ce3> in TF_IDF()
      4     for i in tfDict:
      5         for j in i:
----> 6             tfIdfDict[j] = tfDict[i][j] * idfDict[j]
      7 
      8     return tfIdfDict

TypeError: list indices must be integers or slices, not dict

I understand where and what the problem but I cannot find a solution for it. Please help. Thank you!

Answer 1

tfDict is a list of dictionary. When you do for i in tfDict , i will essentially be having one of the elements (a dictionary) of tfDict and not the integer index.

This is completely fine until you do tfDict[i][j] because tfDict[i] would expect i to be an integer index and not an element value.

Solution : Do i[j] instead of tfDict[i][j] .