如何從文本語料庫構造 PPMI 矩陣？

Question

我正在嘗試使用 SVD model 在布朗語料庫上進行詞嵌入。 為此，我想首先生成一個詞-詞共現矩陣，然后轉換為 PPMI 矩陣以進行 SVD 矩陣乘法過程。

我嘗試使用 SkLearn CountVectorizer 創建一個共現

count_model = CountVectorizer(ngram_range=(1,1))

X = count_model.fit_transform(corpus)
X[X > 0] = 1
Xc = (X.T * X)
Xc.setdiag(0)
print(Xc.todense())

但：

(1) 我不確定如何用這種方法控制上下文 window？ 我想嘗試各種上下文大小，看看它們對過程的影響。

(2) 假設 PMI(a, b) = log p(a, b)/p(a)p(b)，我該如何正確計算 PPMI

任何有關思考過程和實施的幫助將不勝感激！

謝謝 （-：

Answer 1

我嘗試使用提供的代碼，但無法將移動的 window 應用於它。 所以，我做了我自己的 function 這樣做。 This function takes a list of sentences and returns a pandas.DataFrame object representing the co-occurrence matrix and a window_size number:

def co_occurrence(sentences, window_size):
    d = defaultdict(int)
    vocab = set()
    for text in sentences:
        # preprocessing (use tokenizer instead)
        text = text.lower().split()
        # iterate over sentences
        for i in range(len(text)):
            token = text[i]
            vocab.add(token)  # add to vocab
            next_token = text[i+1 : i+1+window_size]
            for t in next_token:
                key = tuple( sorted([t, token]) )
                d[key] += 1

    # formulate the dictionary into dataframe
    vocab = sorted(vocab) # sort vocab
    df = pd.DataFrame(data=np.zeros((len(vocab), len(vocab)), dtype=np.int16),
                      index=vocab,
                      columns=vocab)
    for key, value in d.items():
        df.at[key[0], key[1]] = value
        df.at[key[1], key[0]] = value
    return df

讓我們嘗試一下以下兩個簡單的句子：

>>> text = ["I go to school every day by bus .",
            "i go to theatre every night by bus"]
>>> 
>>> df = co_occurrence(text, 2)
>>> df
         .  bus  by  day  every  go  i  night  school  theatre  to
.        0    1   1    0      0   0  0      0       0        0   0
bus      1    0   2    1      0   0  0      1       0        0   0
by       1    2   0    1      2   0  0      1       0        0   0
day      0    1   1    0      1   0  0      0       1        0   0
every    0    0   2    1      0   0  0      1       1        1   2
go       0    0   0    0      0   0  2      0       1        1   2
i        0    0   0    0      0   2  0      0       0        0   2
night    0    1   1    0      1   0  0      0       0        1   0
school   0    0   0    1      1   1  0      0       0        0   1
theatre  0    0   0    0      1   1  0      1       0        0   1
to       0    0   0    0      2   2  2      0       1        1   0

[11 rows x 11 columns]

現在，我們有了同現矩陣。 讓我們找到（正）逐點互信息或簡稱 PPMI。 我使用了本幻燈片中提供的斯坦福教授 Christopher Potts 提供的代碼，可以在下圖中進行總結

PPMI 與以下帶有positive=True的pmi相同：

def pmi(df, positive=True):
    col_totals = df.sum(axis=0)
    total = col_totals.sum()
    row_totals = df.sum(axis=1)
    expected = np.outer(row_totals, col_totals) / total
    df = df / expected
    # Silence distracting warnings about log(0):
    with np.errstate(divide='ignore'):
        df = np.log(df)
    df[np.isinf(df)] = 0.0  # log(0) = 0
    if positive:
        df[df < 0] = 0.0
    return df

讓我們試一試：

>>> ppmi = pmi(df, positive=True)
>>> ppmi
                .       bus        by  ...    school   theatre        to
.        0.000000  1.722767  1.386294  ...  0.000000  0.000000  0.000000
bus      1.722767  0.000000  1.163151  ...  0.000000  0.000000  0.000000
by       1.386294  1.163151  0.000000  ...  0.000000  0.000000  0.000000
day      0.000000  1.029619  0.693147  ...  1.252763  0.000000  0.000000
every    0.000000  0.000000  0.693147  ...  0.559616  0.559616  0.559616
go       0.000000  0.000000  0.000000  ...  0.847298  0.847298  0.847298
i        0.000000  0.000000  0.000000  ...  0.000000  0.000000  1.252763
night    0.000000  1.029619  0.693147  ...  0.000000  1.252763  0.000000
school   0.000000  0.000000  0.000000  ...  0.000000  0.000000  0.559616
theatre  0.000000  0.000000  0.000000  ...  0.000000  0.000000  0.559616
to       0.000000  0.000000  0.000000  ...  0.559616  0.559616  0.000000

[11 rows x 11 columns]