单词聚类列表列表

Question

Let's say I have a list of lists of words, for example

[['apple','banana'],
 ['apple','orange'],
 ['banana','orange'],
 ['rice','potatoes','orange'],
 ['potatoes','rice']]

The set is much bigger. I want to cluster the words that words usually existing together will have the same cluster. So in this case the clusters will be ['apple', 'banana', 'orange'] and ['rice','potatoes'] .
What is the best approach to archive this kind of clustering?

Answer 1

I think it is more natural to think of the problem as a graph.

You can assume for example that apple is node 0, and banana is node 1 and the first list indicates there is an edge between 0 to 1.

so first convert the labels to numbers:

from sklearn.preprocessing import LabelEncoder
le=LabelEncoder()
le.fit(['apple','banana','orange','rice','potatoes'])

now:

l=[['apple','banana'],
 ['apple','orange'],
 ['banana','orange'],
 ['rice','potatoes'], #I deleted orange as edge is between 2 points, you can  transform the triple to 3 pairs or think of different solution
 ['potatoes','rice']]

convert the labels to numbers:

edges=[le.transform(x) for x in l]

>>edges

[array([0, 1], dtype=int64),
array([0, 2], dtype=int64),
array([1, 2], dtype=int64),
array([4, 3], dtype=int64),
array([3, 4], dtype=int64)]

now, start to build the graph and add the edges:

import networkx as nx #graphs package
G=nx.Graph() #create the graph and add edges
for e in edges:
    G.add_edge(e[0],e[1])

now you can use the connected_component_subgraphs function to analyze connected vertices.

components = nx.connected_component_subgraphs(G) #analyze connected subgraphs
comp_dict = {idx: comp.nodes() for idx, comp in enumerate(components)}
print(comp_dict)

output:

{0: [0, 1, 2], 1: [3, 4]}

or

print([le.inverse_transform(v) for v in comp_dict.values()])

output:

[array(['apple', 'banana', 'orange']), array(['potatoes', 'rice'])]

and those are your 2 clusters.

Answer 2

It will be more meaningful to look for frequent itemsets instead.

If you cluster such short sets of words, everything will be connected at usually just a few levels: nothing in common, one element in common, two elements in common. That is too coarse to be usable for clustering. You'll get everything or nothing connected, and i0the results may be highly sensitive to data changes and ordering.

So abandoned the paradigm of partitioning the data - look for frequent combinations instead.

Answer 3

So, after lots of Googling around, I figured out that I, in fact, can't use clustering techniques because I lack feature variables on which I can cluster the words. If I make a table where I note how often each word exists with other words (in fact cartesian product) is in fact adjacency matrix and clustering doesn't work well on it.

So, the solution I was looking for is graph community detection. I used igraph library (or the python's python-ipgraph wrapper) to find the clusters and it runs very well and fast.

More informations:

• similar question: https://stats.stackexchange.com/questions/142297/finding-natural-groups-clusters-in-an-undirected-graph-over-several-undirect
• Community detection in graphs paper: https://arxiv.org/pdf/0906.0612.pdf
• Basic descrption of various algorithms: What are the differences between community detection algorithms in igraph?