如何使用 seaborn 有效地 plot 距离矩阵？

Question

. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 17 hours . JayJona wants to to this question.

So I have a dataset of more ore less 11.000 records, with 4 features all them are discrete or continue. I perform clustering using K-means, then I add the column "cluster" to the dataframe using kmeans.labels_ . Now I want to plot the distance matrix so I used pdist from scipy , but the matrix is not plotted.

Here is my code.

from scipy.spatial.distance import pdist
from scipy.spatial.distance import squareform
import gc

# distance matrix
def distance_matrix(df_labeled, metric="euclidean"):
    df_labeled.sort_values(by=['cluster'], inplace=True)
    dist = pdist(df_labeled, metric)
    dist = squareform(dist)    
    sns.heatmap(dist, cmap="mako")
    print(dist)
    del dist
    gc.collect()

distance_matrix(finalDf)

Output:

[[ 0.          2.71373462  3.84599479 ...  7.59910903  8.10265588
   8.27195104]
 [ 2.71373462  0.          2.94410672 ...  7.90444283  8.28225031
   8.48094661]
 [ 3.84599479  2.94410672  0.         ...  9.78706347 10.42014451
  10.61261498]
 ...
 [ 7.59910903  7.90444283  9.78706347 ...  0.          1.27795469
   1.44711258]
 [ 8.10265588  8.28225031 10.42014451 ...  1.27795469  0.
   0.52333107]
 [ 8.27195104  8.48094661 10.61261498 ...  1.44711258  0.52333107
   0.        ]]

I get the following graph:

As you can see, the plot is empty. Also I have to free up some RAM because google colab crashes.

How can I solve the problem?

Answer 1

It is possible that the issue could be with the size of the distance matrix. 11,000 records would result in a distance matrix with 121,110,000 elements, which may be too large to effectively plot. One solution could be to try reducing the number of records in the dataframe before performing the distance matrix calculation. This could be done through techniques such as sampling or feature selection.

Another potential issue could be with the metric being used. Some metrics, such as Euclidean distance, may not be effective for discrete or categorical features. It may be worth trying a different metric, such as Manhattan distance, which is more suitable for these types of features.

It is also possible that there could be other issues with the code, such as incorrect imports or syntax errors. It may be worth double checking the code and ensuring that all necessary libraries are imported and properly referenced.

Answer 2

The original question was well-phrased but was not a reprex . Its code, at least the part we can see, appears to work fine.

Here is a demo of producing a heatmap for another dataset that also has 11 K rows.

from scipy.spatial.distance import pdist, squareform
from uszipcode import SearchEngine
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns


def distance_matrix(df: pd.DataFrame, metric="euclidean"):
    df = df[["zipcode", "lat", "lng", "population_density"]]
    df = df.sort_values(by=["zipcode"])
    print(df)
    dist = pdist(df, metric)
    dist = squareform(dist)
    sns.heatmap(dist, cmap="mako")
    print(dist)
    plt.show()


def get_df() -> pd.DataFrame:
    zips = SearchEngine().by_population_density(lower=100, returns=11_000)
    df = pd.DataFrame(z.to_dict() for z in zips)
    df["zipcode"] = df.zipcode.astype(int)
    return df


distance_matrix(get_df())

It consumes at least ten GiB under MacOS 12.6.2, using cPython 3.10.8, matplotlib 3.6.2, scipy 1.9.3, seaborn 0.12.1.

It displays this:

Answer 3

To plot a distance matrix using seaborn, you can use the seaborn.heatmap() function.

Answer 4

It is possible that the distance matrix is too large to be plotted using a heatmap. The sns.heatmap function expects a 2D array, but the output of pdist is a 1D array of distances between all pairs of points in the input. When you use squareform to convert this array to a 2D distance matrix, the resulting matrix may have too many rows and columns to be plotted effectively.

One solution to this problem is to downsample the distance matrix by selecting a subset of the rows and columns to plot. You can do this by using the iloc indexer to select a slice of the distance matrix:

sns.heatmap(dist.iloc[:100, :100], cmap="mako")

This will plot a 100x100 submatrix of the distance matrix, which may be small enough to be plotted effectively. You can adjust the size of the submatrix as needed to find a balance between detail and legibility.

Alternatively, you can try using a different plot type to visualize the distance matrix. For example, you could use a scatterplot to plot the distance between each pair of points, or you could use a dendrogram to show the hierarchical structure of the distances.