Drop pandas dataframe rows AND columns in a batch fashion based on value

Question

Background: I have a matrix which represents the distance between two points. In this matrix both rows and columns are the data points. For example:

   A   B   C
 A 0   999 3
 B 999 0   999
 C 3   999 0

In this toy example let's say I want to drop C for some reason, because it is far away from any other point. So I first aggregate the count:

df["far_count"] = df[df == 999].count()

and then batch remove them:

df = df[df["far_count"] == 2]

In this example this looks a bit redundant but please imagine that I have many data points like this (say in the order of 10Ks)

The problem with the above batch removal is that I would like to remove rows and columns in the same time (instead of just rows) and it is unclear to me how to do so elegantly. A naive way is to get a list of such data points and put it in a loop and then:

for item in list:
  df.drop(item, axis=1).drop(item, axis=0)

But I was wondering if there is a better way. (Bonus if we could skip the intermdiate step far_count )

Answer 1

np.random.seed([3,14159])
idx = pd.Index(list('ABCDE'))
a = np.random.randint(3, size=(5, 5))
df = pd.DataFrame(
    a.T.dot(a) * (1 - np.eye(5, dtype=int)),
    idx, idx)

df

   A  B  C  D  E
A  0  4  2  4  2
B  4  0  1  5  2
C  2  1  0  2  6
D  4  5  2  0  3
E  2  2  6  3  0

---

l = ['A', 'C']
m = df.index.isin(l)

df.loc[~m, ~m]

   B  D  E
B  0  5  2
D  5  0  3
E  2  3  0

---

For your specific case, because the array is symmetric you only need to check one dimension.

m = (df.values == 999).sum(0) == len(df) - 1

Answer 2

In [66]: x = pd.DataFrame(np.triu(df), df.index, df.columns)

In [67]: x
Out[67]:
   A    B    C
A  0  999    3
B  0    0  999
C  0    0    0

In [68]: mask = x.ne(999).all(1) | x.ne(999).all(0)

In [69]: df.loc[mask, mask]
Out[69]:
   A  C
A  0  3
C  3  0