PostgreSQL 查詢在存儲 PII 數據的多個表列中搜索多個詞

Question

有沒有一種有效的方法來搜索多個單詞，我的意思是在包含 PII 數據的表字段中出現 50 個單詞

截至目前我有這個查詢

 SELECT reporter FROM case_detail cd WHERE (cd.reporter)::text IN (‘ID_ISSUE_PLACE’,‘USER_NAME’,‘STATE’,‘PROOFTYPE_1’,‘ID_EXPIRES_ALLOWED’,‘KIN_MIDDLENAME’,‘EMAIL’,‘GENDER’,‘COUNTRY’,‘PROOFTYPE_2’,‘RESIDENCE_COUNTRY’,‘KIN_AGE’,‘ID_TYPE’,‘CONTACT_PERSON’,‘BIRTHPLACE’,‘PROOFTYPE_3’,‘NATIONALITY’,‘KIN_CONTACTNUMBER’,‘ID_NO’,‘CONTACT_NO’,‘NOMINATION_DETAIL’,‘MIDDLE_NAME’,‘EMPLOYER_NAME’,‘KIN_NATIONALITY’,‘SSN’,‘MSISDN’,‘IMSI’,‘LAST_NAME’,‘DOB’,‘POSTAL_CODE’,‘KIN_NATIONALITYNO’,‘ADDRESS1’,‘DIST_MSISDN’,‘ID_ISSUE_DATE’,‘REFERENCEID’,‘KIN_RELATIONSHIP’,‘ADDRESS2’,‘RET_MSISDN’,‘ID_ISSUE_COUNTRY’,‘REGION’,‘SOURCE_OF_INCOME’,‘CITY’,‘BUSINESS_NAME’,‘ID_EXPIRY_DATE’,‘KIN_FIRSTNAME’,‘ORGANIZATION_NAME’,‘PROOFID_1’,‘PROOFID_2’,'PROOFID_3’,'KIN_LASTNAME’,‘Wallet')

這是拋出 SQL 語法錯誤

我們需要加密包含在這些表字段中的數據

任何幫助將不勝感激

   SELECT reporter FROM case_detail cd WHERE (cd.reporter)::text IN (‘ID_ISSUE_PLACE’,‘USER_NAME’,‘STATE’,‘PROOFTYPE_1’,‘ID_EXPIRES_ALLOWED’,‘KIN_MIDDLENAME’,‘EMAIL’,‘GENDER’,‘COUNTRY’,‘PROOFTYPE_2’,‘RESIDENCE_COUNTRY’,‘KIN_AGE’,‘ID_TYPE’,‘CONTACT_PERSON’,‘BIRTHPLACE’,‘PROOFTYPE_3’,‘NATIONALITY’,‘KIN_CONTACTNUMBER’,‘ID_NO’,‘CONTACT_NO’,‘NOMINATION_DETAIL’,‘MIDDLE_NAME’,‘EMPLOYER_NAME’,‘KIN_NATIONALITY’,‘SSN’,‘MSISDN’,‘IMSI’,‘LAST_NAME’,‘DOB’,‘POSTAL_CODE’,‘KIN_NATIONALITYNO’,‘ADDRESS1’,‘DIST_MSISDN’,‘ID_ISSUE_DATE’,‘REFERENCEID’,‘KIN_RELATIONSHIP’,‘ADDRESS2’,‘RET_MSISDN’,‘ID_ISSUE_COUNTRY’,‘REGION’,‘SOURCE_OF_INCOME’,‘CITY’,‘BUSINESS_NAME’,‘ID_EXPIRY_DATE’,‘KIN_FIRSTNAME’,‘ORGANIZATION_NAME’,‘PROOFID_1’,‘PROOFID_2’,'PROOFID_3’,'KIN_LASTNAME’,‘Wallet')

還有這個

SELECT reporter FROM case_detail WHERE reporter CONTAINS 'ID_ISSUE_PLACE And USER_NAME And STATE And PROOFTYPE_1 And ID_EXPIRES_ALLOWED And KIN_MIDDLENAME And EMAIL And GENDER And COUNTRY And PROOFTYPE_2 And RESIDENCE_COUNTRY And KIN_AGE And ID_TYPE And CONTACT_PERSON And BIRTHPLACE And PROOFTYPE_3 And NATIONALITY And KIN_CONTACTNUMBER And ID_NO And CONTACT_NO And NOMINATION_DETAIL And MIDDLE_NAME And EMPLOYER_NAME And KIN_NATIONALITY And SSN And MSISDN And IMSI And LAST_NAME And DOB And POSTAL_CODE And KIN_NATIONALITYNO And ADDRESS1 And DIST_MSISDN And ID_ISSUE_DATE And REFERENCEID And KIN_RELATIONSHIP And ADDRESS2 And RET_MSISDN And ID_ISSUE_COUNTRY And REGION And SOURCE_OF_INCOME And CITY And BUSINESS_NAME And ID_EXPIRY_DATE And KIN_FIRSTNAME And ORGANIZATION_NAME And PROOFID_1 And PROOFID_2 And PROOFID_3 And KIN_LASTNAME And Wallet';

Answer 1

您可以將 reporter 存儲為 tsvector，這有效地將文本轉換為可索引的單詞數組。 可以使用以下查詢將單獨的自動生成的 tsvector 列添加到表中。

ALTER TABLE case_detail ADD COLUMN
text_search tsvector GENERATED ALWAYS AS to_tsvector('english', reporter);

然后，您可以使用 @@ contains 運算符和適當的 tsquery 搜索相關案例。

SELECT reporter
FROM case_detail cd
WHERE cd.text_search @@ to_tsquery('word1 & word2 & ... & wordn');

您甚至可以在表中添加 GIN 索引，以提高大型表的文本搜索性能。

CREATE INDEX case_detail_text_search_idx ON case_detail USING GIN (text_search);

但是，您應該注意的一件事是，由於下划線不是常規單詞的一部分，postgres 將假定由下划線分隔的文本各自是它們自己的單詞，因此 'USER_NAME' 和 'USER' & 'NAME' 被相同對待。 如果包含下划線的文本是必須的，並且您對文本搜索的要求僅限於一組包含另一組關鍵字的關鍵字，那么也可以將文本存儲為文本數組。 在創建生成的列時，只需將 to_tsvector('english', reporter) 替換為 regexp_split_to_array(reporter, '\s+') ，使用 @> ['word1, 'word2', ...] 代替 select 並且 GIN 索引可以與 tsvector 一樣在 text[] 類型上創建。