Python - 从大型 .csv 文件中的文本文件中搜索字符串列表

Question

Apologies, as I am probably making a whole host of errors here, but I am trying to search with a list of strings from a file (justgenes.txt) against a large CSV file and return the lines featuring the strings from the justgenes list.

I've been working largely with BASH, but the code I have takes more than 100GB of memory and crashes:

grep -f justgenes.txt allDandHunique.csv > HPCgenesandbugs.csv

Therefore, I am attempting to do it in python, assuming that it will be more efficient, but I have very little knowledge of it.

I use this code (which I've grabbed from the web), but getting an empty file at the end:

data = open('allDandHunique.csv')
                
with open('justgenes.txt', "r+") as file1:
    fileline1= file1.readlines()
    for x in data: # <--- Loop through the list to check      
        for line in fileline1: # <--- Loop through each line
            if x in line:
                 print(x)

The justgenes file looks like this:

1A0N_B
1A1A_A
1A4I_A
1A5Y_A
1ACO_A
1AGN_A
1AGS_A
1AJE_A
1AJJ_A
1AP0_A
1APQ_A

whilst the csv looks like this:

"0403181A:PDB=1BP2,2BPP",
"0403181A:PDB=1BP2,2BPP",,,
"0706243A:PDB=1HOE,2AIT,3AIT,4AIT",
"0706243A:PDB=1HOE,2AIT,3AIT,4AIT",,,
"1309311A:PDB=1EMD,2CMD",
"1309311A:PDB=1EMD,2CMD",,,
"1513188A:PDB=1BBC,1POD",
"1513188A:PDB=1BBC,1POD",,,
0308206A,
0308206A,,,
0308221A,
0308221A,,,
0308230A,
0308230A,,,

Any help would be gratefully received.

Answer 1

I would use pandas to accomplish this.

Try something like:

import pandas as pd

df = pd.read_csv('allDandHunique.csv')

with open('justgenes.txt', "r+") as file1:
    fileline1= file1.readlines()
    for x in fileline1: 
      for col in df:
         if col.str.contains(x, regex=False):
             ##do something here##

If when reading the file in you are getting a blank file, I would check and make sure the path is correct.

Answer 2

Since I don't have the files, I couldn't test it myself but I assume this code might help.

data = open('allDandHunique.csv')
        
for x in data: # <--- Loop through the list to check      
    with open('justgenes.txt', "r+") as file1:
        fileline1= file1.readlines()
        for line in fileline1: # <--- Loop through each line
            if x in line:
                    print(x)

For each x in data you have to loop over all lines in file1. If I'm not wrong, you need to open your file for each iteration, otherwise, when you reach EOF it returns nothing then.

Answer 3

import csv

with open('justGenes') as infile:
    searchTargets = set(line.strip() for line in infile)


with open('allDandHunique.csv') as infile:
    for row in csv.reader(infile):
        if any(target in row for target in searchTargets):
            print(row)