Spark在csv文件python的所有行之间找到特定的字符串

Question

I am using pyspark and i have a large csv file. The csv file is having multiple lines

<ABCosmswkmwPQR>
<ABCasdfasdfadsPQR>
 ...
 ...

I need to iterate through each line and find the text between the particular string in it. I am using regex to do it

text_file = sc.textFile("file:///path/subset.tsv")
s = text_file.first()
conf = SparkConf().setAppName('MyFirstStandaloneApp')
links = re.findall(r'ABC(.*?)\PQR', s)


sc = SparkContext(conf=conf)

But I am only able to do this for only the first line. How do i do it for all files of the line. I need to iterate line by line and write the output of matched regex to a list if it fits into memory or a file.

I have opened the file using sparkcontext and I have to do the same since I have to read the file from HDFS.

Answer 1

Try something like this:

read_lines = open("file.csv", "r")
for line in read_lines:
    #if line matches regex:
        #do something

read_lines reads the entire file and the for loop will loop each line in the file. You just have to plug in the regex code.

Answer 2

You can use regexp_extract from module pyspark.sql.functions . IF your file is temp.csv

spark.createDataFrame(sc.textFile("temp.csv"), schema=StringType()).
                        select(regexp_extract(regexpattern, col, idx)