每当pyspark中的一行出现任何错误词时，我如何获取文件中的下一行？

Question

I have a log file in which i need to check on each line. Whenever "ERROR" word come in any line then i need to take the next two line after that line. I have to do this in pyspark.

for example: Input log File:

line 1

line 2

line...ERROR... 3

line 4

line 5

line 6

Output will be :

line 4

line 5

I have created an rdd using the log file and using map() to traverse each line but i am not getting the exact idea.

Thanks in advance.

Answer 1

what about something like:

# open your file as f
lines = f.readlines()
for i, line in enumerate(lines):
    if "ERROR" in line:
        print(lines[i+1])
        print(lines[i+2])
        # Exit or something you want to do.

Answer 2

Here is a method using windowing functions:

from pyspark.sql import functions as F
from pyspark.sql.window import Window

# set up DF
df = sc.parallelize([["line1"], ["line2"], ["line3..ERROR"], ["line4"], ["line5"]]).toDF(['col'])

# create an indicator that created a boundary between consecutive errors
win1 = Window.orderBy('col')
df = df.withColumn('hit_error', F.expr("case when col like '%ERROR%' then 1 else 0 end"))
df = df.withColumn('cum_error', F.sum('hit_error').over(win1))

# now count the lines between each error occurrence
win2 = Window.partitionBy('cum_error').orderBy('col')
df = df.withColumn('rownum', F.row_number().over(win2))

# the lines we want are rows 2,3
df.filter("cum_error>0 and rownum in (2,3)").select("col").show(10)```