read log file by using readlines() in python

Question

I have sample log file in which for each msisdn value(eg "msisdn":"22969000034") there are 3 different state values like (eg '"state":"COMPLETED"', '"state":"IN_PROGRESS"' & "state":"ENTERED"). I have to create a report which contains column as "msisdn (which contains msisdn field value), requestID (which contains requestID field value), state1 (which contains state field value as ENTERED), state2 (which contains state field value as IN_PROGRESS), state3 (which contains state field value as COMPLETED), text (which contains text field value only if state value is COMPLETED for msisdn else if state value is IN_PROGRESS/ENTERED it will be 'NA' ) ".

Log file:

{"methodName":"offlineNotifQueueListener()","className":"com.ericsson.ms.business.jms.listener.impl.ActiveMqListener","text":"Processing Offline request","errorText":"NA","status":"NA","requestID":"OFFLINE_Notification_10.10.46.95_e6444dbc-b7ef-41d6-9111-eaa384717b27","orgTransactionID":"895139242296900003","msisdn":"22969000034","action":"Notification","input":"pre_Adhoc","sourceChannel":"OFFLINE","product":"1399","transactionDateTime":"2020-03-24T22:59:59+0100","state":"ENTERED"} {"methodName":"addNotificationText()","className":"com.ericsson.ms.business.service.notification.impl.MSProductNotificationImpl","text":"Evaluating notification for product : 1399","errorText":"NA","status":"NA","requestID":"OFFLINE_Notification_10.10.46.95_e6444dbc-b7ef-41d6-9111-eaa384717b27","orgTransactionID":"895139242296900003","msisdn":"22969000034","action":"Notification","input":"pre_Adhoc","sourceChannel":"OFFLINE","product":"1399","transactionDateTime":"2020-03-24T22:59:59+0100","state":"IN_PROGRE SS"} {"methodName":"executeConstraint()","className":"com.ericsson.ms.business.service.constraint.impl.MSConstraintsImpl","text":"Executing constraint : OfferCHeck","errorText":"NA","status":"NA","requestID":"OFFLINE_Notification_10.10.46.95_e6444dbc-b7ef-41d6-9111-eaa384717b27","orgTransactionID":"895139242296900003","msisdn":"22969000034","action":"Notification","input":"pre_Adhoc","sourceChannel":"OFFLINE","product":"NA","transactionDateTime":"2020-03-24T22:59:59+0100","state":"IN_PROGRESS"} {"methodName":"getValueFromAIR()","className":"com.ericsson.ms.business.service.templateparam.impl.TemplateParamServiceImpl","text":"Evaluating parameter --\> ${OFFER_123089_VALUE}","errorText":"NA","status":"NA","requestID":"OFFLINE_Notification_10.10.46.95_e6444dbc-b7ef-41d6-9111-eaa384717b27","orgTransactionID":"895139242296900003","msisdn":"22969000034","action":"Notification","input":"pre_Adhoc","sourceChannel":"OFFLINE","product":"1399","transactionDateTime":"2020-03-24T22:59:59+0100", "state":"IN_PROGRESS"} {"methodName":"executeCommand","className":"com.ericsson.ms.business.service.templateparam.impl.TemplateParamServiceImpl","text":"Executing AIR command ---\> GetOffers","errorText":"NA","status":"NA","requestID":"OFFLINE_Notification_10.10.46.95_e6444dbc-b7ef-41d6-9111-eaa384717b27","orgTransactionID":"895139242296900003","msisdn":"22969000034","action":"Notification","input":"pre_Adhoc","sourceChannel":"OFFLINE","product":"1399","transactionDateTime":"2020-03-24T22:59:59+0100","state":"COMPLETED"}

The code that i am using is giving output only from first line of log file. How can i read the complete file to get desired output?

code:

with open('home/msLog.log', 'r') as f:
    data = '['+f.readlines()[0].replace('} {','}, {')+']'
    print(data)
    data1 = json.loads(data)
#    print(data1)
    #i += 1
    for x in data1:
    #print(f"msisdn: {x['msisdn']}, status: {x['state']}, requestID: {x['requestID']}")
        print(x['msisdn'],x['state'],x['product'])
    if x['state'] == 'ENTERED':
        msd = x['msisdn']
        RID = x['requestID']
        state1 = x['state']
        value = msd, RID, state1
        list1.append(value)
    elif x['state'] == 'IN_PROGRESS' and x['msisdn'] == msd:
        state2 = x['state']
        list1.append(state2)
    elif x['state'] == 'COMPLETED' and x['msisdn'] == msd:
        state3 = x['state']
        text = x['text']
        value = state3, text
        list1.append(value)
print(list1)

Answer 1

It looks like the file is in jsonlines format - each line is a separate json object. So you can deserialise line by line, like this:

with open('home/msLog.log', 'r') as f:
    for line in f:
        x = json.loads(line)

        # Process the line (code unchanged from the question)
        if x['state'] == 'ENTERED':
            msd = x['msisdn']
            RID = x['requestID']
            state1 = x['state']
            value = msd, RID, state1
            list1.append(value)
        elif x['state'] == 'IN_PROGRESS' and x['msisdn'] == msd:
            state2 = x['state']
            list1.append(state2)
        elif x['state'] == 'COMPLETED' and x['msisdn'] == msd:
            state3 = x['state']
            text = x['text']
            value = state3, text
            list1.append(value)
print(list1)