pyqt5 qlistwidget performance with large text data

Question

I am writing a pyqt5 GUI application to parse a specific log file with the below structure. Each line is already split up as strings in a list and stored as self.calls. This list stores about 60K lines of data.

The goal is to search using user input string, find all call ID (1,2,3,etc) that has the input string in the line. Then find all messages relating to that call ID and display them in order using qlistwidget (I like the clean look of Qlistwidget. I've tried Qplaintext, but it took a long time to display just self.calls)

• I have a qlineedit to gather user input
• I take that input, use regex to find each item in self.calls that match user input
• Use regex again to find all the call ID number (of lines that match user input) and put them in a list
• Then use for loops to add all lines with that call ID to qlistwidget

I can use qlistwidget addItem and a simple for loop to display all 50K lines in self.calls, and it takes a few seconds (acceptable).

for call in self.calls:
        self.output.addItem(call)

My problem is when I implement my search function, if there are large amount of lines matching, the app struggles badly and take a long time to display the data even though it's a smaller subset of data from self.calls. I tried using nested for loops with conditional, list comprehension, and for loops with regex. All roughly performed the same, 10K lines of parsed data took maybe 20-30 seconds.

Hoping someone can give me pointers of a faster way to display the data using qlistwidget.

Data:

Oct 12 18:38:34 user.info server1 host:server:  INFO : call 1: allocated for "Web client" conference participation
Oct 12 18:38:34 user.info server1 host:server:  INFO : call 1: setting up combined RTP session for DTLS (combined media and control)
Oct 12 18:38:34 user.info server1 host:server:  INFO : call 1: starting DTLS combined media negotiation (as initiator)
Oct 12 18:38:35 user.info server1 host:server:  INFO : call 1: completed DTLS combined media negotiation
Oct 12 18:38:35 user.info server1 host:server:  INFO : call 1: media framework reporting rx video RTP frequency 0 - fixed up to 90000
Oct 12 18:38:35 user.info server1 host:server:  INFO : call 1: starting DTLS combined media negotiation (as initiator)
Oct 12 18:38:36 user.info server1 host:server:  INFO : call 1: completed DTLS combined media negotiation
Oct 12 18:38:59 user.info server1 host:server:  INFO : call 1: tearing down (conference media)
Oct 12 18:51:27 user.info server1 host:server:  INFO : call 2: recognised as Avaya
Oct 12 18:51:27 user.info server1 host:server:  INFO : call 2: incoming SIP audio call from
Oct 12 18:51:40 user.info server1 host:server:  INFO : call 3: outgoing encrypted SIP call to 
Oct 12 18:51:40 user.info server1 host:server:  INFO : call 3: setting up peer to peer media instantiation
Oct 12 18:51:40 user.info server1 host:server:  INFO : call 3: setting up UDT RTP session for DTLS (combined media and control)
Oct 12 18:51:40 user.info server1 host:server:  INFO : call 3: SIP call ringing
Oct 12 18:51:40 user.info server1 host:server:  INFO : call 3: determined far end good lip sync status, 1
Oct 12 18:51:42 user.info server1 host:server:  INFO : call 3: remote layout version supported by peer now 6
Oct 12 18:51:44 user.info server1 host:server:  INFO : call 4: allocated for "Web client" conference participation
Oct 12 18:51:44 user.info server1 host:server:  INFO : call 4: setting up combined RTP session for DTLS (combined media and control)
Oct 12 18:51:44 user.info server1 host:server:  INFO : call 4: starting DTLS combined media negotiation (as initiator)
Oct 12 18:51:44 user.info server1 host:server:  INFO : call 4: completed DTLS combined media negotiation
Oct 12 18:52:47 user.info server1 host:server:  INFO : call 3: ending; local teardown - connected for 1:07

Code:

if len(self.searchbar.text()) > 2:

    self.output_filtered.setHidden(False)

    #Regex compiles
    re_searchstring = re.compile(r"(^.*?%s.*?$)" % self.searchbar.text(), re.IGNORECASE)
    re_callindex = re.compile(r"call\s(\d+):")

    #Get results of all calls matching searchbar input
    result = list(filter(re_searchstring.match, self.calls))

    #Build a temp list with call index number
    call_list_temp = []
    for item in result:
        callindex = re_callindex.findall(item)
        call_list_temp.append(callindex)
    #Merge the call list, remove dups, and sort
    call_list_temp = list(itertools.chain(*call_list_temp))
    call_list = list(OrderedDict.fromkeys(call_list_temp))

    ##############################
    # For loops with conditional
    for index in call_list:
        for calls in self.calls:
            if "call " + str(index) + ":" in calls:
                self.output_filtered.addItem(calls)

    # List Comprehension
    test = [calls for index in call_list for calls in self.calls if "call " + str(index) + ":" in calls]
    for call in test:
        self.output_filtered.addItem(call)

    # For loops with regex
    for index in call_list:
        re_callfinder = re.compile(r"(^.*call\s%s.*$)" % index)
        for item in self.calls:
            call = re_callfinder.findall(item)
            for line in call:
                self.output_filtered.addItem(line)

Answer 1

You could switch to a Model/View system , and use QListView instead of QListWidget. That way you will be able to take advantage of the QModel class which is designed for this kind of jobs...