How to iterate over a list, removing elements?

Question

I would like to iterate over a dictionary contained in a list and remove the ARRIVAL DATE TIME items that are older than a certain date. I've reviewed similar previous questions ( similar question , another one , and another one ) but given the complexity of my data structure I'm struggling to apply the solutions suggested. This is what I attempted so far. The solution seems I need to iterate on a duplicate of my data structure, but how do I do it?

from datetime import datetime
ferry = [ {
      "NAME":"SNAV Atlair",
      "LAST CALLING PORT":"Casamicciola Terme",
      "CALLING PORTS HISTORY":{
         "MONDAY":[
              {
               "PORT":"Casamicciola Terme",
               "ARRIVAL DATE TIME":"2019-03-19 20:12:10"
            },
            {
               "PORT":"Casamicciola Terme",
               "ARRIVAL DATE TIME":"2019-03-19 16:12:10"
            }
         ] ,
          "TUESDAY":[
            {
               "PORT":"Casamicciola Terme",
               "ARRIVAL DATE TIME":"2019-03-19 20:12:10"
            }
         ],
         "WEDNESDAY":[

         ],
         "THURSDAY":[

         ],
         "FRIDAY":[

         ],
         "SATURDAY":[

         ],
         "SUNDAY":[

         ]
      }
   }]

datetime_last_position = '2019-03-23 12:50:00'
for entry in list(ferry):
  for i in range(len(entry['CALLING PORTS HISTORY']['MONDAY'])):
    if datetime.strptime(entry['CALLING PORTS HISTORY']['MONDAY'][i]['ARRIVAL DATE TIME'], '%Y-%m-%d %H:%M:%S').date() < datetime.strptime(datetime_last_position, '%Y-%m-%d %H:%M:%S').date():
      del entry['CALLING PORTS HISTORY']['MONDAY'][i]

Answer 1

An alternative to deleting elements from the list (which can be done if you iterate in reverse order, as pointed out in another answer ) is to just create a new list containing only the elements that you want to keep in each case:

from datetime import datetime

ferry = [...]
datetime_last_position = '2019-03-23 12:50:00'
ref_date = datetime.strptime(datetime_last_position, '%Y-%m-%d %H:%M:%S').date()
for entry in list(ferry):
  callings = entry['CALLING PORTS HISTORY']
  for day in callings:
    callings[day][:] = [call for call in callings[day]
                        if datetime.strptime(call['ARRIVAL DATE TIME'], '%Y-%m-%d %H:%M:%S').date() >= ref_date]

print(ferry)
# [{'NAME': 'SNAV Atlair', 'LAST CALLING PORT': 'Casamicciola Terme', 'CALLING PORTS HISTORY': {'MONDAY': [], 'TUESDAY': [], 'WEDNESDAY': [], 'THURSDAY': [], 'FRIDAY': [], 'SATURDAY': [], 'SUNDAY': []}}]

Also, note that you are currently editing the objects within ferry , so the list(ferry) in for entry in list(ferry): is not really doing anything. If you want to create a new list with the modified objects preserving the old ones you need to make copies of each object, or just use copy.deepcopy before applying the modification.

Answer 2

If you iterate over the items in reverse order, you can safely remove them:

for i in range(len(entry['CALLING PORTS HISTORY']['MONDAY'])-1, -1, -1):
    if datetime.strptime(entry['CALLING PORTS HISTORY']['MONDAY'][i]['ARRIVAL DATE TIME'], '%Y-%m-%d %H:%M:%S').date() < datetime.strptime(datetime_last_position, '%Y-%m-%d %H:%M:%S').date():
        del entry['CALLING PORTS HISTORY']['MONDAY'][i]

Of course, you probably don't want to rewrite this for each weekday, so something like this might be better:

for weekday in entry["CALLING PORTS HISTORY"]:
    for i in range(len(entry['CALLING PORTS HISTORY'][weekday])-1, -1, -1):
        if datetime.strptime(entry['CALLING PORTS HISTORY'][weekday][i]['ARRIVAL DATE TIME'], '%Y-%m-%d %H:%M:%S').date() < datetime.strptime(datetime_last_position, '%Y-%m-%d %H:%M:%S').date():
            del entry['CALLING PORTS HISTORY'][weekday][i]

Answer 3

You can "skip" element of an entry if it should be deleted and then replace it with the next acceptable element, and then shorten the list to the count of "unskipped" elements:

datetime_last_position = '2019-03-23 12:50:00'
for entry in list(ferry):
    ptr = 0  # index where algorithm will store element
    for value in entry['CALLING PORTS HISTORY']['MONDAY']:
        if datetime.strptime(
            entry['CALLING PORTS HISTORY']['MONDAY'][i]['ARRIVAL DATE TIME'],
            '%Y-%m-%d %H:%M:%S'
        ).date() < datetime.strptime(datetime_last_position, '%Y-%m-%d %H:%M:%S').date():
            pass  # say, skip element
        else:
            # write then increment pointer
            entry['CALLING PORTS HISTORY']['MONDAY'][ptr] = value
            ptr += 1
    # shorten a list
    entry['CALLING PORTS HISTORY']['MONDAY'] = entry['CALLING PORTS HISTORY']['MONDAY'][:ptr]

ferry
Out:
[{'CALLING PORTS HISTORY': {'FRIDAY': [],
   'MONDAY': [],
   'SATURDAY': [],
   'SUNDAY': [],
   'THURSDAY': [],
   'TUESDAY': [{'ARRIVAL DATE TIME': '2019-03-19 20:12:10',
     'PORT': 'Casamicciola Terme'}],
   'WEDNESDAY': []},
  'LAST CALLING PORT': 'Casamicciola Terme',
  'NAME': 'SNAV Atlair'}]

It has O(n) complexity and no unexpected behaviour.

Answer 4

From what I understand of your question, you want to remove the elements in each weekday list that whose ARRIVAL DATE TIME is older than a given date.

Since the dates you are supplying are strings, we'll need to create a function to convert them to datetime objects:

strToDate = lambda date_string: datetime.strptime(date_string, '%Y-%m-%d %H:%M:%S')

From there, it's a pretty simple matter of looping through your various elements (assuming your structure is constant) and creating a new list for each weekday based on what you want. You can modify the dictionary in place, but not the list, which is why I use a list comprehension below to create the new list:

for item in ferry:
    for weekday in item["CALLING PORTS HISTORY"]:
        item["CALLING PORTS HISTORY"][weekday] = [subitem 
                                                  for subitem in item["CALLING PORTS HISTORY"][weekday] 
                                                  if strToDate(subitem["ARRIVAL DATE TIME"]) >= date]

This will keep those objects whose dates are earlier than or equal to date while eliminating those objects that aren't, resulting in the structure you want!