How to Concatenate bunch of videos using python?

Question

So, I have over 5000 small clips that I need to combine. To apply various custom filter over their names, I want to do it with python. I have the following code:

import os
from moviepy.editor import *
os.chdir('D:/videos')
list1, list2 = os.listdir(), []

for i in list1: #filtering
    if i[-6:] != '-l.mp4' and i[-7:] != 'ALT.mp4':
        list2.append(i)
print('Getting Video Info:')

final = VideoFileClip(list2[0])


for i in range(1,len(list2)):
    final = concatenate_videoclips([final, VideoFileClip(list2[i])])
    print('\r' + str(i+1) + '/' + str(len(list2)), end='')


os.chdir('D:')
final.write_videofile('Merged.mp4')

But the program is creating lots of process and just after reading 150 clips it's crashing. It keeps increasing? Is there any easier way/alternative to do this? Thanks!

Edit:
I've tried using ffmpeg too, but concatenation removes the audio since concat protocol doesn't support.mp4 extension. In that case. Even if I convert all the files to.ts extension and try to concatenate them , WindowsError: [Error 206] The filename or extension is too long pops up because of too many files separated by |. I did the following changes after converting all the files to.ts format:

import os
import ffmpeg
os.chdir('D:/videos')
list1 = os.listdir()
list2 = [i for i in list1 if i[-3:] == '.ts']
list2[0] = ffmpeg.input(list2[0])
for i in range(1, len(list2)):
    list2[i] = ffmpeg.concat(list2[i-1], ffmpeg.input(list2[i]))
    print('\r' + str(i) + '/' + str(len(list2)), end='')
ffmpeg.output(list2[-1], 'D:\Merged.mp4')
ffmpeg.run(list2[-1])

But now I'm getting RecursionError: maximum recursion depth exceeded while calling a Python object .

Answer 1

Can you try deleting explicitly and garbage collection in between like this.

import os
import gc
from moviepy.editor import *
os.chdir('D:/videos')
list1, list2 = os.listdir(), []

for i in list1: #filtering
    if i[-6:] != '-l.mp4' and i[-7:] != 'ALT.mp4':
        list2.append(i)
print('Getting Video Info:')

final = VideoFileClip(list2[0])


for i in range(1,len(list2)):
    curclip = VideoFileClip(list2[i])
    final = concatenate_videoclips([final, curclip])
    print('\r' + str(i+1) + '/' + str(len(list2)), end='')
    curclip.close()
    del curclip
    gc.collect()

os.chdir('D:')
final.write_videofile('Merged.mp4')

If above doesn't work, try saving final video at a certain frequency eg 8. Saving and reading as raw file will release extra memory accumulating inside final object

for i in range(1,len(list2)):
    if (i % 8) == 7:
        final.write_videofile('D:/Merged.mp4')
        final = VideoFileClip('D:/Merged.mp4')
    curclip = VideoFileClip(list2[i])
    final = concatenate_videoclips([final, curclip])
    print('\r' + str(i+1) + '/' + str(len(list2)), end='')
    curclip.close()
    del curclip
    gc.collect()

os.chdir('D:')
final.write_videofile('Merged.mp4')