How do I sync 4 videos in a grid to play the same frame at the same time?

Question

• 4 of us have recorded ourselves playing a game and want to create a 4 x 4 video grid
• The game has cutscenes at the beginning followed by each person having their unique part for the rest of the video
• I am looking to synchronize the grid such that it starts at the same place in the cutscene for everyone
• Kindly take a look at what is happening currently. The cutscene is off by a few seconds for everyone
• Imagine a time offset a,b,c,d such that when I add this offet to each video, the entire video grid will be in sync
• How to find this a,b,c,d and more importantly how to add it in filter_complex

I used the ffmpeg command below to generate a 4 x 4 video grid and it seems to work

ffmpeg
    -i nano_prologue.mkv -i macko_nimble_guardian.mkv -i nano_nimble_guardian.mkv -i ghost_nimble_guardian_subtle_arrow_1.mp4
    -filter_complex "
        nullsrc=size=1920x1080 [base];
        [0:v] setpts=PTS-STARTPTS, scale=960x540 [upperleft];
        [1:v] setpts=PTS-STARTPTS, scale=960x540 [upperright];
        [2:v] setpts=PTS-STARTPTS, scale=960x540 [lowerleft];
        [3:v] setpts=PTS-STARTPTS, scale=960x540 [lowerright];
        [base][upperleft] overlay=shortest=1 [tmp1];
        [tmp1][upperright] overlay=shortest=1:x=960 [tmp2];
        [tmp2][lowerleft] overlay=shortest=1:y=540 [tmp3];
        [tmp3][lowerright] overlay=shortest=1:x=960:y=540
    "
    -c:v libx264 output.mkv

My problem though is that since each of us starts recording at slightly different times, the cutscenes are out of sync

As per the screenshot below, you can see that each video has the same scene starting at a slightly different time.

Is there a way to find where the same frame will start on all videos and then sync each video to start from that frame or 20 seconds before that frame?

UPDATE 1

i have figured out the offset for each video in millisecond precision using the following technique

take a screenshot of the first video at a particular point in the cutscene and save image as png and run the script below for the remaining 3 videos to find out where this screenshot appears in each video


ffmpeg -i "video2.mp4" -r 1 -loop 1 -i screenshot.png -an -filter_complex "blend=difference:shortest=1,blackframe=90:32" -f null -

Use the command above to search for the offset in every video for that cutscene

It gave me this

VIDEO 3 OFFSET

[Parsed_blackframe_1 @ 0x600003af00b0] frame:3144 pblack:92 pts:804861 t:52.399805 type:P last_keyframe:3120

[Parsed_blackframe_1 @ 0x600003af00b0] frame:3145 pblack:96 pts:805117 t:52.416471 type:P last_keyframe:3120

VIDEO 2 OFFSET

[Parsed_blackframe_1 @ 0x6000014dc0b0] frame:3629 pblack:91 pts:60483 t:60.483000 type:P last_keyframe:3500

VIDEO 4 OFFSET

[Parsed_blackframe_1 @ 0x600002f84160] frame:2885 pblack:93 pts:48083 t:48.083000 type:P last_keyframe:2880

[Parsed_blackframe_1 @ 0x600002f84160] frame:2886 pblack:96 pts:48100 t:48.100000 type:P last_keyframe:2880

Now how do I use filter_complex to say start each video at either the frame above or the timestamp above?. I would like to include say 10 seconds before the above frame in each video so that it starts from the beginning

UPDATE 2

This command currently gives me a 100% synced video, how do I make it start 15 seconds before the specified frame numbers and how to make it use the audio track from video 2 instead?

ffmpeg
    -i v_nimble_guardian.mkv -i macko_nimble_guardian.mkv -i ghost_nimble_guardian_subtle_arrow_1.mp4 -i nano_nimble_guardian.mkv
    -filter_complex "
        nullsrc=size=1920x1080 [base];
        [0:v] trim=start_pts=49117,setpts=PTS-STARTPTS, scale=960x540 [upperleft];
        [1:v] trim=start_pts=50483,setpts=PTS-STARTPTS, scale=960x540 [upperright];
        [2:v] trim=start_pts=795117,setpts=PTS-STARTPTS, scale=960x540 [lowerleft];
        [3:v] trim=start_pts=38100,setpts=PTS-STARTPTS, scale=960x540 [lowerright];
        [base][upperleft] overlay=shortest=1 [tmp1];
        [tmp1][upperright] overlay=shortest=1:x=960 [tmp2];
        [tmp2][lowerleft] overlay=shortest=1:y=540 [tmp3];
        [tmp3][lowerright] overlay=shortest=1:x=960:y=540
    "
    -c:v libx264 output.mkv

Answer 1

By "how do I make it start 15 seconds before the specified frame numbers" you simply want all those four videos to start 15 seconds back, compared to your current timings? Any reason why you would use start_pts instead of start or start_frame ? Those two seem more convenient. If i got it right, if the offset for cutscene image in the third video starts at t:52.416471 , and you want this video track start ten seconds before, just use trim=start=42.416471 in your filters. If all steams have the same pixel format, you can remove nullsrc, and replace the overlay parts with [upperleft][upperright][lowerleft][lowerright]xstack=inputs=4:layout=0_0|w0_0|0_h0|w0_h0[out_v] . End your filter chain with named output (out_v in example above), then map it like -map '[out_v]' . If you want to use second audio stream, you can map it with -map 1:a, but something tells me you would want to manipulate its timing, too, and add another line to your filtering: [1:a]atrim=<your options to match the video>[out_a] , then -map '[out_a]' .