Use the PointOnPointOverlayer (POPO) with 600,000+ records without having "creating spatial index" writing to the logfile for 7+ hours

Question

When using PointOnPointOverlayer (POPO), after FME 2018, I experience performance issues. With FME 2018.1.2.0 (20190225 - Build 18590 - WIN64) I could use the POPO with 600,000 records and this portion of the job use to run in 30+ minutes.

Now with FME 2020.2.1.0 (20201130 - Build 20806 - WIN64) it takes 7+ hours.

I also tested briefly with FME 2021.1.3 (Build 21631 - win64) but saw no significant improvement.

What options are available for removing this additional 6.5 hours of processing time?

Highlights from the logfile:
2021-10-21 14:11:59|  0.4| 0.4|INFORM|FME 2020.2.1.0 (20201130 - Build 20806 - WIN64)
2021-10-21 14:11:59|  0.4| 0.0|INFORM|FME_HOME is 'C:\Program Files\FME2020\'
2021-10-21 14:11:59|  0.4| 0.0|INFORM|FME Desktop ESRI Edition (floating)
.
.
.
2021-10-21 14:12:50| 50.4| 0.0|INFORM|Creating spatial index of 2 point features
2021-10-21 14:12:50| 50.4| 0.0|INFORM|Completed spatial index creation
2021-10-21 14:12:50| 50.4| 0.0|INFORM|Creating spatial index of 1 point features
2021-10-21 14:12:50| 50.4| 0.0|INFORM|Completed spatial index creation
.
.
.
2021-10-21 21:25:57|4641.4| 0.0|INFORM|Creating spatial index of 1 point features
2021-10-21 21:25:57|4641.4| 0.0|INFORM|Completed spatial index creation
2021-10-21 21:25:57|4641.4| 0.0|INFORM|Creating spatial index of 1 point features
2021-10-21 21:25:57|4641.4| 0.0|INFORM|Completed spatial index creation
2021-10-21 21:25:57|4641.4| 0.0|INFORM|PointOnPointOverlayer_10 (OverlayFactory): Group 626779 / 626779: Finished processing group 626779 / 626779
2021-10-21 21:25:57|4641.4| 0.0|INFORM|PointOnPointOverlayer_10 (OverlayFactory): Group 626779 / 626779: 100% complete

Took 7+ hours to complete PoPo on 626,000+ records

Answer 1

Are you using the "process by group" here in the POPO? From the log it looks like each point is being treated as its own group?

If you disable everything after this part of the process is it still really slow? In some cases performance issues can be further on but appear to be at a different spot.

Another thing to note, you can see the number of seconds in the log there "4641.4". This suggests that FME process time (the time fme has been actively using the CPU) was only 77mins. The rest of the time is likely something else - if a database or network is involved these are common culprits.

To see where the time is going you need to take a look at the timestamp. Look for big jumps in the timestamps with minimal cpu time.

The end of the log file should provide a summary of job time vs cpu time. If the cpu and total time are different it means that fme is waiting at some point during the process.