Dealing with Recursion in MySQL/PHP

Question

I've developed a subscription based application built using Laravel5/MySQL. And I'm dealing with fetching invoices from mysql. On every invoice we've to display a few columns ie "Previous Balance", "Payments", "This Month's Charge", "Payable By Due Date", "Payable After Due Date". Now the "Previous Balance" is "Payable by Due Date" from last invoice, which in-turn is calculated as "Previous Balance" - "Payments" + "This Month's Charge". So here starts the recursion and will continue until all records in the db are fetched. This too works fine for one single invoice. But when I try to fetch multiple (100~200) server will run out of memory. With a table of ~13k invoices.

As a fix to this problem, I added a new column to the table "payable_by_due_date" at the time of invoice generation update this column. After the change and updating all the records, I tried with ~1000 invoices and it took 3~4 seconds to fetch all the records, make PDF for all the invoices and pack them all in a zip file and handover that zip file to the user for download.

But there is a big problem with this approach, what if in future any of the invoices requires rectification, all of the invoices that were generated after that particular invoice will require manual update.

So, if anyone of you could help me get out of this problem, I'll be thankful.

Answer 1

There is no distinct answer to your question - its more about approach more than anything else, and is very app specific.

A few points:

Regarding memory - yes, this will most likely blow up your server, the bigger your data set gets, the worse it will get - if you are talking about recursively walking through large data sets.

On this, if possible, use laravel queues to off load heavy data processing into the background. For example, trigger the event when the user requests it, then display a notification on the front end to the user saying "we are currently processing your request, we will email you with a link once complete" - kind of like how Google does their "takeout" ( https://www.google.com/settings/takeout ) works.

Regarding recalculating - pretty much the same as point one. If its push into a background process, large data processing will hurt less.

Again, it's pretty app specific. If possible, break down the question into smaller answerable questions with very specific requirements/outcomes.