Generate pdf in chunks(ex. 10 number of records at a time and then update other records to the same file). I am using wicked_pdf with delayed job

Question

I want to generate pdf in chunks of data, which can generate some(given) amount of rows at a time and then update the pdf in same way until all records are generated. My goal is to not take too much memory at a time and result in a crash. Because when we generate millions of data at time it consumes all the memory.

Have a look on my code below.

EnforcementReports::CreatePaymentPdfReportJob = Struct.new(:report_id, :page, :report_file_path) do
    PAGE_SIZE = 50000
    def queue_name
        "create_payment_pdf_report_job_queue"
    end

    def enqueue(job)
        job.save!
    end

    def perform
        return unless report.present?
        return if report.status == "generated"
        pdf_export
    end

    def max_attempts
        1
    end

    private

    def helpers
        @helpers ||= ApplicationController.helpers
    end

    def locations
        @locations ||= helpers.get_locations
    end

    def av
        @av ||= ActionView::Base.new()
    end

    def report_file
        if report_file_path.present?
            File.open(report_file_path)
        else
            Tempfile.new ['payment', '.pdf'], "#{Rails.root}/tmp"
        end
    end

    def pdf_export
        av.view_paths = ActionController::Base.view_paths
        file1 = report_file
        pdf_html = av.render :template => "admin/reports/payments/index.pdf.erb", :layout => nil, locals: { violations: payment_violations, locations: locations, users: payment_users, report: report }
        pdf_html = WickedPdf.new.pdf_from_string(pdf_html)
        File.write(file1.path, mode: 'ab') do |file1|
            file1 << pdf_html
        end
        report.update(report_update_params.merge(file: file1))
        #File.delete(file1) if File.exists? file1
        Delayed::Job.enqueue(EnforcementReports::CreatePaymentPdfReportJob.new(report.id, page + 1, report.file.path))
    end

    def payment_violations
        @payment_violations ||= Violations::Filter.call(violation_issued: additional_data["violation_issued"].to_i, user_locations: user_locations, time: additional_data["time"], search_by: additional_data["search_by"] , search: additional_data["search"], user_id: additional_data["user_id"], order: additional_data["order"], column: additional_data["column"] , page_size: ENV["REPORT_PAGE_SIZE"].to_i, page: 1)["violations"]
    end

    def user_locations
        @user_locations ||= helpers.location_ids(additional_data)
    end

    def report
        @report ||= Report.find_by(id: report_id)
    end

    def report_update_params
        {
            file_size: file_size,
            status: status
        }
    end

    def status
        return "generated" if file_size >= report.rows_count.to_i
        "generating"
    end

    def file_size
        report.file_size.to_i + ENV["REPORT_PAGE_SIZE"].to_i
    end

    def additional_data
        report.additional_data.merge(userid: report.user_id)
    end

    def payment_users
        @payment_users ||= User.all_user
    end
end

Answer 1

I am not sure if you can edit a pdf once created but you can certainly use find_in_batches method to process chunks of data. Read more about the method

It takes batch_size as an argument to limit the number of records.