Rails GET request pulling down previously downloaded data

Question

I'm working with an external API to receive pending orders. I've successfully written a GET request method that saves any pending orders to PostgreSQL.

The issue I'm having is that during each GET request, I'm creating duplicates because it's downloaded all the pending orders (old and new) again.

Is there a way during my GET request, I can check against the database for previously downloaded pending orders, ignore them, and download only newly created pending orders?

REST Gem I'm using:

rest-client

Here's my working GET request:

def download_pf_orders
  download_count = 0
  uri = "#{PF_TEST_PENDING_API_URL}"
  rest_resource = RestClient::Resource.new(uri, PF_TEST_USERNAME, PF_TEST_PASSWORD)

  begin
    response = rest_resource.get(accept: 'application/json')
    json = JSON.parse(response)

    json.each do |data|

      sequence = data['SequenceNumber']
      puts "### Last Sequence Number: #{sequence}"

      PfOrder.create(
        sequence_number: data['SequenceNumber'],
        message_guid: data['MessageGuid'],
        hl7_document: data['Hl7Document']
        )
      download_count += 1
    end
  rescue => e
    puts "### Status Code: #{e.response.code} ###"
  end
  puts "### Downloaded Orders: #{download_count} ###"
end

Answer 1

Do you have control of the remote server too? can you update it to only send you things you haven't fetched before? if not, then how about asking for only posts that have been created after date X (which is the date of the last time you fetched)? if not, then fetch them all, and just check your db and only create a new record if you haven't already (sequence numbers are unique right? if the sequence number is already in your db., then skip that row).

Answer 2

I figured this out. Here's my updated GET method:

def download_pf_orders
  download_count = 0
  uri = "#{PF_TEST_PENDING_API_URL}"
  rest_resource = RestClient::Resource.new(uri, PF_TEST_USERNAME, PF_TEST_PASSWORD)

  begin
    response = rest_resource.get(accept: 'application/json')
    json = JSON.parse(response)

    json.each do |data|

      sequence = data['SequenceNumber']
      puts "### Last Sequence Number: #{sequence}"

      pending_order = PfOrder.new(
        sequence_number: data['SequenceNumber'],
        message_guid: data['MessageGuid'],
        hl7_document: data['Hl7Document'],
        )

      sn = pending_order.sequence_number
      mg = pending_order.message_guid
      hd = pending_order.hl7_document

      PfOrder.find_or_create_by(sequence_number: sn, message_guid: mg, hl7_document: hd)

      download_count += 1
    end
  rescue => e
    puts "### Status Code: #{e.response.code} ###"
  end
  puts "### Downloaded Orders: #{download_count} ###"
end