加快搜索大型 json 数组（用于 jQuery/Rails 自动完成文本框）

Question

I have a location text field hooked up to the jquery plugin EasyAutocomplete :

It's working, but the problem is the dataset is large - 100,000 records or so. It takes an unacceptably long time to match any results.

I have a json endpoint in my app that loads the records for the autocomplete. The code in the Rails controller looks like this:

def index
  @locations = Location.select(:id,:canonical_name)

  respond_to do |format|
    format.json { render json: @locations }
  end
end

So far what I've tried is putting an index on the canonical_name column, and putting a Rails.cache.fetch around the loading of the records, but neither of these things helped much.

What can be done to speed up this operation?

Answer 1

The problem here is not searching through a large JSON array. Its that you're not doing the search in the database in the first place.

You're first selecting 100,000 rows from your database and using it to initialize 100,000 model objects. That not only will take a lot of time but eat through a gigantic chunk of RAM.

You then go through and serialize that collection of 100,000 rows into a mega long JSON string which eats even more RAM.

This is then sent across the internet to the poor client that gets a monster JSON response and runs out of data on their mobile plan. Even if you implemented some kind of magic stack sort that's phenomenally fast the costs if just getting the data here are staggering.

Caching might slightly speed up the server side but its going to do nothing for the initial cold cache hit or the amount of data sent across the wire and the amount of RAM this will use on the client.

Implementing a server side search like that provided by geonames or google is actually quite an undertaking and you might want to investigate the existing options.

A really naive implementation would be:

@locations = Location.where(
  "canonical_name like ?", "%#{params[:query]}%"
).select(:id, :canonical_name)