存储和搜索复杂数字数据的方式有哪些？

Question

I have some numerical data that must be searchable from a web front-end with the following format:

Toy type: Dog
Toy subtype: Spotted
Toy maker: John
Color: White
Estimated spots: 10
Actual spots: 11

Toy type: Cat
Toy subtype: Striped
Toy maker: Jane
Color: White
Estimated stripes: 5
Actual stripes: [Not yet counted]

A search query might be something like "Type:Cat, Stripes:4-6", or "Type:Dog, Subtype:Spotted", or "Color:White", or "Color:White, Maker:John".

I'm not sure if the data is best suited for a relational database because there are several types and subtypes, each with their own properties. On top of that, there are estimated and actual values for each property.

I'd like some recommendations for how to store and search this data. Please help!

EDIT: I changed the search queries so they are no longer free-form.

Answer 1

You have structured the problem in such a way as to make this very difficult to solve. Your data is structured data, with specific columns. Yet, you are trying to use free-form queries to search through it.

So, the normal way to do this is to allow search terms for each of the fields.

The next way to approach this is as a full-text problem. This definitely has its issues. For instance, numbers are typically stop words. And values in different fields would get confused with each other.

Of course, you can try to do free form search on structured data. This is, after all, something that Google and Microsoft are doing. If you search "airfare from New York to London" on Google, you will get lists of flights. But this is a hard problem to approach through understanding the query.

Answer 2

I recommend using Apache Solr to index and search your data.

How you use Solr depends on your requirements. I use it as a searchable cache of my data. Extremely useful when the raw master data must be keep as files. Lots of frameworks integrate Solr as their search backend.

For building front-ends to a Solr index, checkout solr-ajax .

Example

Install Solr

Download Solr distribution:

wget http://www.apache.org/dist/lucene/solr/4.7.0/solr-4.7.0.tgz
tar zxvf solr-4.7.0.tgz

Start Solr using embedded Jetty container:

cd solr-4.7.0/example
java -jar start.jar

Solr should now be running locally

http://localhost:8983/solr

data.xml

You did not specify a data format so I used the native XML supported by Solr:

<add>
  <doc>
    <field name="id">1</field>
    <field name="toy_type_s">Dog</field>
    <field name="toy_subtype_s">Spotted</field>
    <field name="toy_maker_s">John</field>
    <field name="color_s">White</field>
    <field name="estimated_spots_i">10</field>
    <field name="actual_spots_i">11</field>
  </doc>
  <doc>
    <field name="id">2</field>
    <field name="toy_type_s">Cat</field>
    <field name="toy_subtype_s">Striped</field>
    <field name="toy_maker_s">Jane</field>
    <field name="color_s">White</field>
    <field name="estimated_spots_i">5</field>
  </doc>
</add>

Notes:

• Every document in Solr must have a unique id
• The field names have a trailing "_s" and "_i" in their names to indicate field types. This is a cheat to take advantage of Solr's dynamic field feature.

Index XML file

Lots of ways to get data into Solr. The simplest way is the curl command:

curl http://localhost:8983/solr/update?commit=true -H "Content-Type: text/xml" --data-binary @data.xml

It's worth noting that Solr supports other data formats, such as JSON and CSV.

Search indexed file

Again there are language libraries to support Solr searches, the following examples use curl. The Solr search syntax is along the lines you've required.

Here's a simple example:

$ curl http://localhost:8983/solr/select/?q=toy_type_s:Cat
<response>
  <lst name="responseHeader">
    <int name="status">0</int>
    <int name="QTime">1</int>
    <lst name="params">
      <str name="q">toy_type_s:Cat</str>
    </lst>
  </lst>
  <result name="response" numFound="1" start="0">
    <doc>
      <str name="id">2</str>
      <str name="toy_type_s">Cat</str>
      <str name="toy_subtype_s">Striped</str>
      <str name="toy_maker_s">Jane</str>
      <str name="color_s">White</str>
      <int name="estimated_spots_i">5</int>
      <long name="_version_">1463999035283079168</long>
    </doc>
  </result>
</response>

A more complex search example:

$ curl "http://localhost:8983/solr/select/?q=toy_type_s:Cat%20AND%20estimated_spots_i:\[2%20TO%206\]" 
<response>
  <lst name="responseHeader">
    <int name="status">0</int>
    <int name="QTime">2</int>
    <lst name="params">
      <str name="q">toy_type_s:Cat AND estimated_spots_i:[2 TO 6]</str>
    </lst>
  </lst>
  <result name="response" numFound="1" start="0">
    <doc>
      <str name="id">2</str>
      <str name="toy_type_s">Cat</str>
      <str name="toy_subtype_s">Striped</str>
      <str name="toy_maker_s">Jane</str>
      <str name="color_s">White</str>
      <int name="estimated_spots_i">5</int>
      <long name="_version_">1463999035283079168</long>
    </doc>
  </result>
</response>