REST and complex search queries

Question

I'm looking for a robust way to model search queries in a REST api.

In my api, you can specify the search criteria in the URI of a resource using query parameters.

For example:

/cars?search=color,blue;AND;doors,4 --> Returns a list of blue cars with 4 doors

/cars?search=color,blue;OR;doors,4 --> Returns a list of cars that are blue or have 4 doors

On the server side, the search string is mapped to the desired underlying technology. Depending on the rest resource, this can be a SQL query, the Hibernate Criteria api, another webservice call, ...

The 2 examples are simple enough to support, but I also need more complex search features like substring search, searching before/after dates, NOT, ...

This is a common problem I think. Is there a library (or a pattern) that I can use that:

• Maps search queries specified as a string to a generic Criteria model. The search format does not have to be the same as I listed above.
• Allows me to map that Criteria model to any technology I need to use.
• Offers mapping support for Hibernate/JPA/SQL, but this is a bonus ;)

Kind regards,

Glenn

Answer 1

Whenever I face these problems, I ask myself "How would I present this to a user, if I was creating a traditional webpage"? The simple answer is that I wouldn't present those sort of options in a single page. The interface would be too complex; however what I could do is provide an interface that allowed users to build up more and more complex queries over a number of pages and that's the solution I think you should go for in this case.

The HATEOAS constraint specifies that we must include the hypermedia controls (links and forms) in our responses. So let's say we have a paginated collections of cars at /cars with a search option, so that when you get /cars it returns something like (BTW I'm using a custom media-type here, but the forms and links should be pretty obvious. Let me know if it isn't):

<cars href="/cars">
    <car href="/cars/alpha">...</car>
    <car href="/cars/beta">...</car>
    <car href="/cars/gamma">...</car>
    <car href="/cars/delta">...</car>
    ...
    <next href="/cars?page=2"/>
    <search-color href="/cars" method="GET">
        <color type="string" cardinality="required"/>
        <color-match type="enumeration" cardinality="optional" default="substring">
            <option name="exact"/>
            <option name="substring"/>
            <option name="regexp"/>
        </color-match>
        <color-logic type="enumeration" cardinality="optional" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </color-logic>
    </search>
    <search-doors href="/cars" method="GET">
        <doors type="integer" cardinality="required"/>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

So just say we search for white cars, we would GET /cars?color=white and we might get back something like:

<cars href="/cars?color=white">
    <car href="/cars/beta">...</car>
    <car href="/cars/delta">...</car>
    ...
    <next href="/cars?color=white&page=2"/>
    <search-color href="/cars?color=white" method="GET">
        <color2 type="string" cardinality="required"/>
        <color2-match type="enumeration" cardinality="optional" default="substring">
            <option name="exact"/>
            <option name="substring"/>
            <option name="regexp"/>
        </color2-match>
        <color2-logic type="enumeration" cardinality="optional" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </color2-logic>
    </search>
    <search-doors href="/cars?color=white" method="GET">
        <doors type="integer" cardinality="required"/>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

This result then let's us refine our query. So just say we wanted white cars but not "off-white" cars, we could then GET '/cars?color=white&color2=off-white&color2-logic=not', which might return

<cars href="/cars?color=white&color2=off-white&color2-logic=not">
    <car href="/cars/beta">...</car>
    <car href="/cars/delta">...</car>
    ...
    <next href="/cars?color=white&color2=off-white&color2-logic=not&page=2"/>
    <search-color href="/cars?color=white&color2=off-white&color2-logic=not" method="GET">
        <color3 type="string" cardinality="required"/>
        <color3-match type="enumeration" cardinality="optional" default="substring">
            <option name="exact"/>
            <option name="substring"/>
            <option name="regexp"/>
        </color3-match>
        <color3-logic type="enumeration" cardinality="optional" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </color3-logic>
    </search>
    <search-doors href="/cars?color=white&color2=off-white&color2-logic=not" method="GET">
        <doors type="integer" cardinality="required"/>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

We could then further refine our query, but the point is that at each step along the way, the hypermedia controls tells us what is possible.

Now, if we think about the search options for cars, the colors, doors, makes and models aren't unbounded, so we could make the options more explicit by providing enumerations. For instance

<cars href="/cars">
    ...
    <search-doors href="/cars" method="GET">
        <doors type="enumeration" cardinality="required">
            <option name="2"/>
            <option name="3"/>
            <option name="4"/>
            <option name="5"/>
        </doors>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

However the only white cars that we have might be 2 and 4 door, in which case GETing /cars?color=white might give us

<cars href="/cars?color=white">
    ...
    <search-doors href="/cars?color=white" method="GET">
        <doors type="enumeration" cardinality="required">
            <option name="2"/>
            <option name="4"/>
        </doors>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

Similarly, as we refine the colors, we might find their are only a few options, in which case we can switch from providing a string search, to providing an enumeration search. eg, GETing /cars?color=white might give us

<cars href="/cars?color=white">
    ...
    <search-color href="/cars?color=white" method="GET">
        <color2 type="enumeration" cardinality="required">
            <option name="white"/>
            <option name="off-white"/>
            <option name="blue with white racing stripes"/>
        </color2>
        <color2-logic type="enumeration" cardinality="optional" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </color2-logic>
    </search>
    ...
</cars>

You could do the same for other search categories. For instance, initially you wouldn't want to enumerate all the makes, so you would provide some sort of text search. Once the collection has been refined, and there are only a few models to pick from, then it makes sense to provide an enumeration. The same logic applies to other collections. For instance you wouldn't want to enumerate all of the cities in the world, but once you have refined the area to 10 or so cities, then enumerating them can be very helpful.

Is there a library that will do this for you? None that I know of. Most I've seen don't even support hypermedia controls (ie, you've got to add the links and forms yourself ). Is there a pattern you can use? Yes, I believe the above is a valid pattern for solving this sort of problem.

Answer 2

hmmm... this is a tricky area. There is no framework out there that is tailored for your problem. Even ODATA that @p0wl mentions has certain limitations about the way fields are mapped to domain models. It gets weird after a point.

The simplest way to get around this is to accept the query as a String which you then validate against a domain specific language using a AST or whatever it is that you want. This allows you to be flexible while accepting the query while still validating it for correctness. What you lose is the ability to describe the query in a more meaningful manner through the URL.

Take a look at chapter 8 of the Restful recipes cookbook. It highlights one of the solutions I proposed and also recommends other approaches. Choose one based on the flexibility needed by your client versus the expressiveness of your query. There is a balance you can strike somewhere.

Answer 3

@GlennV

I'm not sure there is a pattern/rule directly related to this and if this is a common problem(i mean logical operators in query string), but i'd ask following question to myself when designing something like this:

• How client is going to construct such queries?

The problem is that URIs(as a whole, including query string part) must be entirely controlled by service provider and clients must not be directly coupled to URI construction logic which are specific to your problem domain.

There are several ways how clients use URIs and how they construct them:

• Client's may follow links which are included either in HTTP headers via "Link: <...>" header or in entity body(like atom or HTML links or something similar)
• Client's may construct URIs by rules defined by media type or other spec.

Several well known approaches of guided URI construction process:

• HTML GET forms where form fields are encoded according to RFC3986 and appended to the URI provided in form's action attribute value(media type guided);
• URI Templates(quite broad though worths reading, spec offers really interesting features) RFC6570
• Open search queries(media type guided)

Based on information above you can choose existing approach or design your own but with simple rule in mind that service must coordinate clients how they can construct queries(ie define your own spec, custom media type, or extend existing).

Another thing is how you are going to map URIs to underlying implementation and there are two very important things in terms of REST:

• Connector - which includes HTTP + URI specs and actually only connector matters for clients.
• Component - which is underlying implementation and no one cares except you. This part needs to be entirely hidden from clients and you can choose any binding approach here, no one can tell you exact way, it entirely depends on particular requirement(s).

Answer 4

Do you search for a library like ODATA ( Link )?

This would solve your query parsing problems and you could just forward the query to the database of your choice.

Answer 5

As mentioned before, still there isnt a generic solution to do this. Probably, the best option (at tleast the best ive found) is Spring DATA REST ( http://static.springsource.org/spring-data/rest/docs/1.1.0.M1/reference/htmlsingle/ ). Using the CrudRepository you can get methods like findOne, findAll, etc. AND, if you extend it, you can add your own generic mappings. In example, what we have done is to extend it to have generic query options like: customer?country=notArgentina, in order to get customers that does not belong to Argentina. Or, for example: customer?country=like( Island ) to get all customers with country like "Island" (in a SQL way). Hope it helps.