我应该如何在Cassandra中存储日期间隔？

Question

I'm working on an application that stores sensor measurements. Sometimes, the sensors will send erroneous measurements (eg the measured value is out of bound). We do not want to persist each measurement error separately, but we want to persist statistics about these errors, such as the sensor id, the date of the first error, the date of the last error, and other infos like the number of successive errors, which I'll omit here...

Here is a simplified version of the "ErrorStatistic" class:

package foo.bar.repository;

import org.joda.time.DateTime;

import javax.annotation.Nonnull;
import javax.annotation.Nullable;

import static com.google.common.base.Preconditions.checkNotNull;

public class ErrorStatistic {

    @Nonnull
    private final String sensorId;
    @Nonnull
    private final DateTime startDate;
    @Nullable
    private DateTime endDate;

    public ErrorStatistic(@Nonnull String sensorId, @Nonnull DateTime startDate) {
        this.sensorId = checkNotNull(sensorId);
        this.startDate = checkNotNull(startDate);
        this.endDate = null;
    }

    @Nonnull
    public String getSensorId() {
        return sensorId;
    }

    @Nonnull
    public DateTime getStartDate() {
        return startDate;
    }

    @Nullable
    public DateTime getEndDate() {
        return endDate;
    }

    public void setEndDate(@Nonnull DateTime endDate) {
        this.endDate = checkNotNull(endDate);
    }

}

I am currently persisting these ErrorStatistic using Hector as follows:

private void persistErrorStatistic(ErrorStatistic errorStatistic) {
    Mutator<String> mutator = HFactory.createMutator(keyspace, StringSerializer.get());

    String rowKey = errorStatistic.getSensorId();
    String columnName = errorStatistic.getStartDate().toString(YYYY_MM_DD_FORMATTER);
    byte[] value = serialize(errorStatistic);

    HColumn<String, byte[]> column = HFactory.createColumn(columnName, value, StringSerializer.get(), BytesArraySerializer.get());
    mutator.addInsertion(rowKey, COLUMN_FAMILY, column);

    mutator.execute();
}

private static final DateTimeFormatter YYYY_MM_DD_FORMATTER = DateTimeFormat.forPattern("yyyy-MM-dd");

When we receive the first measurement in error, we create an ErrorStatistic with sensorId and startDate set, and a null endDate . This ErrorStatistic is kept in our in-memory model, and persisted in Cassandra. We then update the ErrorStatistic in memory for the next measurements in error, until we receive a valid measurement, at which point the ErrorStatistic is persisted and removed from our in-memory model.

Cassandra thus contains ErrorStatistics with open-ended intervals (eg [2012-08-01T00:00Z|null]), and closed intervals (eg [2012-08-01T00:00Z|2013-01-12T10:23Z]).

I want to be able to query these ErrorStatistics by date.

For example, if I have these 3 error statistics:

sensorId  = foo
startDate = 2012-08-01T00:00Z
endDate   = 2012-09-03T02:10Z

sensorId  = foo
startDate = 2012-10-04T03:12Z
endDate   = 2013-02-01T12:28Z

sensorId  = foo
startDate = 2013-03-05T23:22Z
endDate   = null
(this means we have not received a valid measurement since 2013-03-05)

If I query Cassandra with the date:

• 2012-08-04T10:00Z --> it should return the first ErrorStatistic
• 2012-09-04T00:00Z --> it should return that there were no errors at this time
• 2014-01-03T00:00Z --> it should return the last ErrorStatistic (since it is open-ended)

I am not sure how I should store and "index" these ErrorStatistic objects, to efficiently query them. I am quite new to Cassandra, and I might be missing something obvious.

---

Edit: the following was added in response to Joost's suggestion that I should focus on the type of queries I am interested in.

I will have two types of query:

• The first, as you guessed, is to list all ErrorStatistics for a given sensor and time range. This seems relatively easy. The only problem I will have, is when an ErrorStatistics starts before the time range I'm interested in (eg I query all errors for the months of april, and I want my query to return ErrorStatistics[2012-03-29:2012-04-02] too...)
• The second query seems harder. I want to find, for a given sensor and date, the ErrorStatistics whose interval contains the given date, or whose startDate precedes the given date, with a null endDate (this means that we are still receiving errors for this sensor). I don't know how to do this efficiently. I could just load up all ErrorStatistics for the given sensor, then check the intervals in Java... But I'd like to avoid this if possible. I guess I want Cassandra to start at a given date and look backward until it finds the first ErrorStatistics with a startDate that precedes the given date (if any), then load it and check in Java if its endDate is null or after the given date. But I have no idea if that's possible, and how efficient that would be.

Answer 1

The question you have to ask yourself is what questions you have towards the ErrorStatistics. Cassandra schema design typically starts with a 'Table per query' approach. Don't start with the data (entities) you have, but with your questions/queries. This is a different mindset than 'traditional' rdbms design, and I found it takes some time to get used to.

For example, do you want to query the statistics per Sensor? Than a table with a composite key (sensor id, timeuuid) could be a solution. Such a table allows for quick lookup per sensor id, sorting the results based on time.

If you want to query the sensor statistics based on time only, a (composite) key with a time unit may be of more help, possibly with sharding elements to better distribute the load over nodes. Note that there is catch: range queries on primary keys are not feasible using the Cassandra random or murmur partitioners. There are other partitioners, but they easily tend to uneven load distribution in your cluster.

In short, start with the answers you want, and then work 'backwards' to your table design. With a proper schema, your code will follow.

---

Addition (2013-9-5): What is good to know is that Cassandra sorts data within the scope of a single partition key. That is something very useful. For example the measurements would be ordered by start_time in descending order (newest first) if you define a table as:

create table SensorByDate
(
    sensor_id uuid,
    start_date datetime,
    end_date datetime,
    measurement int
    primary key (sensor_id, start_date)
)
with clustering order by (start_time DESC);

In this example the sensor_id is the partition key and determines the node this row is stored on. The start_date is the second item in the composite key and determines the sort order.

To get the first measurement after a certain start date in this table you could formulate a query like

select * from SensorByDate 
where sensor_id = ? and start_date < ? limit 1