关于多租户设置的分区策略问题

Question

I've been watching some videos and reading articles about azure cosmos db, and I think it will be a good fit for my app. I am currently using SQL Server, but I think I will be reaching limits of SQL Server, and I am already seeing issues where queries from several large customers are causing throttling of queries by other smaller customers. I'm setting up a new database for my app, and I want to make sure I am setting it up correctly.

I want to find a good partition key strategy and container strategy, while maintaining a good separation between all customers. My knowledge in this area is limited, as I come from a SQL background (for the past 15 years or so). Here are some specifics for the app and the data it stores:

1. My app saves phone call records and text/sms records for my customers.
2. Some customers are big, and the app saves around 10,000 call records per day for these customers.
3. Some customers are small, and the app saves around 10 call records per day for these customers (and there are ranges in between for other customers).
4. As of now, after running the app for around 5 years, there are around 42,000,000 call and text records total in the SQL Server database across all customers, and it grows daily. The current storage size for the entire database is 80GB.
5. On average, the app adds around 65,000 new records per day, across all customers.
6. The app is much more write-heavy than read-heavy, with most queries using date range and free text searching of the phone numbers.
7. The app has great potential to grow, and we even have a possible single customer coming on that will double this volume (this customer alone will have around 65,000 calls per day, bringing the total to 130,000 call records per day).
8. I need to maintain text searching capabilities that are currently in the app (free text searching of these fields: phone number, names, text message content)

Is it a good idea to use the account id for the partition key? Or can I just use the account id for the container, and something else as the partition key? What is a good strategy in this situation?

If it helps, this is an example of what the call records look like:

{
  "accountId": "9153849867",
  "id": "I8uToEX1hjmwzUA",
  "type": "Voice",
  "direction": "Inbound",
  "action": "Phone Call",
  "result": "Accepted",
  "callTime": "2020-05-26T16:58:14.675Z",
  "duration": 235,
  "callers": [
  {
    "phoneNumber": "7537547442",
    "extensionNumber": null,
    "location": "Edina, MN",
    "name": "WIRELESS CALLER",
    "toInd": false
    "legInd": false
    "extensionInd": false
  },{
    "phoneNumber": "2564572486",
    "extensionNumber": null,
    "location": null,
    "name": null,
    "toInd": true
    "legInd": false
    "extensionInd": false
      }
  ],
  "files": {
    /*More data here that does not need to be searched*/
  }
}

Answer 1

It is impossible to answer this with much precision because ultimately you will need to measure with load tests on both the write and query paths. I would think about the following.

Is this a write heavy or read heavy application? If write heavy then optimize your partition strategy around distributing writes with a partition key with high cardinality. If read heavy then optimize around single-partition reads/queries, or bounded partition reads/queries. If it's both then look at using Change Feed with write optimized for one container and read optimized to serve queries. However you'll need to consider the cost of two containers and using change feed to copy data as part of whether this more efficient than serving queries from a single container.

Next consideration is storage. For whatever partition key you choose (let's say account id) how fast do you get to the maximum partition size of 20 GB? Do you TTL data at some point? Is it before you reach the 20 GB maximum.

Given the asymmetric load between accounts, account id may not be a good partition key as it will result in uneven distribution of writes. Since throughput is distributed evenly, this is inefficient and likely won't perform well under heavy load. It likely could result in larger accounts hitting the 20GB limit quickly. But for queries it's likely a good candidate as I'm sure customers query for their own data. You could consider using account id with a time element as a synthetic key. But for queries you'd need to time bound these so you aren't doing a fan out across all partitions.

Not sure any of this is helpful. Partition strategy is difficult. You have to deeply understand your access patterns to know what are the high volume writes and reads. Measure those and the right key should emerge.