Firebase：对大型数据集的查询

Question

I'm using Firebase to store user profiles. I tried to put the minimum amount of data in each user profile (following the good practices advised in the documentation about structuring data) but as I have more than 220K user profiles, it still represents 150MB when downloading as JSON all user profiles. And of course, it will grow bigger and bigger as I intend to have a lot more users :)

I can't do queries on those user profiles anymore because each time I do that, I reach 100% Database I/O capacity and thus some other requests, performed by users currently using the app, end up with errors.

I understand that when using queries, Firebase need to consider all data in the list and thus read it all from disk. And 150MB of data seems to be too much.

So is there an actual limit before reaching 100% Database I/O capacity? And what is exactly the usefulness of Firebase queries in that case? If I simply have small amounts of data, I don't really need queries, I could easily download all data. But now that I have a lot of data, I can't use queries anymore, when I need them the most...

Answer 1

The core problem here isn't the query or the size of the data, it's simply the time required to warm the data into memory (ie load it from disk) when it's not being frequently queried. It's likely to be only a development issue, as in production this query would likely be a more frequently used asset.

But if the goal is to improve performance on initial load, the only reasonable answer here is to query on less data. 150MB is significant. Try copying a 150MB file between computers over a wireless network and you'll have some idea what it's like to send it over the internet, or to load it into memory from a file server.

A lot here depends on the use case, which you haven't included.

Assuming you have fairly standard search criteria (eg you search on email addresses), you can use indices to store email addresses separately to reduce the data set for your query.

/search_by_email/$user_id/<email address>

Now, rather than 50k per record, you have only the bytes to store the email address per records--a much smaller payload to warm into memory.

Assuming you're looking for robust search capabilities, the best answer is to use a real search engine. For example, enable private backups and export to BigQuery, or go with ElasticSearch (see Flashlight for an example).