Are Google Cloud Disks OK to use with SQLite?

Question

. 。 Answers to this question are eligible for a +500 reputation bounty. zino is looking for an : ：

Must have specific knowledge of how Google Disks manage locking and fsync over the network, and an authoritative reference for why this is safe with SQLite's particular locking and transactions.

Google Cloud disks are network disks that behave like local disks. SQLite expects a local disk so that locking and transactions work correctly.

A. Is it safe to use Google Cloud disks for SQLite?

B. Do they support the right locking mechanisms? How is this done over the network?

C. How does disk IOP's and Throughput relate to SQLite performance? If I have a 1GB SQLite file with queries that take 40ms to complete locally, how many IOP's would this use? Which disk performance should I choose between (standard, balanced, SSD)?

Thanks.

Related

https://cloud.google.com/compute/docs/disks#pdspecs

Persistent disks are durable network storage devices that your instances can access like physical disks

https://www.sqlite.org/draft/useovernet.html

the SQLite library is not tested in across-a- network scenarios, nor is that reasonably possible. Hence, use of a remote database is done at the user's risk.

Answer 1

Yeah, the article you referenced, essentially stipulates that since the reads and writes are "simplified", at the OS level, they can be unpredictable resulting in "loss in translation" issues when going local-network-remote.

They also point out, it may very well work totally fine in testing and perhaps in production for a time, but there are known side effects which are hard to detect and mitigate against -- so its a slight gamble.

Again the implementation they are describing is not Google Cloud Disk, but rather simply stated as a remote networked arrangement.

My point is more that Google Cloud Disk may be more "virtual" rather than purely networked attached storage... to my mind that would be where to look, and evaluate it from there.

---

Checkout this thread for some additional insight into the issues, https://serverfault.com/questions/823532/sqlite-on-google-cloud-persistent-disk

Additionally, I was looking around and I found this thread, where one poster suggest using SQLite as a read-only asset, then deploying updates in a far more controlled process. https://news.ycombinator.com/item?id=26441125

Answer 2

the persistend disk acts like a normal disk in your vm. and is only accessable to one vm at a time.

so it's safe to use, you won't lose any data.

---

For the performance part. you just have to test it. for your specific workload. if you have plenty of spare ram, and your database is read heavy, and seldom writes. the whole database will be cached by the os (linux) disk cache. so it will be crazy fast. even on hdd storage.

but if you are low on spare ram. than the database won't be in the os cache. and writes are always synced to disk. and that causes lots of I/O operations. in that case use the highest performing disk you can / are willing to afford.