Firebase Firestore query bug if the document uses a large Map field

Question

After I upgraded the Firestore android sdk from v20.1.0 to v21.6.0, my Firestore queries are not working as expected. If I make a simple get() query or a snapshotListener query (realtime) on a single document, the query works once (or twice) then stops working or getting updates (if realtime query), the onEvent() method wouldn't be triggered.

After deep investigation and multiple tests, we (my work team) found that what causes this is that our documents, each, contain a large Map field with at least 5 levels of multi key-value pairs. Our map field is called "items", when we tried to remove it from the document, the realtime query works perfectly as expected.

PS: we tried to check the size of the documents, the average was between 6k bytes and 5k bytes, and Firestore allows 1Mb (1M bytes) as a maximum document size.

To reproduce the bug, follow below steps :

1. Add this line into your project gradle file :

    dependencies { classpath 'com.google.gms:google-services:4.3.3' }

2. Add those two lines into the module:app gradle file :

    dependencies { implementation 'com.google.firebase:firebase-firestore:21.6.0' implementation 'com.google.code.gson:gson:2.8.6' }

3. Add this code to create and write a copy of my document into firestore :

    private void createAndWriteDocument() { GsonBuilder gsonBuilder = new GsonBuilder(); gsonBuilder.setDateFormat("MMM d, yyyy HH:mm:ss"); gson = gsonBuilder.create(); FirebaseFirestore.getInstance() .collection(COLLECTION_ORDERS) .document("ZYaJKmQF9n1RknziD1tj") .set(gson.fromJson(jsonOrder2, FireStoreOrder.class)); } private final String jsonOrder2 = "{\\n" + " \\"app_name\\":\\"Firestore-Sample\\",\\n" + " \\"app_version\\":\\"1.0.0\\",\\n" + " \\"canceled_at\\":\\"Sep 4, 2020 15:12:34\\",\\n" + " \\"comments_count\\":4,\\n" + " \\"countryID\\":\\"fr\\",\\n" + " \\"created_at\\":\\"Sep 4, 2020 12:21:55\\",\\n" + " \\"currency\\":\\"EUR\\",\\n" + " \\"delivered_at\\":\\"Sep 4, 2020 15:11:30\\",\\n" + " \\"delivery_agent_id\\":\\"1rvlTtfJKNVPVxZ2mp4T2CcI1Fj2\\",\\n" + " \\"delivery_assignment_status\\":\\"accepted\\",\\n" + " \\"delivery_fee\\":5.0,\\n" + " \\"distance\\":3.2547,\\n" + " \\"hearts_count\\":0,\\n" + " \\"in_progress_at\\":\\"Sep 4, 2020 15:06:51\\",\\n" + " \\"is_friends\\":false,\\n" + " \\"is_private\\":false,\\n" + " \\"is_public\\":true,\\n" + " \\"is_visible\\":true,\\n" + " \\"items\\":{\\n" + " \\"-MGNqMoKy8YovI11-QUK\\":{\\n" + " \\"category_id\\":\\"-LlbKEnwkC4KZtJyN691\\",\\n" + " \\"discount_price\\":0.0,\\n" + " \\"extras\\":{\\n" + " \\"-M1-qhcXr8tp5KpnLp9p\\":{\\n" + " \\"max\\":10,\\n" + " \\"name\\":\\"Chèvre\\",\\n" + " \\"price\\":0.6,\\n" + " \\"quantity\\":5,\\n" + " \\"row\\":6\\n" + " },\\n" + " \\"-M1-rHnm7Xzjo7rE73Pk\\":{\\n" + " \\"max\\":10,\\n" + " \\"name\\":\\"Vache kiri\\",\\n" + " \\"price\\":0.6,\\n" + " \\"quantity\\":7,\\n" + " \\"row\\":4\\n" + " },\\n" + " \\"-M1-me-3qKOqWzul67Sr\\":{\\n" + " \\"is_sub_extra\\":true,\\n" + " \\"max\\":3,\\n" + " \\"name\\":\\"Crispy Tenders\\",\\n" + " \\"price\\":0.0,\\n" + " \\"quantity\\":1,\\n" + " \\"row\\":0\\n" + " },\\n" + " \\"-M1-mKvK76SYLJBf3zh-\\":{\\n" + " \\"is_sub_extra\\":true,\\n" + " \\"max\\":3,\\n" + " \\"name\\":\\"Escalope de Poulet\\",\\n" + " \\"price\\":0.0,\\n" + " \\"quantity\\":2,\\n" + " \\"row\\":0\\n" + " },\\n" + " \\"-M1-rNaEFCyHlf9WIOaH\\":{\\n" + " \\"max\\":10,\\n" + " \\"name\\":\\"Raclette\\",\\n" + " \\"price\\":0.6,\\n" + " \\"quantity\\":4,\\n" + " \\"row\\":5\\n" + " }\\n" + " },\\n" + " \\"extras_title\\":\\"Suppléments\\",\\n" + " \\"ingredients\\":{\\n" + " \\"-MEYbO7SlOBFWUVzcvg6\\":{\\n" + " \\"description\\":\\"Ingredients description\\",\\n" + " \\"name\\":\\"Cheddar\\",\\n" + " \\"row\\":0\\n" + " },\\n" + " \\"-MEYbLvKvqPu3YZPPAjo\\":{\\n" + " \\"description\\":\\"Ingredients description\\",\\n" + " \\"name\\":\\"Laitus\\",\\n" + " \\"row\\":0\\n" + " }\\n" + " },\\n" + " \\"ingredients_title\\":\\"Ingrédients : Décocher pour enlever\\",\\n" + " \\"itemPrice\\":0.0,\\n" + " \\"main_image\\":{\\n" + " \\"ref\\":\\"/ZKRBeLgCauXKZ15rfRcuQTY2S1k1/-LlXe_ZbTQytlKujrhLF/products/1582717663739.jpg\\",\\n" + " \\"url\\":\\"https://firebasestorage.googleapis.com/v0/b/urban-food-a9a70.appspot.com/o/ZKRBeLgCauXKZ15rfRcuQTY2S1k1%2F-LlXe_ZbTQytlKujrhLF%2Fproducts%2F1582717663739.jpg?alt\\\=media\\\&token\\\=1bbfd8a0-be5a-44c2-8dd9-d5a7dacf2243\\"\\n" + " },\\n" + " \\"max_extras\\":30,\\n" + " \\"name\\":\\"Tacos\\",\\n" + " \\"options\\":{\\n" + " \\"-LlfWYJR_zKtO1l0DjnF\\":{\\n" + " \\"elements\\":{\\n" + " \\"-LlfWj5L6weXYZUPUoGW\\":{\\n" + " \\"extras_title\\":\\"3 Viandes\\",\\n" + " \\"max\\":1,\\n" + " \\"max_extras\\":3,\\n" + " \\"min_extras\\":3,\\n" + " \\"name\\":\\"L : Choisir 3 Viandes\\",\\n" + " \\"price\\":9.5,\\n" + " \\"row\\":0\\n" + " }\\n" + " },\\n" + " \\"name\\":\\"Taille\\",\\n" + " \\"row\\":-1\\n" + " },\\n" + " \\"-LlfX5HY1rzcQpz1p6mG\\":{\\n" + " \\"elements\\":{\\n" + " \\"-LlfXbVB-QA-utApD3HO\\":{\\n" + " \\"max\\":3,\\n" + " \\"name\\":\\"Barbecue\\",\\n" + " \\"price\\":0.0,\\n" + " \\"quantity\\":2,\\n" + " \\"row\\":0\\n" + " },\\n" + " \\"-LlfXxF-7RRTZhFwXhyX\\":{\\n" + " \\"max\\":3,\\n" + " \\"name\\":\\"Marocaine\\",\\n" + " \\"price\\":0.0,\\n" + " \\"quantity\\":2,\\n" + " \\"row\\":0\\n" + " }\\n" + " },\\n" + " \\"max_quantity\\":4,\\n" + " \\"min_quantity\\":1,\\n" + " \\"name\\":\\"Sauces\\",\\n" + " \\"row\\":3\\n" + " },\\n" + " \\"-LlfWPIAjjgTiqtqEzep\\":{\\n" + " \\"elements\\":{\\n" + " \\"-LlfWPI9_w1gH1A3HTsB\\":{\\n" + " \\"name\\":\\"Tacos\\",\\n" + " \\"price\\":0.0,\\n" + " \\"row\\":0\\n" + " }\\n" + " },\\n" + " \\"name\\":\\"Type\\",\\n" + " \\"row\\":2\\n" + " }\\n" + " },\\n" + " \\"price\\":0.0,\\n" + " \\"product_id\\":\\"-LlfTFlgrNBdLZRn5xgn\\",\\n" + " \\"quantity\\":2,\\n" + " \\"sub_category_id\\":\\"-LlbKEnvPMIV3DnSiYlJ\\"\\n" + " },\\n" + " \\"-MGNqOGcRfYb-myLgc1d\\":{\\n" + " \\"category_id\\":\\"-LlkUx5iBSXhKdMYGxTA\\",\\n" + " \\"discount_price\\":3.5,\\n" + " \\"extras\\":{\\n" + " \\"-M2JOJaB_Gsc28GT8wEg\\":{\\n" + " \\"max\\":2,\\n" + " \\"name\\":\\"Harissa\\",\\n" + " \\"price\\":0.0,\\n" + " \\"quantity\\":1,\\n" + " \\"row\\":2\\n" + " }\\n" + " },\\n" + " \\"extras_title\\":\\"Sauces\\",\\n" + " \\"itemPrice\\":0.0,\\n" + " \\"main_image\\":{\\n" + " \\"ref\\":\\"/ZKRBeLgCauXKZ15rfRcuQTY2S1k1/-LlXe_ZbTQytlKujrhLF/products/1582809519364.jpg\\",\\n" + " \\"url\\":\\"https://firebasestorage.googleapis.com/v0/b/urban-food-a9a70.appspot.com/o/ZKRBeLgCauXKZ15rfRcuQTY2S1k1%2F-LlXe_ZbTQytlKujrhLF%2Fproducts%2F1582809519364.jpg?alt\\\=media\\\&token\\\=abc457cb-2462-44be-886a-af83db3cb501\\"\\n" + " },\\n" + " \\"max_extras\\":2,\\n" + " \\"name\\":\\"Oignons frites\\",\\n" + " \\"options\\":{\\n" + " \\"-M15mpsXwAzsDRlLDHJI\\":{\\n" + " \\"elements\\":{\\n" + " \\"-M15mpsXwAzsDRlLDHJG\\":{\\n" + " \\"name\\":\\"Cheddar\\",\\n" + " \\"price\\":0.0,\\n" + " \\"row\\":0\\n" + " }\\n" + " },\\n" + " \\"name\\":\\"Au choix :\\",\\n" + " \\"row\\":-1\\n" + " }\\n" + " },\\n" + " \\"price\\":3.5,\\n" + " \\"product_id\\":\\"-M15mV_4Ui2B90GlzsaR\\",\\n" + " \\"quantity\\":1,\\n" + " \\"sub_category_id\\":\\"-LlkUx5h8rXECP8L_hjQ\\"\\n" + " },\\n" + " \\"-MGNqOxoHFGoigUXao96\\":{\\n" + " \\"category_id\\":\\"-LlkUx5iBSXhKdMYGxTA\\",\\n" + " \\"discount_price\\":3.5,\\n" + " \\"extras\\":{\\n" + " \\"-M2JOmBf1oN1BmUUHeiY\\":{\\n" + " \\"name\\":\\"Biggy burger\\",\\n" + " \\"price\\":0.0,\\n" + " \\"quantity\\":1,\\n" + " \\"row\\":0\\n" + " }\\n" + " },\\n" + " \\"itemPrice\\":0.0,\\n" + " \\"main_image\\":{\\n" + " \\"ref\\":\\"/ZKRBeLgCauXKZ15rfRcuQTY2S1k1/-LlXe_ZbTQytlKujrhLF/products/1582809369789.jpg\\",\\n" + " \\"url\\":\\"https://firebasestorage.googleapis.com/v0/b/urban-food-a9a70.appspot.com/o/ZKRBeLgCauXKZ15rfRcuQTY2S1k1%2F-LlXe_ZbTQytlKujrhLF%2Fproducts%2F1582809369789.jpg?alt\\\=media\\\&token\\\=705593e0-e403-4931-b040-5b2261cb96e5\\"\\n" + " },\\n" + " \\"name\\":\\"Bacon frites\\",\\n" + " \\"options\\":{\\n" + " \\"-M15lyQyHs1GujtZxMtS\\":{\\n" + " \\"elements\\":{\\n" + " \\"-M15lyQxrRFaLAWGaa3i\\":{\\n" + " \\"name\\":\\"Sauce fromagère\\",\\n" + " \\"price\\":0.0,\\n" + " \\"row\\":0\\n" + " }\\n" + " },\\n" + " \\"name\\":\\"Au choix :\\",\\n" + " \\"row\\":-1\\n" + " }\\n" + " },\\n" + " \\"price\\":3.5,\\n" + " \\"product_id\\":\\"-M15lqmhF16ZOLqebPzs\\",\\n" + " \\"quantity\\":1,\\n" + " \\"sub_category_id\\":\\"-LlkUx5h8rXECP8L_hjQ\\"\\n" + " }\\n" + " },\\n" + " \\"itemsCount\\":0,\\n" + " \\"level_one_zone_id\\":\\"Île-de-France\\",\\n" + " \\"level_two_zone_id\\":\\"Paris\\",\\n" + " \\"order_id\\":\\"ZYaJKmQF9n1RknziD1tj\\",\\n" + " \\"order_number\\":31,\\n" + " \\"order_type\\":\\"delivery\\",\\n" + " \\"paid\\":false,\\n" + " \\"paid_with_loyalty\\":0.0,\\n" + " \\"picked_at\\":\\"Sep 4, 2020 15:12:46\\",\\n" + " \\"platform\\":\\"android\\",\\n" + " \\"processedAt\\":{\\n" + " \\"nanoseconds\\":994000000,\\n" + " \\"seconds\\":1599228406\\n" + " },\\n" + " \\"restaurant_photo\\":\\"https://firebasestorage.googleapis.com/v0/b/menutium-319d0.appspot.com/o/LHx3fTdtoaRlrAHSjznJSaPz9rP2%2F-MEYUb7iepk3pLwUyWX3%2Fprofile%2F1597250393877.jpg?alt\\\=media\\\&token\\\=cbccd3fe-91fd-4397-ab34-79f5f241def2\\",\\n" + " \\"status\\":\\"picked\\",\\n" + " \\"store_id\\":\\"-MEYUb7iepk3pLwUyWX3\\",\\n" + " \\"store_name\\":\\"La Fourchette\\",\\n" + " \\"total_price\\":50.2,\\n" + " \\"updated_at\\":\\"Sep 7, 2020 18:54:02\\",\\n" + " \\"user_address\\":\\"166 Quai de Stalingrad, 92130 Issy-les-Moulineaux, France\\\\nIssy-les-Moulineaux\\",\\n" + " \\"user_coordinates\\":\\"48.8256954,2.2579879\\",\\n" + " \\"user_name\\":\\"Mo Salah\\",\\n" + " \\"user_phone\\":\\"+21650001002\\",\\n" + " \\"user_photo\\":\\"https://firebasestorage.googleapis.com/v0/b/menutium-319d0.appspot.com/o/2Bz8euUmPgMqc4Z7QZWJjLfsgV72%2Fphoto_profile?alt\\\=media\\\&token\\\=f5dcee5b-fdda-45bd-9ec3-982cfe7832bd\\",\\n" + " \\"user_uid\\":\\"2Bz8euUmPgMqc4Z7QZWJjLfsgV72\\",\\n" + " \\"validated_at\\":\\"Sep 4, 2020 15:12:42\\",\\n" + " \\"validatedBy\\":{\\n" + " \\"name\\":\\"Urban-Admin\\",\\n" + " \\"id\\":\\"admin\\"\\n" + " }\\n" + "}";

4. Start a realtime query on this document :

    FirebaseFirestore.getInstance() .collection("orders") .document("ZYaJKmQF9n1RknziD1tj") .addSnapshotListener(new EventListener<DocumentSnapshot>() { @Override public void onEvent(@Nullable DocumentSnapshot value, @Nullable FirebaseFirestoreException error) { Log.d("testing", "onEvent triggered"); if (error != null) { Log.i("testing", "test error : "+ error.getMessage()); } } });

We created two Android projects (Java code), put them into public Github repos so that anyone could reproduce the issue we are facing.

Everything is detailed into the Readme repos.

First Project repository

Second Project repository with minimal code

Answer 1

Before i create this question, i created an issue within the official firebase-android-sdk repository. After five days, someone from the Firebase team answered me, they did found a bug in the Firestore sdk, fixed it, but still didn't released it, so he recommended to downgrade to older version than 21.5.0 to avoid the bug as a temporary workaround. Github issue link here