Firebase Firestore - does creating a snapshot event listener cost excessive downloads?

Question

Initially, my use case was paginating data with snapshot listener, like here: Firestore Paginating data + Snapshot listener

But the answer there said it is not currently supported, so I tried to find a workaround, which I found here https://medium.com/@650egor/firestore-reactive-pagination-db3afb0bf42e . Which is nice, but kind of complicated. Also, it has potentially n times the downloads as normal because each change in the early listener is also caught by the later listeners as well.

So now, I'm thinking of dropping the pagination. Instead, each time I want to get more data, I will simply recreate the snapshot listener, but with a 2x limit. Like so:

const [limit, setLimit] = useState(50);
const [data, setData] = useState([]);
...
useEffect(()=> {
  const datalist = [];
  db.collection('chats')
  .where('id','==', chatId)
  .limit(limit)
  .onSnapshot((querySnapshot) =>{
    querySnapshot.forEach((item) => datalist.push(item));
    setData(datalist);
  }
  
}, [limit]);

return <Button title="get more data" onPress={()=> { setLimit(limit * 2}} />;

My question is, is that bad in terms of excessive downloads (in terms of the spark plan)? When I do snapshot for the first time, it should be downloading 50 items, then for the second time 100, then 200. I'd like to confirm if that's how it works.

Also, if there's a reason this approach won't work on a more funadamental level, I'd like to know.

Answer 1

Each time you perform a query that does not specifically target the local persistence cache, it will retrieve the full set of documents from Firestore. That's the only piece of information you need to know. A subsequent query will not deliver partially cached results from a prior query.

The code you show right now is actually very problematic in that it leaks listeners. There is nothing in place that stops the prior listener if the limit changes and causes the hook to execute again. You should return a function from your useEffect hook that unsubscribes the listener when it's no longer needed. Just return the unsubscribe function returned by onSnapshot . You should also read the documentation foruseEffect hooks that require cleanup , as your does here. The potential cost of leaking a listener is potentially much worse than the cost of the repeating the query with new limits, as a leaked listener will continually read new documents as they are changed - that's why you have to unsubscribe as soon as you don't need them any more.

Answer 2

You do understand it correctly.

With your implementation, you will be billed for 50 reads the first time, 100 reads for the second time, 200 reads for the third time and so on (if the number of documents were less than the limit, you will be billed for the number of documents).

I actually do use approach very similar to this approach in one of my published app, but instead of doubling the number of documents to load every time, I add a certain number to the limit.

Answer 3

I do paginated Listeners, by constructing an object that tracks some internal state, and has PageBack, PageForward, ChangeLimit and Unsubscribe methods. For Listeners, it is best to unsubscribe the previous listener and set up a new one; this code does just that. Probably more efficient to add a layer, here: use somewhat larger pages to Paginate from Firestore (a little compute-expensive setting up and tearing down) weighed against actual number of records fetched (actual cost), and then serving smaller pages locally. BUT, for the PaginatedListener:

/**
 * ----------------------------------------------------------------------
 * @function filterQuery
 * builds and returns a query built from an array of filter (i.e. "where")
 * consitions
 * @param {Query} query collectionReference or Query to build filter upong
 * @param {array} filterArray an (optional) 3xn array of filter(i.e. "where") conditions
 * @returns Firestor Query object
 */
export const filterQuery = (query, filterArray = null) => {
  return filterArray
    ? filterArray.reduce((accQuery, filter) => {
        return accQuery.where(filter.fieldRef, filter.opStr, filter.value);
      }, query)
    : query;
};

/**
 * ----------------------------------------------------------------------
 * @function sortQuery
 * builds and returns a query built from an array of filter (i.e. "where")
 * consitions
 * @param {Query} query collectionReference or Query to build filter upong
 * @param {array} sortArray an (optional) 2xn array of sort (i.e. "orderBy") conditions
 * @returns Firestor Query object
 */
export const sortQuery = (query, sortArray = null) => {
  return sortArray
    ? sortArray.reduce((accQuery, sortEntry) => {
        return accQuery.orderBy(sortEntry.fieldRef, sortEntry.dirStr || "asc");
        //note "||" - if dirStr is not present(i.e. falsy) default to "asc"
      }, query)
    : query;
};

/**
 * ----------------------------------------------------------------------
 * @classdesc 
 * An object to allow for paginating a listener for table read from Firestore.
 * REQUIRES a sorting choice
 * masks some subscribe/unsubscribe action for paging forward/backward
 * @property {Query} Query that forms basis for the table read
 * @property {number} limit page size
 * @property {QuerySnapshot} snapshot last successful snapshot/page fetched
 * @property {enum} status status of pagination object
 *
 * @method PageForward Changes the listener to the next page forward
 * @method PageBack Changes the listener to the next page backward
 * @method Unsubscribe returns the unsubscribe function
 * ----------------------------------------------------------------------
 */

export class PaginatedListener {
  _setQuery = () => {
    const db = this.ref ? this.ref : fdb;
    this.Query = sortQuery(
      filterQuery(db.collection(this.table), this.filterArray),
      this.sortArray
    );
    return this.Query;
  };

  /**
   * ----------------------------------------------------------------------
   * @constructs PaginatedListener constructs an object to paginate through large
   * Firestore Tables
   * @param {string} table a properly formatted string representing the requested collection
   * - always an ODD number of elements
   * @param {array} filterArray an (optional) 3xn array of filter(i.e. "where") conditions
   * @param {array} sortArray a 2xn array of sort (i.e. "orderBy") conditions
   * @param {ref} ref (optional) allows "table" parameter to reference a sub-collection
   * of an existing document reference (I use a LOT of structered collections)
   *
   * The array is assumed to be sorted in the correct order -
   * i.e. filterArray[0] is added first; filterArray[length-1] last
   * returns data as an array of objects (not dissimilar to Redux State objects)
   * with both the documentID and documentReference added as fields.
   * @param {number} limit (optional)
   * @param {function} dataCallback
   * @param {function} errCallback
   * **********************************************************/

  constructor(
    table,
    filterArray = null,
    sortArray,
    ref = null,
    limit = PAGINATE_DEFAULT,
    dataCallback = null,
    errCallback = null
  ) {
    this.table = table;
    this.filterArray = filterArray;
    this.sortArray = sortArray;
    this.ref = ref;
    this.limit = limit;
    this._setQuery();
    /*this.Query = sortQuery(
      filterQuery(db.collection(this.table), this.filterArray),
      this.sortArray
    );*/
    this.dataCallback = dataCallback;
    this.errCallback = errCallback;
    this.status = PAGINATE_INIT;
  }

  /**
   * @method PageForward
   * @returns Promise of a QuerySnapshot
   */
  PageForward = () => {
    const runQuery =
      this.unsubscriber && !this.snapshot.empty
        ? this.Query.startAfter(_.last(this.snapshot.docs))
        : this.Query;

    //IF unsubscribe function is set, run it.
    this.unsubscriber && this.unsubscriber();

    this.status = PAGINATE_PENDING;

    this.unsubscriber = runQuery.limit(Number(this.limit)).onSnapshot(
      (QuerySnapshot) => {
        this.status = PAGINATE_UPDATED;
        //*IF* documents (i.e. haven't gone back ebfore start)
        if (!QuerySnapshot.empty) {
          //then update document set, and execute callback
          this.snapshot = QuerySnapshot;
        }
        this.dataCallback(
          this.snapshot.docs.map((doc) => {
            return {
              ...doc.data(),
              Id: doc.id,
              ref: doc.ref
            };
          })
        );
      },
      (err) => {
        this.errCallback(err);
      }
    );
    return this.unsubscriber;
  };

  /**
   * @method PageBack
   * @returns Promise of a QuerySnapshot
   */
  PageBack = () => {
    const runQuery =
      this.unsubscriber && !this.snapshot.empty
        ? this.Query.endBefore(this.snapshot.docs[0])
        : this.Query;

    //IF unsubscribe function is set, run it.
    this.unsubscriber && this.unsubscriber();

    this.status = PAGINATE_PENDING;

    this.unsubscriber = runQuery.limitToLast(Number(this.limit)).onSnapshot(
      (QuerySnapshot) => {
        //acknowledge complete
        this.status = PAGINATE_UPDATED;
        //*IF* documents (i.e. haven't gone back ebfore start)
        if (!QuerySnapshot.empty) {
          //then update document set, and execute callback
          this.snapshot = QuerySnapshot;
        }
        this.dataCallback(
          this.snapshot.docs.map((doc) => {
            return {
              ...doc.data(),
              Id: doc.id,
              ref: doc.ref
            };
          })
        );
      },
      (err) => {
        this.errCallback(err);
      }
    );
    return this.unsubscriber;
  };

  /**
   * @method ChangeLimit
   * sets page size limit to new value, and restarts the paged listener
   * @param {number} newLimit
   * @returns Promise of a QuerySnapshot
   */
  ChangeLimit = (newLimit) => {
    const runQuery = this.Query;

    //IF unsubscribe function is set, run it.
    this.unsubscriber && this.unsubscriber();

    this.limit = newLimit;

    this.status = PAGINATE_PENDING;

    this.unsubscriber = runQuery.limit(Number(this.limit)).onSnapshot(
      (QuerySnapshot) => {
        this.status = PAGINATE_UPDATED;
        //*IF* documents (i.e. haven't gone back ebfore start)
        if (!QuerySnapshot.empty) {
          //then update document set, and execute callback
          this.snapshot = QuerySnapshot;
        }
        this.dataCallback(
          this.snapshot.docs.map((doc) => {
            return {
              ...doc.data(),
              Id: doc.id,
              ref: doc.ref
            };
          })
        );
      },
      (err) => {
        this.errCallback(err);
      }
    );
    return this.unsubscriber;
  };

  ChangeFilter = (filterArray) => {
    //IF unsubscribe function is set, run it (and clear it)
    this.unsubscriber && this.unsubscriber();

    this.filterArray = filterArray; // save the new filter array
    const runQuery = this._setQuery(); // re-build the query
    this.status = PAGINATE_PENDING;

    //fetch the first page of the new filtered query
    this.unsubscriber = runQuery.limit(Number(this.limit)).onSnapshot(
      (QuerySnapshot) => {
        this.status = PAGINATE_UPDATED;
        //*IF* documents (i.e. haven't gone back ebfore start)
        this.snapshot = QuerySnapshot;
        this.dataCallback(
          this.snapshot.empty
            ? null
            : this.snapshot.docs.map((doc) => {
                return {
                  ...doc.data(),
                  Id: doc.id,
                  ref: doc.ref
                };
              })
        );
      },
      (err) => {
        this.errCallback(err);
      }
    );
    return this.unsubscriber;
  };

  unsubscribe = () => {
    //IF unsubscribe function is set, run it.
    this.unsubscriber && this.unsubscriber();
    this.unsubscriber = null;
  };
}