How do I get documents where a specific field exists/does not exists in Firebase Cloud Firestore?

Question

In Firebase Cloud Firestore, I have "user_goals" in collections and goals may be a predefined goal (master_id: "XXXX") or a custom goal (no "master_id" key)

In JavaScript, I need to write two functions, one to get all predefined goals and other to get all custom goals.

I have got some workaround to get custom goals by setting "master_id" as "" empty string and able to get as below:

db.collection('user_goals')
    .where('challenge_id', '==', '')  // workaround works
    .get()

Still this is not the correct way, I continued to use this for predefined goals where it has a "master_id" as below

db.collection('user_goals')
    .where('challenge_id', '<', '')  // this workaround
    .where('challenge_id', '>', '')  // is not working
    .get()

Since Firestore has no "!=" operator, I need to use "<" and ">" operator but still no success.

Question: Ignoring these workarounds, what is the preferred way to get docs by checking whether a specific field exists or does not exists?

Answer 1

As @Emile Moureau solution. I prefer

.orderBy(`field`)

To query documents with the field exists. Since it will work with any type of data with any value even for null .

But as @Doug Stevenson said:

You can't query for something that doesn't exist in Firestore. A field needs to exist in order for a Firestore index to be aware of it.

You can't query for documents without the field. At least for now.

Answer 2

The preferred way to get docs where a specified field exists is to use the:

.orderBy(fieldPath)

As specified in the Firebase documentation :

Thus the answer provided by @hisoft is valid. I just decided to provide the official source, as the question was for the preferred way.

Answer 3

The solution I use is:

Use: .where('field', '>', ''),

Where "field" is the field we are looking for!

Answer 4

Firestore is an indexed database . For each field in a document, that document is inserted into that field's index as appropriate based on your configuration . If a document doesn't contain a particular field (like challenge_id ) it will not appear in that field's index and will be omitted from queries on that field. Normally, because of the way Firestore is designed, queries should read an index in one continuous sweep. Prior to the introduction of the != and not-in operators, this meant you couldn't exclude particular values as this would require jumping over sections of an index. This limitation is still encountered when trying to use exclusive ranges ( v<2 || v>4 ) in a single query.

Field values are sorted according to the Realtime Database sort order except that the results can be sorted by multiple fields when duplicates are encountered instead of just the document's ID.

Firestore Value Sort Order

Priority	Sorted Values	Priority	Sorted Values
1	null	6	strings
2	false	7	DocumentReference
3	true	8	GeoPoint
4	numbers	9	arrays
5	Timestamp	10	maps

Inequality != / <>

This section documents how inequalities worked prior to the release of the != and not-in operators in Sep 2020 . See the documentation on how to use these operators. The following section will be left for historical purposes.

To perform an inequality query on Firestore, you must rework your query so that it can be read by reading from Firestore's indexes. For an inequality, this is done by using two queries - one for values less than the equality and another for values greater than the equality.

As a trivial example, let's say I wanted the numbers that aren't equal to 3.

const someNumbersThatAreNotThree = someNumbers.filter(n => n !== 3)

can be written as

const someNumbersThatAreNotThree = [
   ...someNumbers.filter(n => n < 3),
   ...someNumbers.filter(n => n > 3)
];

Applying this to Firestore, you can convert this ( formerly ) incorrect query:

const docsWithChallengeID = await colRef
  .where('challenge_id', '!=', '')
  .get()
  .then(querySnapshot => querySnapshot.docs);

into these two queries and merge their results:

const docsWithChallengeID = await Promise.all([
  colRef
    .orderBy('challenge_id')
    .endBefore('')
    .get()
    .then(querySnapshot => querySnapshot.docs),
  colRef
    .orderBy('challenge_id')
    .startAfter('')
    .get()
    .then(querySnapshot => querySnapshot.docs),
]).then(results => results.flat());

Important Note: The requesting user must be able to read all the documents that would match the queries to not get a permissions error.

Missing/Undefined Fields

Simply put, in Firestore, if a field doesn't appear in a document, that document won't appear in that field's index. This is in contrast to the Realtime Database where omitted fields had a value of null .

Because of the nature of NoSQL databases where the schema you are working with might change leaving your older documents with missing fields, you might need a solution to "patch your database". To do this, you would iterate over your collection and add the new field to the documents where it is missing.

To avoid permissions errors, it is best to make these adjustments using the Admin SDK with a service account, but you can do this using a regular SDK using a user with the appropriate read/write access to your database.

This function is recursive, and is intended to be executed once .

async function addDefaultValueForField(queryRef, fieldName, defaultFieldValue, pageSize = 100) {
  let checkedCount = 0, pageCount = 1;
  const initFieldPromises = [], newData = { [fieldName]: defaultFieldValue };

  // get first page of results
  console.log(`Fetching page ${pageCount}...`);
  let querySnapshot = await queryRef
    .limit(pageSize)
    .get();

  // while page has data, parse documents
  while (!querySnapshot.empty) {
    // for fetching the next page
    let lastSnapshot = undefined;

    // for each document in this page, add the field as needed
    querySnapshot.forEach(doc => {
      if (doc.get(fieldName) === undefined) {
        const addFieldPromise = doc.ref.update(newData)
          .then(
            () => ({ success: true, ref: doc.ref }),
            (error) => ({ success: false, ref: doc.ref, error }) // trap errors for later analysis
          );

        initFieldPromises.push(addFieldPromise);
      }

      lastSnapshot = doc;
    });

    checkedCount += querySnapshot.size;
    pageCount++;

    // fetch next page of results
    console.log(`Fetching page ${pageCount}... (${checkedCount} documents checked so far, ${initFieldPromises.length} need initialization)`);
    querySnapshot = await queryRef
      .limit(pageSize)
      .startAfter(lastSnapshot)
      .get();
  }

  console.log(`Finished searching documents. Waiting for writes to complete...`);

  // wait for all writes to resolve
  const initFieldResults = await Promise.all(initFieldPromises);

  console.log(`Finished`);

  // count & sort results
  let initializedCount = 0, errored = [];
  initFieldResults.forEach((res) => {
    if (res.success) {
      initializedCount++;
    } else {
      errored.push(res);
    }
  });

  const results = {
    attemptedCount: initFieldResults.length,
    checkedCount,
    errored,
    erroredCount: errored.length,
    initializedCount
  };

  console.log([
    `From ${results.checkedCount} documents, ${results.attemptedCount} needed the "${fieldName}" field added.`,
    results.attemptedCount == 0
      ? ""
      : ` ${results.initializedCount} were successfully updated and ${results.erroredCount} failed.`
  ].join(""));

  const errorCountByCode = errored.reduce((counters, result) => {
    const code = result.error.code || "unknown";
    counters[code] = (counters[code] || 0) + 1;
    return counters;
  }, {});
  console.log("Errors by reported code:", errorCountByCode);

  return results;
}

You would then apply changes using:

const goalsQuery = firebase.firestore()
  .collection("user_goals");

addDefaultValueForField(goalsQuery, "challenge_id", "")
  .catch((err) => console.error("failed to patch collection with new default value", err));

The above function could also be tweaked to allow the default value to be calculated based on the document's other fields:

let getUpdateData;
if (typeof defaultFieldValue === "function") {
  getUpdateData = (doc) => ({ [fieldName]: defaultFieldValue(doc) });
} else {
  const updateData = { [fieldName]: defaultFieldValue };
  getUpdateData = () => updateData;
}

/* ... later ... */
const addFieldPromise = doc.ref.update(getUpdateData(doc))

Answer 5

As you correctly state, it is not possible to filter based on != . If possible, I would add an extra field to define the goal type. It is possible to use != in security rules, along with various string comparison methods, so you can enforce the correct goal type, based on your challenge_id format.

Specify the goal type

Create a type field and filter based on this field.

type: master or type: custom and search .where('type', '==', 'master') or search for custom.

Flag custom goals

Create a customGoal field which can be true or false .

customGoal: true and search .where('customGoal', '==', true) or false (as required).

Update

It is now possible to perform a != query in Cloud Firestore

Answer 6

Firestore does pick up on boolean, which is a thing! and can be orderBy 'd.

So often, like now, for this, I add this into the array-pushing from onSnapshot or get , use .get().then( for dev...

if (this.props.auth !== undefined) {
  if (community && community.place_name) {
    const sc =
      community.place_name && community.place_name.split(",")[1];
      const splitComma = sc ? sc : false
    if (community.splitComma !== splitComma) {
      firebase
        .firestore()
        .collection("communities")
        .doc(community.id)
        .update({ splitComma });
    }
    const sc2 =
      community.place_name && community.place_name.split(",")[2];
      const splitComma2 =sc2 ? sc2 : false
    console.log(splitComma2);
    if (community.splitComma2 !== splitComma2) {
      firebase
        .firestore()
        .collection("communities")
        .doc(community.id)
        .update({
          splitComma2
        });
    }
  }

This way, I can query with orderBy instead of where

browseCommunities = (paginate, cities) => {
  const collection = firebase.firestore().collection("communities");
    const query =
      cities === 1 //countries
        ? collection.where("splitComma2", "==", false) //without a second comma
        : cities //cities
        ? collection
            .where("splitComma2", ">", "")
            .orderBy("splitComma2", "desc") //has at least two
        : collection.orderBy("members", "desc");
  var shot = null;
  if (!paginate) {
    shot = query.limit(10);
  } else if (paginate === "undo") {
    shot = query.startAfter(this.state.undoCommunity).limit(10);
  } else if (paginate === "last") {
    shot = query.endBefore(this.state.lastCommunity).limitToLast(10);
  }
  shot &&
    shot.onSnapshot(
      (querySnapshot) => {
        let p = 0;
        let browsedCommunities = [];
        if (querySnapshot.empty) {
          this.setState({
            [nuller]: null
          });
        }
        querySnapshot.docs.forEach((doc) => {
          p++;
          if (doc.exists) {
            var community = doc.data();
            community.id = doc.id;

Answer 7

It is not an ideal solution, but here is my workaround when a field does not exist:

let user_goals = await db.collection('user_goals').get()
user_goals.forEach(goal => {
  let data = goal.data()
  if(!Object.keys(data).includes(challenge_id)){
    //Perform your task here
  }
})

Note that it would impact your read counts a lot so only use this if you have small collection or can afford the reads.