Importing data from firestore to python

Question

I need to transfer some collections from firestore to pandas dataframe for analysis ans have some problems

Method from firebase docs:

docs = db.collection(COLLECTION_NAME).stream()

for doc in docs:
    print(f'{doc.id} => {doc.to_dict()}')

my collection contains 100k elements, and when iterator at loop is near 50k i receive an error:

'_UnaryStreamMultiCallable' object has no attribute '_retry' 

I load only one collection, wout parallel downloads

Answer 1

As mentioned in the link_1 and link_2 , you can check for the syntax of importing collections from firestore to pandas dataframe.

Example 1:

 import pandas as pd ref = db.collection(u'user') docs = ref.stream() items = list(map(lambda x: {**x.to_dict(), 'id': x.id}, docs)) df = pd.DataFrame(items) #, columns=['id', 'email'] df.set_index('id', inplace=True)

Example 2:

 import pandas as pd from google.cloud import firestore db = firestore.Client() users = list(db.collection(u'users').stream()) users_dict = list(map(lambda x: x.to_dict(), users)) df = pd.DataFrame(users_dict)

For more information, You can refer to the Stackoverflow thread1 and thread2 :