Dynamically remove substring from string

Question

I have a somewhat straight forward dilemma but I have not been able to find a solution for it.

I am getting a JSON file from a MongoDB database and using json.dumps to convert it into a string:

client = MongoClient("XXXX")
db = client.testdb
table = db.testingtable
document = table.find()
document = list(document)
docs = json.dumps(document,default=str)
print(docs)

Current Output:

[{"_id": "67", "userId": 167, "productID": "Reference('product_id', ObjectId('5f4523b893d7b757bcbd2d71'))"}]

How would I go about dynamically searching the entire string for the word Reference and if found, it should remove everything apart from the alphanumeric value within the brackets without doing this:

docs.replace("Reference('product_id', ObjectId('",'').replace("'))",'')

Expected Output:

[{"_id": "67", "userId": 167, "productID": "5f4523b893d7b757bcbd2d71"}]

Answer 1

from pandas.io.json import json_normalize
df=pd.json_normalize([{"_id": "67", "userId": 167, "productID": "Reference('product_id', ObjectId('5f4523b893d7b757bcbd2d71'))"}])#convert to df


#Using regex, extract string between `ObjectId(`and  `)` and turn it back to json
df.assign(productID=df.productID.str.extract('(?<=ObjectId\()(.*?)(?=\))')).to_json(orient="records")


[{"_id":"67","userId":167,"productID":"'5f4523b893d7b757bcbd2d71'"}]