通过云函数将数据从 pubsub 写入 bigtable

Question

I am a beginner at cloud big table and have big issues using cloud functions writing data from pub/sub to bigtable.

Cloud functions gets the messages from pubsub, but the issue is in the next step, writing it into bigtable.

The message is created in a python script and sent to pub/sub.

One example for a message:

b'{"eda":2.015176,"temperature":33.39,"bvp":-0.49,"x_acc":-36.0,"y_acc":-38.0,"z_acc":-128.0,"heart_rate":83.78,"iddevice":15.0,"timestamp":"2019-12-01T20:01:36.927Z"}'

For writing it into bigtable I created a table:

 from google.cloud import bigtable 
 from google.cloud.bigtable import column_family

 client = bigtable.Client(project="projectid", admin=True) 
 instance = client.instance("bigtableinstance")
 table = instance.table("bigtable1")
 print('Creating the {} table.'.format(table)) 
 print('Creating columnfamily cf1 with Max Version GC rule...')
 max_versions_rule = column_family.MaxVersionsGCRule(2)
 column_family_id = 'cf1'
 column_families = {column_family_id: max_versions_rule}
 if not table.exists():
     table.create(column_families=column_families)
     print("Table {} is created.".format(table)) 
 else:
     print("Table {} already exists.".format(table))

This works without problems.

Now I tried to write the message via pub/sub to bigtable with the following python code in cloud functions using the main method:

import json
import base64
import os
from google.cloud import bigtable
from google.cloud.bigtable import column_family, row_filters


project_id = os.environ.get('projetid', 'UNKNOWN')
INSTANCE = 'bigtableinstance'
TABLE = 'bigtable1'

client = bigtable.Client(project=project_id, admin=True)
instance = client.instance(INSTANCE)

colFamily = "cf1"
def writeToBigTable(table, data):
#    Parameters row_key (bytes) – The key for the row being created.
#    Returns A row owned by this table.
        row_key = data[colFamily]['iddevice'].value.encode()
        row = table.row(row_key)
        for colFamily in data.keys():
            for key in data[colFamily].keys():
                row.set_cell(colFamily,
                                        key,
                                        data[colFamily][key])
        table.mutate_rows([row])
        return data

def selectTable():
    stage = os.environ.get('stage', 'dev')
    table_id = TABLE + stage
    table = instance.table(table_id)
    return table


def main(event, context):
    data = base64.b64decode(event['data']).decode('utf-8')
    print("DATA: {}".format(data))
    eda, temperature, bvp, x_acc, y_acc, z_acc, heart_rate, iddevice, timestamp = data.split(',')

    table = selectTable()

    data = {'eda': eda,
         'temperature': temperature,
         'bvp': bvp,
         'x_acc':x_acc,
         'y_acc':y_acc,
         'z_acc':z_acc,
         'heart_rate':heart_rate,
         'iddevice':iddevice,
         'timestamp':timestamp}
    writeToBigTable(table, data)
    print("Data Written: {}".format(data))

I tried different versions but cannot find a solution.

Thanks for the help.

All the best

Dominik

Answer 1

I think this line is wrong:

    row_key = data[colFamily]['iddevice'].value.encode()

You're passing in the data object, but it doesn't have a 'cf1' property. You also don't have to encode it. Give this a try:

    row_key = data['iddevice']

Your for loop will also have the same issue. I think this is what you want instead

    for col in data.keys():
        row.set_cell(colFamily, key, data[key])

Also, I know you're just playing with it, but using a device id as the only value for a rowkey will end up poorly. What is recommended might be to combine the rowkey and the date or one of your other properties (depending on your query,) and use that as your rowkey. There is a document on Cloud Bigtable schema that is helpful, and a codelab using a more realistic sample dataset and walks through how to pick a schema for that example. It's in Java, but you can still import the data and run your own queries.

Answer 2

first thanks a lot for the help.

I tried to fix it with you code recommendation which is , but unfortunately it doesn't work now due to other errors.

AttributeError: 'DirectRow' object has no attribute 'append'

I guess this is within the following line of code

        row.set_cell(colFamily,
                     key,
                     data[key])

I could imagine that the errors origin is in the split of the string "data"

eda, temperature, bvp, x_acc, y_acc, z_acc, heart_rate, iddevice, timestamp = data.split(',')

Eg eda would look like this:

"'eda':2.015176"

which looks pretty wrong to me.

Especially when I insert it into the following dict:

 data = {'eda': eda,....}

The error

AttributeError: 'DirectRow' object has no attribute 'append' seems to say, that there is a problem with the data I want to process with set_cell. There is said set_cell with row as a list or any other iterable of Direct Row Instance. Shouldn't fit a dic for it?

I tried a workaround with a list, but this seems to make it even worse.

client = bigtable.Client(project=project_id, admin=True)
instance = client.instance(INSTANCE)

colFamily = "cf1"
def writeToBigTable(table, dat):

    row_key = "{}-{}".format(dat[16], dat[17])
    row = table.row(row_key)
    for n in range(len(dat)):
        row.set_cell(colFamily,
                     dat[n],
                     dat[n+9])
    table.mutate_rows([row])
    return dat

def selectTable():
    stage = os.environ.get('stage', 'dev')
    table_id = TABLE + stage
    table = instance.table(table_id)
    return table


def main(event, context):
    data = base64.b64decode(event['data']).decode('utf-8')
    print("DATA: {}".format(data))
    var_1, eda, var_2, temperature, var_3, bvp, var_4, x_acc, var_5, y_acc, var_6, z_acc, var_7, heart_rate, var_8, iddevice, var_9, timestamp = data.replace(':',',').split(',')

    table = selectTable(); dat = [var_1, var_2, var_3, var_4, var_5, var_6, var_7, var_8, var_9, eda, temperature, bvp, x_acc, y_acc, z_acc, heart_rate, iddevice, timestamp]; 

#   data = {'eda': eda,
#         'temperature': temperature,
#         'bvp': bvp,
#         'x_acc':x_acc,
#         'y_acc':y_acc,
#         'z_acc':z_acc,
#         'heart_rate':heart_rate,
#         'iddevice':iddevice,
#         'timestamp':timestamp}
    writeToBigTable(table, dat)
    print("Data Written: {}".format(data))

I am really hard stuck at this problem and have no further ideas how to solve it.