使用 Python / Boto 更新 DynamoDB 原子计数器

Question

I am trying to update an atomic count counter with Python Boto 2.3.0, but can find no documentation for the operation.

It seems there is no direct interface, so I tried to go to "raw" updates using the layer1 interface, but I was unable to complete even a simple update.

I tried the following variations but all with no luck

dynoConn.update_item(INFLUENCER_DATA_TABLE, 
                     {'HashKeyElement': "9f08b4f5-d25a-4950-a948-0381c34aed1c"}, 
                     {'new': {'Value': {'N':"1"}, 'Action': "ADD"}})    

dynoConn.update_item('influencer_data', 
                     {'HashKeyElement': "9f08b4f5-d25a-4950-a948-0381c34aed1c"}, 
                     {'new': {'S' :'hello'}})                                 

dynoConn.update_item("influencer_data", 
                     {"HashKeyElement": "9f08b4f5-d25a-4950-a948-0381c34aed1c"},
                     {"AttributesToPut" : {"new": {"S" :"hello"}}})      

They all produce the same error:

  File "/usr/local/lib/python2.6/dist-packages/boto-2.3.0-py2.6.egg/boto/dynamodb/layer1.py", line 164, in _retry_handler
    data)
boto.exception.DynamoDBResponseError: DynamoDBResponseError: 400 Bad Request
{u'Message': u'Expected null', u'__type': u'com.amazon.coral.service#SerializationException'}

I also investigated the API docs here but they were pretty spartan.

I have done a lot of searching and fiddling, and the only thing I have left is to use the PHP API and dive into the code to find where it "formats" the JSON body, but that is a bit of a pain. Please save me from that pain!

Answer 1

Sorry, I misunderstood what you were looking for. You can accomplish this via layer2 although there is a small bug that needs to be addressed. Here's some Layer2 code:

>>> import boto
>>> c = boto.connect_dynamodb()
>>> t = c.get_table('counter')
>>> item = t.get_item('counter')
>>> item
{u'id': 'counter', u'n': 1}
>>> item.add_attribute('n', 20)
>>> item.save()
{u'ConsumedCapacityUnits': 1.0}
>>> item  # Here's the bug, local Item is not updated
{u'id': 'counter', u'n': 1}
>>> item = t.get_item('counter')  # Refetch item just to verify change occurred
>>> item
{u'id': 'counter', u'n': 21}

This results in the same over-the-wire request as you are performing in your Layer1 code, as shown by the following debug output.

2012-04-27 04:17:59,170 foo [DEBUG]:StringToSign:
POST
/

host:dynamodb.us-east-1.amazonaws.com
x-amz-date:Fri, 27 Apr 2012 11:17:59 GMT
x-amz-security-    token:<removed> ==
x-amz-target:DynamoDB_20111205.UpdateItem

{"AttributeUpdates": {"n": {"Action": "ADD", "Value": {"N": "20"}}}, "TableName": "counter", "Key": {"HashKeyElement": {"S": "counter"}}}

If you want to avoid the initial GetItem call, you could do this instead:

>>> import boto
>>> c = boto.connect_dynamodb()
>>> t = c.get_table('counter')
>>> item = t.new_item('counter')
>>> item.add_attribute('n', 20)
>>> item.save()
{u'ConsumedCapacityUnits': 1.0}

Which will update the item if it already exists or create it if it doesn't yet exist.

Answer 2

For those looking for the answer I have found it. First IMPORTANT NOTE, I am currently unaware of what is going on BUT for the moment, to get a layer1 instance I have had to do the following:

import boto
AWS_ACCESS_KEY=XXXXX
AWS_SECRET_KEY=YYYYY
dynoConn = boto.connect_dynamodb(AWS_ACCESS_KEY, AWS_SECRET_KEY)
dynoConnLayer1 = boto.dynamodb.layer1.Layer1(AWS_ACCESS_KEY, AWS_SECRET_KEY) 

Essentially instantiating a layer2 FIRST and THEN a layer 1. Maybe Im doing something stupid but at this point Im just happy to have it working.... I'll sort the details later. THEN...to actually do the atomic update call:

dynoConnLayer1.update_item("influencer_data", 
                    {"HashKeyElement":{"S":"9f08b4f5-d25a-4950-a948-0381c34aed1c"}},
                    {"direct_influence":
                        {"Action":"ADD","Value":{"N":"20"}}
                    }
                );

Note in the example above Dynamo will ADD 20 to what ever the current value is and this operation will be atomic meaning other operations happening at the "same time" will be correctly "scheduled" to happen after the new value has been established as +20 OR before this operation is executed. Either way the desired effect will be accomplished.

Be certain to do this on the instance of the layer1 connection as the layer2 will throw errors given it expects a different set of parameter types.

Thats all there is to it!!!! Just so folks know, I figured this out using the PHP SDK. Takes a very short time to install and set up AND THEN when you do a call, the debug data will actually show you the format of the HTTP request body so you will be able to copy/model your layer1 parameters after the example. Here is the code I used to do the atomic update in PHP:

<?php 
    // Instantiate the class
    $dynamodb = new AmazonDynamoDB();

    $update_response = $dynamodb->update_item(array(
        'TableName' => 'influencer_data',
            'Key' => array(
                'HashKeyElement' => array(
                    AmazonDynamoDB::TYPE_STRING=> '9f08b4f5-d25a-4950-a948-0381c34aed1c'
                )
            ),
            'AttributeUpdates' => array(
                'direct_influence' => array(
                    'Action' => AmazonDynamoDB::ACTION_ADD,
                    'Value' => array(
                        AmazonDynamoDB::TYPE_NUMBER => '20'
                    )
                )
            )
    ));

    // status code 200 indicates success
    print_r($update_response);

?>

Hopefully this will help other up until the Boto layer2 interface catches up...or someone simply figures out how to do it in level2 :-)

Answer 3

I'm not sure this is truly an atomic counter, since when you increment the value of 1, another call call could increment the number by 1, so that when you "get" the value, it is not the value that you would expect.

For instance, putting the code by garnaat, which is marked as the accepted answer, I see that when you put it in a thread, it does not work:

class ThreadClass(threading.Thread):
    def run(self):
        conn = boto.dynamodb.connect_to_region(aws_access_key_id=os.environ['AWS_ACCESS_KEY'], aws_secret_access_key=os.environ['AWS_SECRET_KEY'], region_name='us-east-1')
        t = conn.get_table('zoo_keeper_ids')
        item = t.new_item('counter')
        item.add_attribute('n', 1)
        r = item.save() #- Item has been atomically updated!
        # Uh-Oh! The value may have changed by the time "get_item" is called!
        item = t.get_item('counter') 
        self.counter = item['n']
        logging.critical('Thread has counter: ' + str(self.counter))

tcount = 3
threads = []
for i in range(tcount):
    threads.append(ThreadClass())

# Start running the threads:
for t in threads:
    t.start()

# Wait for all threads to complete:
for t in threads:
    t.join()

#- Now verify all threads have unique numbers:
results = set()
for t in threads:
    results.add(t.counter)

print len(results)
print tcount
if len(results) != tcount:
    print '***Error: All threads do not have unique values!'
else:
    print 'Success!  All threads have unique values!'

Note: If you want this to truly work, change the code to this:

def run(self):
    conn = boto.dynamodb.connect_to_region(aws_access_key_id=os.environ['AWS_ACCESS_KEY'], aws_secret_access_key=os.environ['AWS_SECRET_KEY'], region_name='us-east-1')
    t = conn.get_table('zoo_keeper_ids')
    item = t.new_item('counter')
    item.add_attribute('n', 1)
    r = item.save(return_values='ALL_NEW') #- Item has been atomically updated, and you have the correct value without having to do a "get"!
    self.counter = str(r['Attributes']['n'])
    logging.critical('Thread has counter: ' + str(self.counter))

Hope this helps!

Answer 4

There is no high-level function in DynamoDB for atomic counters. However, you can implement an atomic counter using the conditional write feature. For example, let's say you a table with an string hash key called like this.

>>> import boto
>>> c = boto.connect_dynamodb()
>>> schema = s.create_schema('id', 's')
>>> counter_table = c.create_table('counter', schema, 5, 5)

You now write an item to that table that includes an attribute called 'n' whose value is zero.

>>> n = 0
>>> item = counter_table.new_item('counter', {'n': n})
>>> item.put()

Now, if I want to update the value of my counter, I would perform a conditional write operation that will bump the value of 'n' to 1 iff it's current value agrees with my idea of it's current value.

>>> n += 1
>>> item['n'] = n
>>> item.put(expected_value={'n': n-1})

This will set the value of 'n' in the item to 1 but only if the current value in the DynamoDB is zero. If the value was already incremented by someone else, the write would fail and I would then need to increment by local counter and try again.

This is kind of complicated but all of this could be wrapped up in some code to make it much simpler to use. I did a similar thing for SimpleDB that you can find here:

http://www.elastician.com/2010/02/stupid-boto-tricks-2-reliable-counters.html

I should probably try to update that example to use DynamoDB

Answer 5

You want to increment a value in dynamodb then you can achieve this by using:

import boto3
import json
import decimal

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            if o % 1 > 0:
                return float(o)
            else:
                return int(o)
        return super(DecimalEncoder, self).default(o)

ddb = boto3.resource('dynamodb') 
def get_counter():
    table = ddb.Table(TableName)
    try:
            response = table.update_item(                                                             
            Key={
                'haskey' : 'counterName'
            },
            UpdateExpression="set currentValue = currentValue +  :val",
            ExpressionAttributeValues={
                ':val': decimal.Decimal(1)
            }, 
            ReturnValues="UPDATED_NEW"
        )
        print("UpdateItem succeeded:")
    except Exception as e:
        raise e
    print(response["Attributes"]["currentValue" ])

This implementetion needs a extra counter table that will just keep the last used value for you.