Localhost Endpoint to DynamoDB Local with Boto3

Question

Although Amazon provides documentation regarding how to connect to dynamoDB local with Java, PHP and .Net, there is no description of how to connect to localhost:8000 using Python. Existing documentation on the web points to the use of the DynamoDBConnection method inside boto.dynamodb2.layer1, but this creates an incompatibility between live and test environments that use the boto3 protocol to manage dynamoDB.

In boto3, you can make a request to dynamo using the following constructor and variables set into the environment:

client = boto3.client('dynamodb')
table = client.list_tables()

Whereas the boto.dynamodb2.layer1 package requires you to construct the following:

client = DynamoDBConnection(
    host='localhost',
    port=8000,
    aws_access_key_id='anything',
    aws_secret_access_key='anything',
    is_secure=False)
table = client.list_tables()

Although it is possible to create logic which determines the proper constructor based upon the local environment, I am wary of building a set of methods which treat each constructor as the same. Instead, I would prefer to use boto3 for everything and to be able to set the endpoint for dynamoDB in the environmental variables. Unfortunately, that option does not appear to be currently be available.

Is there any way to use boto3 to define a dynamoDB local endpoint (like the other languages)? Or any chance that Amazon will plan to support this feature?

Answer 1

It does support DynamoDB Local. You just need to set the appropriate endpoint such as you can do with other language SDKs

Here is a code snippet of how you can use boto3's client and resource interface via DynamoDB Local:

import boto3

# For a Boto3 client.
ddb = boto3.client('dynamodb', endpoint_url='http://localhost:8000')
response = ddb.list_tables()
print(response)

# For a Boto3 service resource
ddb = boto3.resource('dynamodb', endpoint_url='http://localhost:8000')
print(list(ddb.tables.all()))

Answer 2

Note: You will want to extend the above response to include region. I have appended to Kyle's code above. If your initial attempt is greeted with a region error, this will return the appropriate '[]' response.

import boto3

## For a Boto3 client ('client' is for low-level access to Dynamo service API)
ddb1 = boto3.client('dynamodb', endpoint_url='http://localhost:8000', region_name='us-west-2')
response = ddb1.list_tables()
print(response)

# For a Boto3 service resource ('resource' is for higher-level, abstracted access to Dynamo)
ddb2 = boto3.resource('dynamodb', endpoint_url='http://localhost:8000', region_name='us-west-2')
print(list(ddb2.tables.all()))

Answer 3

This is the tutorial python DynamoDb. It depicts how to connect to local instance.

http://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/GettingStarted.Python.01.html

It seems like the minimum required parameters are the following with the help of aws configuration (below).

dynamodb = boto3.resource('dynamodb', endpoint_url='http://localhost:8000/')

The region, access key and secret key parameters can be omitted when configure profile parameters using aws configure command (require install aws cli). However, you can create aws configuration files manually in your home (in case, you don't want to use aws cli).

file ~/.aws/config

[default]
output = json
region = anywhere

file ~/.aws/credentials

[default]
aws_access_key_id = whatever_id
aws_secret_access_key = whatever_key 

You can consult the aws configuration in http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

Note in the local DynamoDb development region , aws_access_key_id and aws_secret_access_key values in those files can be anything. But if you want to use aws cli with the AWS then you must put the valid region, valid id and keys. They are available when you register to the AWS services.

More information, when you call

db = boto3.client('dynamodb')

The host the boto3 connect will base on the region parameter eg region=us-west-1 when call above api, it will connect to dynamodb.us-west-1.amazonaws.com . However, when pass parameter endpoint_url the region will not be used. For more AWS endpoint list please go to http://docs.aws.amazon.com/general/latest/gr/rande.html .

Answer 4

use dummy access key and id otherwise it will throw an exception on running the methods.

 import boto3 dynamodb = boto3.session('dynamodb', aws_access_key_id="anything", aws_secret_access_key="anything", region_name="us-west-2", endpoint_url="http://localhost:8000") 

Answer 5

Here is the boto3 Config Reference (API):

https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

    def resource(self, service_name, region_name=None, api_version=None,
                 use_ssl=True, verify=None, endpoint_url=None,
                 aws_access_key_id=None, aws_secret_access_key=None,
                 aws_session_token=None, config=None):
        """
        Create a resource service client by name.

        :type service_name: string
        :param service_name: The name of a service, e.g. 's3' or 'ec2'. You
            can get a list of available services via
            :py:meth:`get_available_resources`.

        :type region_name: string
        :param region_name: The name of the region associated with the client.
            A client is associated with a single region.

        :type api_version: string
        :param api_version: The API version to use.  By default, botocore will
            use the latest API version when creating a client.  You only need
            to specify this parameter if you want to use a previous API version
            of the client.

        :type use_ssl: boolean
        :param use_ssl: Whether or not to use SSL.  By default, SSL is used.
            Note that not all services support non-ssl connections.

        :type verify: boolean/string
        :param verify: Whether or not to verify SSL certificates.  By default
            SSL certificates are verified.  You can provide the following
            values:

            * False - do not validate SSL certificates.  SSL will still be
              used (unless use_ssl is False), but SSL certificates
              will not be verified.
            * path/to/cert/bundle.pem - A filename of the CA cert bundle to
              uses.  You can specify this argument if you want to use a
              different CA cert bundle than the one used by botocore.

        :type endpoint_url: string
        :param endpoint_url: The complete URL to use for the constructed
            client. Normally, botocore will automatically construct the
            appropriate URL to use when communicating with a service.  You
            can specify a complete URL (including the "http/https" scheme)
            to override this behavior.  If this value is provided,
            then ``use_ssl`` is ignored.

        :type aws_access_key_id: string
        :param aws_access_key_id: The access key to use when creating
            the client.  This is entirely optional, and if not provided,
            the credentials configured for the session will automatically
            be used.  You only need to provide this argument if you want
            to override the credentials used for this specific client.

        :type aws_secret_access_key: string
        :param aws_secret_access_key: The secret key to use when creating
            the client.  Same semantics as aws_access_key_id above.

        :type aws_session_token: string
        :param aws_session_token: The session token to use when creating
            the client.  Same semantics as aws_access_key_id above.

        :type config: botocore.client.Config
        :param config: Advanced client configuration options. If region_name
            is specified in the client config, its value will take precedence
            over environment variables and configuration values, but not over
            a region_name value passed explicitly to the method.  If
            user_agent_extra is specified in the client config, it overrides
            the default user_agent_extra provided by the resource API. See
            `botocore config documentation
            <https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html>`_
            for more details.

        :return: Subclass of :py:class:`~boto3.resources.base.ServiceResource`
        """