如何使用OpenStack的Swift将大型SQLite文件上传到对象存储（通过Python）

Question

I have an Object Storage instance on IBM's Bluemix, and I'm attempting to upload a ~32GB SQLite file. Here is my Python code which uses the OpenStack Swift API (with the credentials removed):

import swiftclient
conn = swiftclient.Connection(key="pw",authurl="url",auth_version='3',os_options={"project_id": "project_id","user_id": "user_id","region_name": "region"})
container_name = 'containerName'
file_name = 'file.sqlite'
with open(file_name, 'rb') as sqlite_file:
    conn.put_object(container_name,file_name,sqlite_file)

I tested this code with a small .html file and it uploaded without a problem. When I changed the file to the SQLite file, it ran for > 5 hours and eventually gave a "requests.exceptions.ConnectionError: [Errno 32] Broken pipe" error. What am I doing wrong?

Answer 1

You'll need to read-up on Swift DLO/SLO support and manifests. Here's a blog post that might help with context on what manifests are and the differences between Static Large Object and Dynamic Large Object support.

Basically, I'd recommend the following approach:

1. Download/install the Python-SwiftClient binary
2. Use its upload command in conjunction with your object storage credentials from the Bluemix service. In the manifest article above, it discusses this approach here . Take note of the upload command, the use of the --use-slo flag and the ability to define the size of the concatenated segments generated. Roughly, the invocation will look like this:

$ swift --os-auth-url=https://identity.open.softlayer.com/v3 --os-user- id=some_hex_value --os-password="weird_characters" --os-project-id=another_hex_value --os-region-name=dallas -V 3 upload my_object_storage_container_name -S int_seg_size_in_bytes my_local_large_file_with_some_extension --use-slo

my_local_large_file_with_some_extension segment 3
my_local_large_file_with_some_extension segment 1
my_local_large_file_with_some_extension segment 2
my_local_large_file_with_some_extension segment 0
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000002
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000003
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000001
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000000
my_local_large_file_with_some_extension

Good luck.