Amazon Web Services S3中的线程安全文件重命名

Question

I need to move rename an object in AWS S3 storage.

All the solutions I have found require a copy followed by a delete. However, this leaves a short time where both files exists, which I do not believe would be thread safe.

Is there a way to do this in a thread safe manner?

The code is in Scala using the Java AWS SDK.

EDIT: Rob, thanks for the reply, I believe I understand that code is doing, but makes me think I asked the wrong question.

Rather than specific AWS functionality, let me describe it in terms of what I am trying to accomplish.

I have an S3 directory that is regularly receiving files from an outside source. I have multiple processes that need to 'process' those files, and each file should only be processed once.

In the past as a cheap way of handling this, I've used a rename to either move the file or mark it as processing. If the rename succeeded, then the process knew it 'owned' the file and would continue processing. If it failed because the source file did not exist, then it would try the next file in the directory.

What I am needing is a way, preferably S3 only, that will allow multiple processes to work on the files, while ensuring that each file is only processed once.

In your solutions below, since 'find' and 'delete' are separate methods, and delete does not fail if the file does not exist, I'm not sure I see how the two processes can't simply (in worst case scenario) both complete in lockstep with the other.

File moving may be the wrong solution, and my inexperience with AWS preventing me from seeing a better way to accomplish this task.

Answer 1

In the past as a cheap way of handling this, I've used a rename to either move the file or mark it as processing. If the rename succeeded, then the process knew it 'owned' the file and would continue processing. If it failed because the source file did not exist, then it would try the next file in the directory.

Let me start by pointing out that this technique of using an atomic rename for a thread to acquire exclusive access to process a file works, but it does risk leaving a file unprocessed. Imagine what happens if the thread (or whole server) dies right after the rename. Without a robust way to keep track of which files are not yet complete and a way to retry them, your system will not be very resilient.

As you note, S3 does not have an atomic rename operation, so your usual technique doesn't work as you desire.

S3 has a nice "notification" feature that can be configured. In your case, you probably want to get notified when a file is created. Notifications can be delivered to SNS, SQS or Lambda. You probably want either SQS or Lambda. With SQS, a message gets added to a queue, which you can have a thread grab and process the file. The SQS model guarantees delivery "at least once" and will retry delivery until the message is deleted (or ages out of the queue). The redeliver-if-not-deleted time is configurable. Note that it is possible for SQS to deliver the same message multiple times - they err on the side of over-delivering rather than not delivering a message. If it is ok to double-process a file on a very infrequent basis, then this probably works fine for you. We make extensive use of SQS queues and are happy.

I am not familiar with the detailed semantics of the Lambda message processing.

I suggest that you google "S3 Event Notifications" for more details.

Original answer to the original question:

I am not sure the issue is "thread safety" - perhaps more "transactional integrity"?

In any case, you are correct that doing an S3 "atomic" rename is not obvious. I think you have to "pick your poison" - either you have to deal with the fact that 1) you have the old and new copies at the same time or 2) you have a period of time where you have neither the old nor the new copy.

In either case, a key issue you need to deal with is persisting the fact that you are doing the rename (until the rename is confirmed to be complete). If you have a row in some database that represents the file, then you can persist the state there. The following assumes that you don't want to use anything other than S3 to persist state.

You are going to actually copy the file twice, using a temporary folder for the intermediate copy. You can have separate threads doing each step (looking for files to work on), or a single thread that checks the various conditions and does the remaining steps. In other words, you need to look for renames that were partially done (but that thread failed to finish) and pick up where it left off.

For this example, we are going to rename from A to B and use a temporary folder called tmp.

If you prefer briefly having both copies:

1. Copy A to tmp/A-B (the file name has before and after names in it).
2. Finding tmp/A-B: copy it to B.
3. Finding tmp/A-B, A and B: delete A.
4. Finding tmp/A-B, A is missing and B exists: delete tmp/A-B.

If you prefer briefly having neither copy:

1. Copy A to tmp/A-B.
2. Finding tmp/A-B and A: delete A.
3. Finding tmp/A-B and A is missing and B is missing: copy tmp/A-B to B.
4. Finding tmp/A-B and A is missing and B exists: delete tmp/A-B.

Answer 2

不，S3 API不支持