Apache Camel：无法删除文件和锁定

Question

I am using the Apache Camel File component to read from a local directory and upload to an AWS S3 bucket. This route has worked flawlessly in the past, but is having issues with certain files now-a-days.

In my debug investigations thus far (3 days of misery), I have found that the route reaches the DelegateSyncProcessor class and does not have any exception set on the exchange (for reference see the relevant code of the previously mentioned class below). Considering there aren't any exceptions, I can't seem to figure out why the onCompletion() methods never fire for specific files. There weren't/aren't any exceptions for problematic files; however, the file and the lock continue to exist after all the logic has executed (including the final transfer of the message to the .to() endpoint) hinting that there is some internal Camel issue. I suspect this because the route behaves normally in every aspect except the file and lock deletion.

After enabling debug logs for Camel, I was frustrated to see that there were no logs related to an error during the route execution. I would love to hear any advice about what could be happening under the hood.

A few extra things to note:

• I am running camel 2.16.0
• No easily detectable issues were seen while debugging through the camel code.
• The logic in the .process() sections runs all the way through (some exceptions occurred, but they are properly handled.

---

Update:

More debugging with debug logs enabled for camel yielded no results. I have only found that the logic within my .process() code is the issue. When the code takes too long to execute (even without exceptions), the file deletion fails.

---

Update 2:

I found where the camel route actually falls apart. The CamelEventLogger.java class completely fails while trying to "logEvent". The related code to that class is added below. When the code reaches matcher.find() , it times out.

MyRouteClass.java:

from(importProcessingEndpoint)
        .convertBodyTo(byte[].class)
        .process((exchange) -> {
           // some logic here
        })
        .to(outgoingEndpoint)
        .threads(MAX_NUMBER_OF_CAMEL_THREADS)
        .process((exchange) -> {
           // some logic here
        })
        .log("Finished processing import file.")
        .to(outgoingEndpoint);

DelegateSyncProcessor.java:

@Override
public boolean process(Exchange exchange, AsyncCallback callback) {
    // force calling the sync method
    try {
        processor.process(exchange);
    } catch (Throwable e) {
        // must catch throwable so we catch all
        exchange.setException(e);
    } finally {
        // we are bridging a sync processor as async so callback with true
        callback.done(true);
    }
    return true;
}

CamelEventLogger.java

private void logEvent(final String label, final Exchange exchange, final Endpoint endpoint,
            final long elapsedTime, final boolean logTID) {
        Matcher matcher = PATTERN.matcher(extractMessage(exchange));
        if (matcher.find()) {
            //CHECKSTYLE:OFF
            String evtType = matcher.group(1);
            String evtName = matcher.group(2);
            String guid = matcher.group(3);
            String tid = matcher.group(4);
            //CHECKSTYLE:ON
            if (tid == null || !logTID) {
                tid = "";
            } else {
                tid = "intuit_tid=" + tid;
            }
            log.info(LOG_FORMAT, label, exchange.getExchangeId(), guid, evtName, evtType, endpoint, elapsedTime, tid);
        } else { // the message is not parseable, fall back to minimum log entry
            log.info("Event {} id={} {} elapsedTimeMs={}", label, exchange.getExchangeId(), endpoint, elapsedTime);
        }
    }

Answer 1

I was initially surprised that this wasn't a common problem with Camel, but after fixing it, I've come to realize that it is definitely a user error (albeit, an interesting one).

Having said that the problem is uncommon, Ill give my solution (hopefully it has some value to someone).

As mentioned above, my camel route has hanging and failing to delete the locking file and the message (which was a file in this case). FOR THE PROBLEMATIC FILE MESSAGES (I want to highlight this because it was an intermittent issue), the debugger seemed to bring me down a road to where the thread started to hang. Here I realized that I was brought to a custom Camel Logging class. The logging class would fail based on different inputs.

How would my situation be helpful to anyone else? I'll describe a strategy for debugging when you get to situations like these.

If your camel route is behaving similarly, find a use case in which the route hangs, pause your debugger (after it has started hanging) and take a thread dump. It is within this thread dump and stage within the paused debugger where you can back track a bit. For me, I noticed that 10 or so frames/steps ago, I reached an odd class. I set a debug point there and could see that it was problematic.