通过REST API批量上传

Question

I have the goal of uploading a Products CSV of ~3000 records to my e-commerce site. I want to utilise the REST API that my e-comm platform provides so I have something I can re-use and build upon for future sites that I may create.

My main issue that I am having trouble working through is: - System.Threading.ThreadAbortException

Which I can only attribute to how long it takes to process through all 3K records via a POST request. My code:

public ActionResult WriteProductsFromFile()
    {
        string fileNameIN = "19107.txt";
        string fileNameOUT = "19107_output.txt";
        string jsonUrl = $"/api/products";

        List<string> ls = new List<string>();

        var engine = new FileHelperAsyncEngine<Prod1>();

        using (engine.BeginReadFile(fileNameIN))
        {
            foreach (Prod1 prod in engine)
            {
                outputProduct output = new outputProduct();

                if (!string.IsNullOrEmpty(prod.name))
                {
                    output.product.name = prod.name;
                    string productJson = JsonConvert.SerializeObject(output);

                    ls.Add(productJson);
                }
            }
        }

        foreach (String s in ls)
            nopApiClient.Post(jsonUrl, s);

        return RedirectToAction("GetProducts");
    }
}

Since I'm new to web-coding, am I going about this the wrong way? Is there a preferred way to bulk-upload that I haven't come across?

I've attempted to use the TaskCreationOptions.LongRunning flag, which helps the cause slightly but doesn't get me anywhere near my goal.

Answer 1

Web and api controller actions are not meant to do long running tasks - besides locking up the UI/thread, you will be introducing a series of opportunities for failure that you will have little recourse in recovering from.

But it's not all bad you have a lot of options here, there is a lot of literature on async/cloud architecture - which explains how to deal with files and these sorts of scenarios.

What you want to do is disconnect the processing of your file from the API request (in your application not the 3rd party)

It will take a little more work but will ultimately create a more reliable application.

Step 1:

Drop the file immediately to disk - I see you have the file on DISK already not sure how it gets there but either way it will work out the same.

Step 2:

Use a process running as
- a console app (easiest)
- a service (requires some sort of install/uninstall of the service)
- or even a thread in your web app (but you will struggle to know when it fails)

Which ever way you choose, the process will watch a directory for file changes, when there is a change it will kick off your method to happily process the file as you like.

Check out the FileSystemWatchers here is a basic example: https://www.dotnetperls.com/filesystemwatcher

Additionally:

• If you are interested in running a thread in your Api/Web app, take a look at https://www.hanselman.com/blog/HowToRunBackgroundTasksInASPNET.aspx for some options.
• You don't have to use a FileSystemWatcher of course, you could trigger via a flag in a DB - that is being checked periodically, or a system event.