Extract data from external website via WebAPI C# ASP.Net MVC

Question

I'm most of the time working in C# WinForm and ASP.Net MVC non-WebAPI... but now I'd like to accomplish a WebAPI task given to me.

I'm actually new to Web API, I practiced for few weeks video like this one and read up few books to understand it; I also have a solid background in.Net network programming in HttpWebquest/HttpClient but never known that WebAPI also use it until I read that in webapi book weeks ago.

Please I need help to achieve something like the below:

Example

Both Website A & B are developed in ASP.Net MVC, and I am the developer for website A.

Website A (www.mysite.com)

Website B ( https://www.cars.com/research/suv/ )

website A needs to extract data from specific page of website B thru WebAPI.

Let assume the page to extract data from website B is ( https://www.cars.com/research/suv/ ), and I need to extract everything from that page (image + title + description + text + details for each items[vehicle] )...

Developers for website B gave me a list of API endpoints + ApiSecretKey and api direct link... and told me that all requests should be called via GET;

and data extracted from website B, should be placed in website A in (www.mysite.com/pageVehicule).

Please how do I accomplish it?

Answer 1

Assuming website B exposes a WebAPI, you will have 2 options.

1) Fetch the data from the server side of your MVC project

Since you claim to have experience with the HttpClient this would be pretty straight forward (example pending).

The steps to take are the following; when loading your view:

• make a call to the web api (most likely from your controller) and fetch the data
• add this data to your viewmodel
• render the data on your page

example: taken from: https://docs.microsoft.com/en-us/aspnet/web-api/overview/advanced/calling-a-web-api-from-a-net-client

Note: this is not the best example: when using this approach please do some more research

static HttpClient client = new HttpClient();
static async Task<Product> GetProductAsync(string path)
{   
    Product product = null;
    HttpResponseMessage response = await client.GetAsync(path);
    if (response.IsSuccessStatusCode)
    {
         var jsonString = await response.Content.ReadAsStringAsync();
         model = JsonConvert.DeserializeObject<YOUROBJECTTYPE>(jsonString);
    }
    return product;
}

One of the benefits here is that authentication is often simpler. The downside is that your page load will be slower.

2) Fetch the data client side ie: from the browser

Through javascript you will be able to fetch the data. Possibly with jquery or other frameworks. Normally you will load your page first, and then fetch the data.

The steps to take are the following;

• call the api from javascript and fetch the data
• manipulate the page to display the data. This is best acieved by jquery , react , angular , vue.js , knockout or a similar framework.

jQuery example:

$.ajax({  
    type: 'GET',  
    url: 'api/Employees',  
    dataType: 'json',  
    success: function (data) {  
       //use the data here
 }});

The page will load fast. After the page is loaded the call to the api is made. So the page is responsive while loading data on the background.

The down side here, is that if you need authentication, you cannot store the credentials on the clientside (eg; javascript) so you need a better authentication flow. Perhaps cookies or jwt tokens.

Answer 2

There seems to be a total lack of understanding of that an API actually is.

Forget about Website B, your job is to get your data from the API. Website B uses the same API, more than likely, to populate its own pages.

Your job is to find out which endpoints give you the data you need and to be fair the API should have some documentation or a Swagger page that shows you how to call it.

So:

• identify the endpoints you need
• learn how to call the endpoints ( talk to the devs if you need help, they should be able to give you some examples)
• use the data fromm the endpoints to populate what you need.

Answer 3

Source: https://www.stevejgordon.co.uk/introduction-to-httpclientfactory-aspnetcore

[Route("api/[controller]")] public class ValuesController: Controller { private readonly IHttpClientFactory _httpClientFactory; public ValuesController(IHttpClientFactory httpClientFactory) { _httpClientFactory = httpClientFactory; }

[HttpGet]
public async Task<ActionResult> Get()
{
    var client = _httpClientFactory.CreateClient();
    var result = await client.GetStringAsync("www.bWebSiteUrl.com/api/???");
    return Ok(result);
}

}