How to implement MapReduce in C# using PLINQ?
Suppose, you have 7-8 WebServices to collect data and on each receiving (async manner) you have to put that data into some tables of a database, in my case it is SQL Server 2008. For instance the data you are getting from each Web Service is:
<employees>
<employee>
<name>Ramiz</name>
</employee>
<employee>
<name>Aamir</name>
</employee>
<employee>
<name>Zubair</name>
</employee>
</employees>
And, on each receiving of response this data goes into a table name - Employee:
Employee
===
EmployeeID (PK)
EmployeeName
Once the data goes into table, it has to return as json
to the client which is ASP.NET (MVC 3) application is making this call using client-side JavaScript (ajax).
Suppose, a WebServiceEmployee1 has returned with data and other 6 are in queue (still trying to get the data). Then, it should goes register the resultset into table instead of waiting other 6 and return data of inserted employee to client in json. And, keep it connected and doing while others do the same way.
Please see, in my toolbelt I have ASP.NET MVC 3 (Razor), SQL SERVER 2008 R2, jQuery.
Thanks.
Without clear understanding on the processing you want to perform, I suggest reading though the following MSDN documentation: http://bit.ly/Ir7Nvk It describes map/reduce with PLINQ with examples like:
public IDMultisetItemList PotentialFriendsPLinq(SubscriberID id,
int maxCandidates)
{
var candidates =
subscribers[id].Friends.AsParallel()
.SelectMany(friend => subscribers[friend].Friends)
.Where(foaf => foaf != id &&
!(subscribers[id].Friends.Contains(foaf)))
.GroupBy(foaf => foaf)
.Select(foafGroup => new IDMultisetItem(foafGroup.Key,
foafGroup.Count()));
return Multiset.MostNumerous(candidates, maxCandidates);
}
The "map" being Friends.AsParallel
, SelectMany
, and Where
and the "reduce" phase is the GroupBy
and Select
This PDF from MS has a Map Reduce example towards the middle/end
public static IEnumerable<TResult> MapReduce<TSource, TMapped, TKey, TResult>(
this IEnumerable<TSource> source,
Patterns of Parallel Programming Page 76
Func<TSource, IEnumerable<TMapped>> map,
Func<TMapped, TKey> keySelector,
Func<IGrouping<TKey, TMapped>, IEnumerable<TResult>> reduce)
{
return source.SelectMany(map)
.GroupBy(keySelector)
.SelectMany(reduce);
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.