Learn Parallel Programming

Introduction

Parallel programming splits the work into independent chunks of work and then carries out these works simultaneously. All these works are being processed at the same time. It is related to concurrent programming. It decomposes a task into different sub-tasks and then allocates each sub-task to a processor for the execution. There are different forms of parallel computations like bit-level, instruction-level, data and task parallelism.

Now, I will mention some points about “Data Parallelism” and “Task Parallelism” to clarify the concept of two different parallelisms. The following are the points:

• Data Parallelism
  
  It focuses on the distribution of data across different computing nodes. It splits the data into different pieces and they are independent to each other. It allocates those pieces to separate threads for processing. It is like a Structured Parallelism that means parallel work units start and finish in the same place.
  
  
• Task Parallelism
  
  It divides tasks and allocates those tasks to separate threads for processing. It is like Unstructured Parallelism. It means parallel work units may start and finish in places scattered across the program.

Now, I will only focus on the concept of Data Parallelism.

Data Parallelism

It performs the same operations like processing of elements in collection or array concurrently. It is supported by namespace: System.Threading.Tasks.Parallel class.

The following are the different ways to perform Data Parallelism:

Basic way to do Data Parallelism

Operations by simple loop without thread:

#region Data Operation with foreach loop

public void DataOperationWithForeachLoop()
{
    var mySource = Enumerable.Range(0, 1000).ToList();
    foreach (var item in mySource)
    {
        Console.WriteLine("Square root of {0} is {1}", item, item * item);
    }
}

#endregion

Operation by Data Parallelism with Parallel.Foreach

Some basic concepts about Parallel.Foreach:

 

1. The Foreach loop iterates over all the elements of a sequence in parallel.
   
2. The following are its options:
   
   
   1. Sequence
      
      
   2. Options: It is optional for controlling the parallelism.
      
      
   3. Delegate instantiating the result value. The following code implements Data Parallelism. It will partition the data of collection and then loop will execute on the different parts of data in asynchronous mode.
      

      #region Data Operation with Data Parallelism
      
      public void DataOperationWithDataParallelism()
      {
          var mySource = Enumerable.Range(0, 1000).ToList();
          Parallel.ForEach(mySource, values => CalculateMyOperation(values));
      }
      
      private void CalculateMyOperation(int values)
      {
          Console.WriteLine("Square root of {0} is {1}", values, values * values);
      }
      
      #endregion

Data Parallelism by PLINQ

PLINQ means Parallel LINQ. It provides a way to run LINQ queries in Parallel way. It is a new Parallel Extension that has been added to .NET 4.0. 

The following is the PLINQ code sample:

   #region Data Operation with PLINQ

   public void DataOperationByPLINQ()
   {
       long mySum = Enumerable.Range(1, 10000).AsParallel().Sum();
       Console.WriteLine("Total: {0}", mySum);
   }

#endregion

More,

public void ShowEvenNumbersByPLINQ()
{
    var numers = Enumerable.Range(1, 10000);
    var evenNums = from number in numers.AsParallel()
                   where number % 2 == 0
                   select number;

    Console.WriteLine("Even Counts :{0} :", evenNums.Count());
}

Execute several methods in Parallel

Parallel.Invoke() method executes several methods in Parallel.

Parallel.Invoke(
() => Method1(mycollection),
() => Method2(myCollection1, MyCollection2),
() => Method3(mycollection));

MaxDegreeOfParallelism

MaxDegreeOfParallelism limits the number of concurrent operations to the set value. By default, For and Foreach uses many threads, so MaxDegreeOfParallelism will limit the concurrent tasks.

If we set -1 then there is no limit on the number of concurrent operations.

Example

var myOptions = new ParallelOptions() {
   MaxDegreeOfParallelism = 2 };
   Parallel.For(0, n, options, i =>
   {
      data[i] = Calculate(i);
   });
}

When should we use MaxDegreeOfParallelism?

 

• There is some code or algorithm where you don’t want to use all of the available processors. Some algorithms become inefficient using lots of concurrent threads. So, you can avoid wasting cycles on additional cores.
  
  
• If you are running multiple algorithms or codes and you want to specify how much system can utilize each system.

Scenario for the use of MaxDegreeOfParallelism

Problem Statement

User wants to download some data or .pdf from some website, but user’s bandwidth is limited, sohe/she can download only some page at a time. User wants to download it asynchronously by using Parallel.Foreach but it executes whole list of desired .pdf data.

So, he/she wants to limit thread number.

Solution

So, MaxDegreeOfParallelism can be used in this case with Parallel.Foreach like the following code snippet:

Parallel.ForEach(
    ListOfOperationsToBeDownloaded,
    new ParallelOptions { MaxDegreeOfParallelism = 4 },
    operation => { Download(operation); }
);

Terminate Parallel loop

Parallel loop can be terminated. Execution of processor can be stopped by terminating the Parallel loop.

Parallel loops have the following two ways to break or stop or terminate loop:

loopState.Break()

It will allow other threads to terminate or break later if those are busy. It let them complete the processing of all prior elements before terminating the loop.

Example:

var mySource = Enumerable.Range(0, 1000).ToList();
int data = 0;
Parallel.ForEach(
    mySource,
    (i, state) =>
    {
        data += i;
        if (data > 100)
        {
            state.Break();
            Console.WriteLine("Break called iteration {0}. data = {1} ", i, data);
        }
    });
Console.WriteLine("Break called data = {0} ", data);
Console.ReadKey();

loopState.Break()

It terminates the loop without allowing any new step to begin.

Example:

int data = 0;
Parallel.ForEach(
    mySource,
    (i, state) =>
    {
        data += i;
        if (data > 100)
        {
            state.Stop();
            Console.WriteLine("Stop called at iteration {0}. data = {1} ", i, data);
            return;
        }
    });
Console.WriteLine("Stop called data = {0} ", data);
Console.ReadKey();

Parallel Aggregation

Parallel aggregation combines multiple inputs into single output. It is also called Parallel Reduction Pattern.

Precaution in Data parallelism

Don’t execute Parallel loop on the UI thread:

If Parallel loop is being executed to update UI control then it can move to state exceptions, delayed updates, deadlocks, etc.