Python MPI



Teaching: 30 min
Exercises: 0 min
  • What is parallel computing?

  • What are the different parallel computing approaches?

  • What are the pro’s and con’s of the different approaches?

  • Explain the basics of parallel computing.


An executable file residing on disk
One or more executing instances of a program. Processes have separate address spaces.
In MPI, a process is sometimes called a task, but these are used interchangably. We will always use “process” rather than “task”.
Thread (or lightweight process)
Ane or more threads of control within a process. Threads share the same address space.


What is parallel computing?

Traditionally, software has been written for serial computation:

serial computer

In the simplest sense, parallel computing is the simultaneous use of multiple compute resources to solve a computational problem:

parallel computer

Why do we need parallel programming?

Need to solve larger problems Require more memory Computation takes much longer Huge amounts of data may be required

Parallel programming provides

Two basic approaches

Shared Memory Computer

shared memory computer

Distributed Memory (ex. Compute cluster)

distributed memory computer

Parallel programming models

Directive-based parallel programming language

Message Passing

Pros and cons

Pros of OpenMP

Cons of OpenMP

Pros of MPI

Cons of MPI

Parallel programming issues

Goal is to reduce execution time

Load Balancing

Minimizing Communication

Where possible - overlap communication and computation

Many problems scale well to only a limited number of processors

Amdahl’s law

Amdahl's law


Basically this is saying that the amount of speedup a program will see by using n cores is based on how much of the program is serial (can only be run on a single CPU core) and how much of it is parallel (can be split up among multiple CPU cores).

amdahl's law

Programming approaches

Two main approaches:

SPMD - Single Program, Multiple Data Streams

MPMD - Multiple Programs, Multiple Data Streams

What is MPI?

MPI stands for Message Passing Interface


Version differences

Examples of Different Implementations

Similiarities in Various Implementations

Difference in Various Implementations

Key Points