Will We Ever Master Parallel Programming?

I learned the fundamentals of parallel programming as an academic exercise in my operating systems classes of two decades ago.  Mutexes, semaphores, and critical sections are important considerations in designing operating systems that have to do multiple things seemingly simultaneously.

And I also learned about SIMD (single instruction, multiple data) and MIMD (multiple instruction, multiple data) architectures, which have implications on what kinds of processing certain computers are good at.  For example, many of the graphics processing units today have SIMD characteristics.

But the truly difficult part is writing applications that can take advantage of parallel architectures.  First, programmers have to be able to identify what parts of an application can be executed in parallel.  This is no small feat, as he or she must understand the functional purpose of the application and what it is doing at each point in the code.  It requires in-depth understanding of both the application domain and how each operation in that code relates to that domain.

Then they have to carefully mark off the code that can’t be executed in parallel, defining it as a set of critical sections using some form of mutex (mutual exclusion).  Any errors in either of these processes will likely result in an incorrect program that may be nearly impossible to debug.

A company called RapidMind, acquired by Intel earlier this year, addressed this problem by incorporating an abstraction layer that helps identify opportunities for parallelism in applications, and routes much existing code to take advantage of those opportunities.

But Intel doesn’t stop there.  Intel Parallel Studio provides essential tools for working with multi-threaded code, enabling developers to more easily identify errors in threads and memory, and to be able to identify and analyze bottlenecks.  It consists of three components – Intel Parallel Composer, Intel Parallel Inspector, and Intel Parallel Amplifier.  Together, these tools enable developers building multi-threaded applications for parallel execution to construct and debug applications that are able to make better use of today’s processors.

At the operating system level, Windows and Linux have to be able to do a better job of dispatching multiple running applications to different processor cores.  Whenever I look at processor utilization on Windows Perfmon, I am chagrined that one of my two cores always is by far more highly utilized than the other.

I’ll have a lot more to say on this topic in upcoming posts.  This is just too important a topic to not discuss in depth.  This morning I was talking to Stefan Möhl, cofounder and chief science officer of Mitrionics, and he raised the point that if we can’t take advantage of multi-core processors, then Moore’s Law is dead for all practical purposes.  And if Moore’s Law is dead, that has serious implications for our information society.

Just saying.