jump to navigation

Will We Ever Master Parallel Programming? December 10, 2009

Posted by Peter Varhol in Architectures, Software development, Software tools.

I learned the fundamentals of parallel programming as an academic exercise in my operating systems classes of two decades ago.  Mutexes, semaphores, and critical sections are important considerations in designing operating systems that have to do multiple things seemingly simultaneously.

And I also learned about SIMD (single instruction, multiple data) and MIMD (multiple instruction, multiple data) architectures, which have implications on what kinds of processing certain computers are good at.  For example, many of the graphics processing units today have SIMD characteristics.

But the truly difficult part is writing applications that can take advantage of parallel architectures.  First, programmers have to be able to identify what parts of an application can be executed in parallel.  This is no small feat, as he or she must understand the functional purpose of the application and what it is doing at each point in the code.  It requires in-depth understanding of both the application domain and how each operation in that code relates to that domain.

Then they have to carefully mark off the code that can’t be executed in parallel, defining it as a set of critical sections using some form of mutex (mutual exclusion).  Any errors in either of these processes will likely result in an incorrect program that may be nearly impossible to debug.

A company called RapidMind, acquired by Intel earlier this year, addressed this problem by incorporating an abstraction layer that helps identify opportunities for parallelism in applications, and routes much existing code to take advantage of those opportunities.

But Intel doesn’t stop there.  Intel Parallel Studio provides essential tools for working with multi-threaded code, enabling developers to more easily identify errors in threads and memory, and to be able to identify and analyze bottlenecks.  It consists of three components – Intel Parallel Composer, Intel Parallel Inspector, and Intel Parallel Amplifier.  Together, these tools enable developers building multi-threaded applications for parallel execution to construct and debug applications that are able to make better use of today’s processors.

At the operating system level, Windows and Linux have to be able to do a better job of dispatching multiple running applications to different processor cores.  Whenever I look at processor utilization on Windows Perfmon, I am chagrined that one of my two cores always is by far more highly utilized than the other.

I’ll have a lot more to say on this topic in upcoming posts.  This is just too important a topic to not discuss in depth.  This morning I was talking to Stefan Möhl, cofounder and chief science officer of Mitrionics, and he raised the point that if we can’t take advantage of multi-core processors, then Moore’s Law is dead for all practical purposes.  And if Moore’s Law is dead, that has serious implications for our information society.

Just saying.



1. A Different Approach to Multi-core Programming « Cutting Edge Computing - January 8, 2010

[…] As a developer, I know just how difficult this is.  One of the most difficult problems we face is understanding the application intimately to know where there are opportunities for separate code streams to run independently, and to make detailed changes to the code in those areas to make sure the code will run on different cores without causing synchronization problems.  I’ve talked about the issues involved in more detail in this post. […]

2. Can Toolkits Make Programming for Parallel Processing Unnecessary? « Cutting Edge Computing - January 18, 2010

[…] program structure and write code so that segments can be executed independently.  Toolkits such as Intel’s Parallel Studio, Nvidia’s Nexus (specifically for Nvidia graphics processing units), and Mitrionics C can help, […]

3. Do We Care About Writing Parallel Code? « Cutting Edge Computing - January 19, 2010

[…] by pvarhol in Software development, Strategy. trackback My writings on developing code to run in parallel beg the question of just how important is the ability for developers to produce code that can […]

4. Can We Abstract Away from Multiple Cores in Coding? « Cutting Edge Computing - March 14, 2010

[…] in Architectures, Software development, Software platforms. trackback Readers know that I’m very interested in how software developers are adapting to programming for multi-core systems.  […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: