When you think about your project, how do you find the parallelism?
At the highest level, parallelism exists either in the form of data on which to operate in parallel, or in the form of tasks to execute concurrently. And these forms are not mutually exclusive.
Data parallelism (Figure 2-1) is easy to picture. Take lots of data and apply the same transformation to each piece of the data. In Figure 2-1, each letter in the data set is capitalized and becomes the corresponding uppercase letter. This simple example shows that given a data set and an operation that can be applied element by element, we can apply the same task concurrently to each element. Programmers writing code for supercomputers love this sort of problem and consider it so easy to do in parallel that it has been called embarrassingly parallel. A word of advice: if you have lots of data parallelism, do not be embarrassed—take advantage of it and be very happy. Consider it happy parallelism.
Figure 2-1. Data parallelism
Data parallelism is eventually limited by the amount of data you want to process, and your thoughts will then turn to task parallelism (Figure 2-2). Task parallelism means lots of different, independent tasks that are linked by sharing the data they consume. This, too, can be embarrassingly parallel. Figure 2-2 uses as examples some mathematical operations that ...