Lecture 4c. Loop-dependence analysis

In this video, we want to talk about identifying loop level parallelism. Solihin explains that scientific codes are characterized by manipulations on large data structures, which are largely iterative and have few dependencies of one iteration on the previous iteration.

It's much different from the situation in non scientific programming where frequently you have data structures like linked lists and trees, where you're trying to search those and you need to get the results from what iteration completed before doing the next iteration.

So it's not possible to do massive amounts of computation in parallel. The kind of array calculations that we saw for the Ocean application though, can be parallelized efficiently because you can have large grids divided into portions that are dealt with by different processors.

So we're dealing here with code level parallelism, not with program level parallelism or algorithm level parallelism, which we discussed a couple of weeks ago, but rather code level parallelism.

Our goal is given a piece of code without knowledge of the algorithm, find parallel tasks. So we're not going to be changing the algorithm

to get this parallelism, but we're rather just going to be seeing what can be done with different portions of the code.

At the same time,

we're going to focus on analyzing dependences within loops, and we'll talk a little bit more about those in the next video.

But for now, we want to look at how you can determine relationships, precedence, relationships among statements.

So we're going to be defining a few notations. Let S be a statement in the source code. S sub ij, whatever denotes a statement in the loop iteration ij.

So if we have a doubly nested loop, and S sub 11, for example, means the first iteration of the outer loop and the first iteration of the inner loop.

If we have s sub1s of 1, 2, that means the first iteration of the outer loop, but the second iteration of the inner loop within it s1, then s2 means that s1 must happen before s2.

For example, if s1 computes one of the inputs to s2, then s1 needs to finish before s2 could start.

If s1, then s2, we have a possibility of several dependencies. First of all, S2 with a true dependence on S1 means that S1 writes to a location that's read by S2.

So of course S1 has to finish first because it has to write that operand of S2's before S2 can continue.

S1 has an anti dependence on S2 means that S1 reads a location that's written by S2. So that's exactly the reverse here, S2 needs to occur before S1 because S1 depends upon something that's written by S2.

S1 with an output dependence on S2 means that S 1 writes to the same location written by S2. So I've got an example here with four statements.

X equals 2, y equals x, y equals x plus 4, and x equals Y. Now, there are many dependences within this group of four statements.

In fact, almost every statement depends upon every other statement. I'm going to list for you the dependences involving S1, and then we're going to work on the other dependences as an exercise.

First of all, we see that S2 has a true dependence on S1 S1 right arrow t S2 because S1 writes the variable X, which is then an input to S2.

We also see that S3 has a true dependence on S1 because S1 writes x, which is subsequently used as an operand by S3.

We also see that S1 has an output dependence with S4 because both S1 and S4 write the variable X. Well, these are not the only dependencies.

There are dependencies between S2, S3, and S4 as well. I've got a set of fill in the blank questions here that ask you to tell me a little bit about those dependencies.

One statement can have any of the three antidependances, true dependences, or output dependences on any other statement.

And sometimes a statement has more than one kind of dependence on another statement. So what I'd like you to do is answer these questions with either T, A or O if it has only one dependence.

But if it has multiple dependence, you might be able to answer something like at, which means that the second statement has both an anti and a true dependence on the first statement.

So that then introduces dependences. And in the next video we'll look at whether these dependencies are loop carried or loop independent.

It means, does one iteration of the loop depend upon a previous iteration? As we'll see in a few minutes,