Lecture 4d. Loop-independent vs. loop-carried dependences

in this video we want to talk about loop

independent versus loop carried

dependencies

this will help us decide which

iterations of which loops can be

executed

in parallel a loop carried dependence

means that dependence exists

across iterations if you take out the

for statement before the loop

then the dependence no longer exists

for a loop independent dependence the

dependence exists within an iteration

so if you take out the loop the

dependence would not change at all it

would still be there

we've got an example over here with

three

nested loops the first loop has two

statements in it

there's a loop carried dependence of

state of the i

plus first iteration of this

of this two statement loop with the i

federation

the reason for this is that a value

that's used on the i

plus first iteration is in fact

calculated

on the i federation that's because each

iteration uses

the value of a or uses an element of a

that was calculated on the previous

iteration also we see that a sub i is

computed here

and used there so this means that

statement s2 is dependent upon statement

s1 finishing before it but this is a

loop independent dependence because if

you take out the for statement

it's still there you still need to

finish s1 before you can overwrite the

value of

a sub i now in s3 notice that there is a

subscript of j

on the left hand side and a subscript of

j minus 1 on the right hand side

this means there's a loop carry

dependence on j

because the value that's written in the

j minus first

iteration is going to be used on the jth

iteration

but there are no dependencies in the i

direction one iteration

the i federation the i throw does not

depend upon the i plus first row or the

i

minus first row so we could actually

compute all of the rows

in parallel here in the fourth statement

which

is actually in the third loop there's no

loop carried dependence on j

because j doesn't change the subscript j

doesn't change

from the right hand side to the left

hand side of the statement

but there is a loop carry dependence on

the foreign loop

because the i throw depends upon values

computed in the i minus first row

and so we could do all the columns of

this loop in parallel

but we can't do the rows in parallel

because of that dependence

two kinds of graphs can help us see

these relationships

first is the iteration space traversal

graph or itg

it shows the order of traversal from one

iteration to another

here a node represents a point in the

iteration space in other words a node

represents

a set of i value and a j value like

i equals one j equals two and a directed

edge

indicates the next point that will be

encountered after the current point is

traversed

so here we see statement s3 which says a

sub ij

equals a sub i j minus 1 plus 1.

well actually as far as the

iteration space traversal graph is

concerned

what matters is not what's in s3 but

rather what's in the four statements

i goes from 1 to 4 but while i is going

from 1 to 4

j is going from 1 to 4 for each value of

i

so we start out with a with i equal 1 j

equal 1 and then we go to i equal 1 j

equal to 2

i and j are 1 and 3 respectively now i

and j

are 2 and 1 in the next iteration of the

loop and it just goes on like that

a loop carried dependence graph shows

the various dependence relationships

graphically

again a node is a point in the iteration

space in other words in this

two-dimensional

graph it's a point in the i j space

value of i and a value of j a directed

edge represents the dependence

what this shows us is that the iteration

1 comma 2 for i and j is dependent upon

the iteration 1 comma 1.

and that's because n1 comma 2 with i

equal to 1 and j equal to 2

we use a value that's computed in the

ij equal 1 1 iteration where we're

looking at

a sub 1 j sub 1. so that's why we have

this edge from 1 1 to 1 2. and this is a

true dependence because the value that

was computed in the previous iteration

of j

will be used in the current iteration of

j

so we see dependencies in this direction

we don't see any dependents from 1 3 to

2 1

because we don't have any dependencies

there the dependencies are all

along the rows they're between the

columns and the rows

there are no dependencies between this

row and this row

here we have another example involving

two nested loops

the first loop is a 4i 4j loop

that has a four point iteration

statement in the middle of it

in other words what this does is

computes the value of a sub i j

by summing up all the neighboring points

the ones above and below

and to the left and to the right because

this is the same

4i4j pattern that we saw before because

we have the same 4i4j pattern that we

did before

theater race and space traversal graph

is exactly the same

we do the 1 1 iteration followed by the

1 2 iteration

followed by dot dot the 1n iteration

followed by the 2 1 iteration and so

forth

now let's look at the dependence

relationship so we can draw the loop

dependence graph

as far as true dependences are concerned

there are two of them

the value of s one i j is used as an

operand in computing

s one of i j plus one note this

subscript here the j minus one subscript

indicates

that it's using the value that was

computed in the previous iteration of j

there are no output dependences because

there's only one statement in the loop

and the anti-dependences actually are

exactly the same

and the reason for that is that we have

these j plus one sub

which means we've got to be sure that we

don't overwrite

the next value the the value at the

point to the right

before we use it in our current

computation so one

thing that's interesting to note is that

the true dependencies

have exactly the same form as the

corresponding antidependences

only that these have a t there and these

have an a

and that means that when we actually

look at the loop dependence graph

we will find that each edge represents

both true and anti-dependences in other

words the

the iteration point to the right is

dependent

in both the true dependence and

antidependence on the point to the left

as is the point below okay so this

raises an interesting question

suppose we dropped off the first half of

statement s1

so we have what's shown below a sub i j

equals a

sub i minus 1 j plus a i

plus 1 j if we did that which of the

dependencies would still exist

and then let's do the same thing

removing the last half of the statement

then let's go on to the second loop nest

right here

the value of a sub i j that's computed

in s2

is used in the next the j the next j

iteration

of s3 so that's the true dependence

as shown here the anti-dependence again

is that b sub ij is used here

and written there so this read has to

occur

before that right the loop dependence

graph is pretty easy

because there are dependencies only in

the jth iteration

no dependences between the various rows

which means that all of the rows could

be

executed in parallel so

why are there no vertical edges in this

graph

well that's because there are no

dependencies involving i

so we could do the i the rows i in

parallel

why is the anti-dependence not shown on

the

note that the graph shows only the the

true dependence here

well the reason for that is because the

anti-dependence

is not loop carry and because it's not

loop parry

carried it does not appear in the ldg

because nodes represent all statements

in the loop body and so

if there is a dependence between two of

those statements the dependence is

within the node

not across nodes