Transformers Multi-Head Attention Mechanism

Instead of having a single attention head, we can use multiple attention heads. We learned how to compute the attention matrix $Z$, Instead of computing a single attention matrix $Z$  we can compute multiple attention matrices. But what is the use of computing multiple attention matrices? Let’s understand this with an example. Consider the phrase: $Alliswell$ Say we need to compute the self-attention of the word ‘well’. After computing the similarity score, suppose we have the following:

As we can observe from the preceding figure, the self-attention value of the word ‘well’ is the sum of the value vectors weighted by the scores. If you look at the preceding figure closely, the attention value of the actual word ‘well’ is dominated by the other word ‘All’. That is, since we are multiplying the value vector of the word ‘All’ by 0.6 and the value vector of the actual word ‘well’ by only 0.4, it implies that $Z_{well}$ will contain 60% of the values from the value vector of the word ‘All’ and only 40% of the values from the value vector of the actual word ‘well’. Thus, here the attention value of the actual word ‘well’ is dominated by the other word ‘All’.

This will be useful only in circumstances where the meaning of the actual word is ambiguous. That is, consider the following sentence: $Adogatethefoodbecauseitwashungry$ Say we are computing the self-attention for the word ‘it’. After computing the similarity score, suppose we have the following:

As we can observe from the preceding equation, the attention value of the word ‘it’ is just the value vector of the word ‘dog’. Here, the attention value of the actual word ‘it’ is dominated by the word ‘dog’. But this is fine here since the meaning of the word ‘it’ is ambiguous, as it may refer to either ‘dog’ or ‘food’.

Thus, if the value vector of other words dominates the actual word in cases as shown in the preceding example, where the actual word is ambiguous, then this dominance is useful; otherwise, it will cause an issue in understanding the right meaning of the word.

How to compute multi-head attention matrices

So, in order to make sure that our results are accurate, instead of computing a single attention matrix, we will compute multiple attention matrices and then concatenate their results. The idea behind using multi-head attention is that instead of using a single attention head, if we use multiple attention heads, then our attention matrix will be more accurate. Let’s explore this in more detail.

Let’s suppose we are computing two attention matrices $Z_1$ and $Z_2$ First, let’s compute the attention matrix $Z_1$

Computing attention matrix Z1

We learned that to compute the attention matrix, we create three new matrices, called query, key, and value matrices. To create the query, $Q_1$, key $K_1$ and value $V_1$ matrices, we introduce three new weight matrices called $W_1^Q$ $W_1^K$ $W_1^V$ We create the query, key, and value matrices by multiplying the input matrix, X by $W_1^Q$ $W_1^K$ $W_1^V$ respectively

Now, the attention matrix $Z_1$ can be computed as follows:

$$Z=softmax(Q_1\ast K_1^T/\sqrt{d_k})\ast V_1$$

Computing attention matrix Z2

We learned that to compute the attention matrix, we create three new matrices, called query, key, and value matrices. To create the query, $Q_2$, key $K_2$ and value $V_2$ matrices, we introduce three new weight matrices called $W_2^Q$ $W_2^K$ $W_2^V$ We create the query, key, and value matrices by multiplying the input matrix, X by $W_2^Q$ $W_2^K$ $W_2^V$ respectively

Now, the attention matrix $Z_2$ can be computed as follows:

Concatenating multiple attention matrices

Similarly we can compute $h$ number of attention matrices. Suppose we have eight matrices $Z_1$ to $Z_8$; then we can just concatenate all the attention heads (attention matrices) and multiply the result by a new weight matrix $W^O$ and create the final attention matrix as shown

$Multi-headattention=Concatenate(Z_1,Z_2,Z_3,Z_4,Z_5,Z_6,Z_7,Z_8)$