Intuition about project input embedding tokens to queries, keys and values

[krishnan-duraisamy]

In Section 3.4.1 we have this definition - These three matrices are used to project the embedded input tokens, x(i), into query, key, and value vectors, respectively, as illustrated in figure 3.14

These weight matrices are then initialized later to random tensors like so:

torch.manual_seed(123)
W_query = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)
W_key   = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)
W_value = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)

• Where then or how is the input being projected then to these respective weight matrices?
• Is the intuition behind the split to also reduce dimensionality?

[rasbt]

Hi there,

Where then or how is the input being projected then to these respective weight matrices?

This would be in the forward method:

def forward(self, x):
    keys = x @ self.W_key
    queries = x @ self.W_query
    values = x @ self.W_value

Where then or how is the input being projected then to these respective weight matrices?

Actually, there is no splitting here, they are all separate matrices. Maybe it's easier to see this in figure 3.18.

With splitting, you perhaps mean the separate heads in figure 3.24? This is to have separate channels that can learn independently, similar to convolutional neural network channels.