# tokens + positions
x = E[ids] + P[:T]
# causal mask (True = allowed here)
mask = np.tril(np.ones((T, T), dtype=bool))
s = np.where(mask, Q @ K.T / np.sqrt(d), -np.inf)
# multi-head shapes
# (T, d) -> (heads, T, d // heads) -> attention -> (T, d)
# pre-norm block
x = x + attention(layer_norm(x), mask)
x = x + feed_forward(layer_norm(x))
# PyTorch: mind these
nn.TransformerEncoderLayer(d, heads, batch_first=True, norm_first=True)
nn.Transformer.generate_square_subsequent_mask(T)Tahák · Python · Neuronové sítě a moderní AI
Transformery
Z attention složíš celý transformer: pozice, více hlav, maska budoucnosti, blok a malý trénink v NumPy.