Tutorial 1: PyTorch and Word Embeddings

1Overview

Lecture 2 introduced the idea that a word can be represented by a dense vector learned from the contexts it appears in. This tutorial turns that idea into working code. Notebook 1 gives you the PyTorch skills you will use all semester: tensors, automatic differentiation, optimizers, nn.Module and data loaders. Notebook 2 uses those skills to train a Skip-gram Word2Vec model on 50,000 Quora questions and to compare it with pre-trained GloVe and FastText vectors.

Learning objectives

• Write the forward pass of a 4-layer feed-forward network with raw tensors, and explain what loss.backward() computes.
• Verify that optimizer.step() performs exactly the update \(W \leftarrow W - \eta\,\nabla_W J\), and why optimizer.zero_grad() is needed.
• Re-implement the same network as an nn.Module and feed it data with Dataset/DataLoader.
• Turn raw text into Skip-gram training pairs (tokenize, build a vocabulary, slide a context window).
• Train a Skip-gram model, read off its word vectors, and query nearest neighbours by cosine similarity.
• Explain how GloVe and FastText differ from Word2Vec, and estimate their memory and compute cost.

Time plan

Downloads add time: CIFAR-10 (170 MB) in Notebook 1, and the pre-trained GloVe (376 MB) and FastText (958 MB) vectors in Notebook 2.

2Before you start

conda create -n nlp26 python=3.11 -y
conda activate nlp26
pip install torch torchvision numpy pandas matplotlib \
            scikit-learn tqdm gensim "nltk>=3.9.2,<4" "bokeh>=3.8,<3.10" jupyterlab
jupyter lab

What you need to know already

• Python basics (functions, classes, list comprehensions) and NumPy arrays.
• Matrix–vector products and the chain rule from calculus.
• Lecture 2: distributional semantics, one-hot vs. dense vectors, cosine similarity.

3Notebook 1 · From raw tensors to nn.Module

3.1 The network and its forward pass

The network maps a 3-dimensional input to one output through three hidden layers of 5 units, with a sigmoid after every layer (\(D = 3\), \(K = 5\)). Its 86 parameters are created by hand with requires_grad=True, which tells PyTorch to track every operation on them.

D, K = 3, 5
W1 = torch.randn((K, D), requires_grad=True)   # parameters you create yourself
b1 = torch.randn((K, 1), requires_grad=True)
# ... W2, b2, W3, b3 (5x5), W4 (1x5), b4

def sigmoid(x):
    return 1 / (1 + torch.exp(-x))

x = torch.randn((D, 1))                         # one input
y_ground = torch.randn((1,))                    # its target
a1 = sigmoid(torch.matmul(W1, x) + b1)
a2 = sigmoid(torch.matmul(W2, a1) + b2)
a3 = sigmoid(torch.matmul(W3, a2) + b3)
y_pred = sigmoid(torch.matmul(W4, a3) + b4)
J = (y_pred - y_ground) ** 2                    # squared error

3.2 Backward pass: who is responsible for the loss?

J.backward() walks the recorded computation graph backwards and applies the chain rule, storing \(\partial J/\partial \theta\) in .grad for every tracked parameter. For the last layer, for example:

and the same pattern repeats layer by layer. The notebook shows this live:

Gradients do not exist until a backward pass creates them — that is why the first print shows None.

3.3 Updating the parameters

Gradient descent moves every parameter a small step against its gradient, \(\theta \leftarrow \theta - \eta\,\nabla_\theta J\) with learning rate \(\eta = 0.1\). The notebook first computes the new \(W_2\) by hand, then lets torch.optim.SGD do it, and checks that both agree:

lr = 0.1
expected_new_W2 = W2 - lr * W2.grad             # the update, by hand

optimizer = torch.optim.SGD([W1, b1, W2, b2, W3, b3, W4, b4], lr=lr)
optimizer.step()                                # the same update, by PyTorch

torch.eq(W2, expected_new_W2).all()             # -> tensor(True)

3.4 The same network, the PyTorch way

nn.Linear owns its weight and bias, nn.Module registers them automatically, and network.parameters() hands them all to the optimizer. Compare the two versions:

W1 = torch.randn((K, D), requires_grad=True)
b1 = torch.randn((K, 1), requires_grad=True)

z1 = torch.matmul(W1, x) + b1
a1 = sigmoid(z1)

class neural_network(torch.nn.Module):
    def __init__(self, D, K):
        super().__init__()
        self.layer1 = torch.nn.Linear(D, K)
        self.layer2 = torch.nn.Linear(K, K)
        self.layer3 = torch.nn.Linear(K, K)
        self.layer4 = torch.nn.Linear(K, 1)

    def forward(self, x):
        a = torch.sigmoid(self.layer1(x))
        a = torch.sigmoid(self.layer2(a))
        a = torch.sigmoid(self.layer3(a))
        return torch.sigmoid(self.layer4(a))

A full training step is now four lines, the pattern you will reuse in every assignment:

network = neural_network(D, K)
optimizer = torch.optim.SGD(network.parameters(), lr=0.1)

y_pred = network(x)                 # 1. forward
J = (y_pred - y_ground) ** 2        # 2. loss
J.backward()                        # 3. backward
optimizer.step()                    # 4. update
optimizer.zero_grad()               #    and reset gradients for the next step

3.5 Feeding data: NumPy, Dataset and DataLoader

torch.from_numpy and .numpy() convert between NumPy and PyTorch without copying memory. For real datasets, a Dataset says how to get one example and a DataLoader turns it into shuffled mini-batches:

class CustomDataset(torch.utils.data.Dataset):
    def __init__(self):
        self.inputs = torch.randn(100, 3)            # 100 samples, 3 features
        self.labels = torch.randint(0, 2, (100,))    # binary labels

    def __getitem__(self, index):                    # one example
        return self.inputs[index], self.labels[index]

    def __len__(self):                               # dataset size
        return len(self.inputs)

train_loader = torch.utils.data.DataLoader(CustomDataset(), batch_size=64, shuffle=True)
inputs, labels = next(iter(train_loader))

The notebook also loads CIFAR-10 through torchvision.datasets: each image is a [3, 32, 32] tensor, and a loader with batch_size=64 yields batches of shape [64, 3, 32, 32].

4Notebook 2 · Word embeddings: Word2Vec, GloVe and FastText

4.1 From raw text to training pairs

Words seen fewer than 5 times are mapped to <unk> (index 0); in the run this affects 36,778 of the 621,721 tokens (5.9%). The window of size 2 then turns every position into a (center, context) example:

4.2 Skip-gram Word2Vec

Skip-gram predicts each context word from the center word. Every word has two vectors: an input vector \(u_w\) (a row of the embedding matrix \(U\)) and an output vector \(v_w\) (a column of the output layer \(V\)). The model turns their dot products into a distribution over the vocabulary:

class Skip_Gram_Model(nn.Module):
    def __init__(self, vocab_size, embedding_dim):
        super().__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)   # U
        self.out_layer = nn.Linear(embedding_dim, vocab_size)       # V

    def forward(self, inputs):                 # inputs: center-word indices
        return self.out_layer(self.embeddings(inputs))   # logits over the vocabulary

model = Skip_Gram_Model(len(word2index), 32).cuda()
criterion = nn.CrossEntropyLoss()              # softmax + negative log-likelihood
optimizer = optim.Adam(model.parameters(), lr=0.001)

for batch in make_skip_gram_batches_iter(contexts, window_size=2, num_skips=4, batch_size=128):
    tokens, labels = batch['tokens'].cuda(), batch['labels'].cuda()
    loss = criterion(model(tokens), labels)
    loss.backward(); optimizer.step(); optimizer.zero_grad()

4.3 Training and what the model learned

The word vectors are the rows of model.embeddings.weight. Nearest neighbours by cosine similarity, \(\cos(u_a, u_b) = \frac{u_a^{\top}u_b}{\lVert u_a\rVert\,\lVert u_b\rVert}\), already show structure:

Most neighbours of my are other possessives (our, your, his, her): they appear in the same contexts, so they get similar vectors. The notebook then draws an interactive t-SNE map of the 100 most frequent words with Bokeh.

4.4 GloVe: learning from global co-occurrence counts

Instead of predicting one context at a time, GloVe first counts how often every pair of words co-occurs in the whole corpus, \(X_{ij}\), and then fits vectors whose dot products match the log counts:

The weight \(f\) (with \(x_{\max}=100,\ \alpha=3/4\) in the paper) stops very frequent pairs such as “of the” from dominating. The notebook loads pre-trained 300-d vectors trained on Wikipedia + Gigaword (6B tokens):

import gensim.downloader as api
glove_model = api.load("glove-wiki-gigaword-300")       # 376 MB, 400K words, 300-d
glove_model.most_similar("king", topn=5)                 # nearest neighbours
glove_model.similarity("king", "queen")                  # cosine similarity

4.5 FastText: words are bags of character n-grams

FastText represents a word by its own vector plus the vectors of its character n-grams (3 to 6 characters, with boundary markers < and >):

Because teach, teacher and teaching share n-grams, rare and inflected words get good vectors, and a trained FastText model can even build vectors for words it never saw.

4.6 Comparing the three methods

The gap is about data and scale, not about Skip-gram itself: our model saw 50K short questions in which king and queen are rare, with 32-d vectors and 4 epochs. GloVe and FastText were trained on billions of tokens with 300-d vectors. Likewise, Skip-gram returns None for serendipity because the word never made it into our 7,251-word vocabulary.

Memory and compute (example: \(|V| = 10^4,\ d = 300,\ T = 10^6\) tokens, window \(C = 5\), \(K = 5\) negatives, \(S = 10^5\) n-grams, \(N = 10^7\) co-occurring pairs; float32)

Worked numbers: \(2\cdot10^4\cdot300\cdot4\,\text{B} = 24\,\text{MB}\); \((6.02\cdot10^6 + 10^7)\cdot 4\,\text{B} \approx 64\,\text{MB}\) (count values only; storing the \((i,j)\) indices adds more); \(1.1\cdot10^5\cdot300\cdot4\,\text{B} = 132\,\text{MB}\). Section 1.6 of the notebook counts only the input matrix for Word2Vec and leaves the co-occurrence entries out of its GloVe total, so its numbers are smaller.

5Pitfalls & troubleshooting

6Exercises

Not graded — but they are the best preparation for Assignment 1.

7Self-check quiz

Click a question to reveal the answer.

8Further reading

Next steps: Lecture 2 slides · Course schedule (Assignment 1 is out).