1.1 Preparation

Word embedding visualization：https://rosierao.github.io/nlp-git-demo/word-embeddings.html

Environment installation and data download

# 1. Install dependencies
%pip install -q "nltk>=3.9.2,<4" "bokeh>=3.8,<3.10" gensim torch numpy pandas matplotlib scikit-learn tqdm

# 2. Imports
import time
import math
import random
import zipfile
from pathlib import Path
from urllib.request import urlretrieve

import nltk
import bokeh
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from nltk.tokenize import word_tokenize as _nltk_word_tokenize
from tqdm.auto import tqdm
from IPython.display import clear_output

%matplotlib inline

# 3. Tokenizer: no punkt / punkt_tab download required
# Treat each question as one text unit and tokenize its words.
def word_tokenize(text):
    return _nltk_word_tokenize(text, preserve_line=True)

# 4. Set random seeds
random.seed(42)
np.random.seed(42)
torch.manual_seed(42)

if torch.cuda.is_available():
    torch.cuda.manual_seed_all(42)

# 5. Prepare the Quora dataset
# Reuse train.csv if it already exists.
csv_path = Path("train.csv")

if not csv_path.exists():
    archive_path = Path("quora.zip")

    # Original notebook's Google Drive mirror
    data_url = (
        "https://drive.google.com/uc?export=download"
        "&id=1ERtxpdWOgGQ3HOigqAMHTJjmOE_tWvoF"
    )

    if not archive_path.exists():
        print("Downloading Quora data...")
        urlretrieve(data_url, archive_path)

    if not zipfile.is_zipfile(archive_path):
        raise RuntimeError(
            "quora.zip is not a valid ZIP file. "
            "Download train.csv from "
            "https://www.kaggle.com/c/quora-question-pairs/data "
            "and place it in this notebook's working folder."
        )

    with zipfile.ZipFile(archive_path) as archive:
        if "train.csv" not in archive.namelist():
            raise RuntimeError(
                "The archive does not contain train.csv. "
                "Please provide the Quora train.csv file."
            )

        with archive.open("train.csv") as source:
            csv_path.write_bytes(source.read())

# 6. Check the dataset
columns = pd.read_csv(csv_path, nrows=0).columns

if not {"question1", "question2"}.issubset(columns):
    raise ValueError(
        "train.csv must contain the columns question1 and question2."
    )

# 7. Confirm setup
print("NLTK version:", nltk.__version__)
print("Bokeh version:", bokeh.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Tokenizer test:", word_tokenize("This is a real text example."))
print("Dataset ready:", csv_path.resolve())

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 30.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 27.8/27.8 MB 55.7 MB/s eta 0:00:00
Downloading Quora data...
NLTK version: 3.10.3
Bokeh version: 3.8.2
CUDA available: True
Tokenizer test: ['This', 'is', 'a', 'real', 'text', 'example', '.']
Dataset ready: /content/train.csv

1. Tokenize and lower-case texts.

quora_data = pd.read_csv('train.csv')

quora_data.question1 = quora_data.question1.replace(np.nan, '', regex=True)
quora_data.question2 = quora_data.question2.replace(np.nan, '', regex=True)

texts = list(pd.concat([quora_data.question1, quora_data.question2]).unique())
texts = texts[:50000] # Accelerated operation
print(len(texts))

tokenized_texts = [word_tokenize(text.lower()) for text in tqdm(texts)]

assert len(tokenized_texts) == len(texts)
assert isinstance(tokenized_texts[0], list)
assert isinstance(tokenized_texts[0][0], str)

50000

tokenized_texts[0]

['what',
 'is',
 'the',
 'step',
 'by',
 'step',
 'guide',
 'to',
 'invest',
 'in',
 'share',
 'market',
 'in',
 'india',
 '?']

1. Collect the indices of the words:

from collections import Counter

MIN_COUNT = 5

words_counter = Counter(token for tokens in tokenized_texts for token in tokens)
word2index = {
    '<unk>': 0
}

for word, count in words_counter.most_common():
    if count < MIN_COUNT:
        break

    word2index[word] = len(word2index)

index2word = [word for word, _ in sorted(word2index.items(), key=lambda x: x[1])]

print('Vocabulary size:', len(word2index))
print('Tokens count:', sum(len(tokens) for tokens in tokenized_texts))
print('Unknown tokens appeared:', sum(1 for tokens in tokenized_texts for token in tokens if token not in word2index))
print('Most freq words:', index2word[1:21])

Vocabulary size: 7251
Tokens count: 621721
Unknown tokens appeared: 36778
Most freq words: ['?', 'the', 'what', 'is', 'how', 'i', 'a', 'to', 'in', 'do', 'of', 'are', 'and', 'can', 'for', ',', 'you', 'why', 'it', 'best']

1. collect the context words

First of all, we need to collect all the contexts from our corpus.

def build_contexts(tokenized_texts, window_size):
    contexts = []
    for tokens in tokenized_texts:
        for i in range(len(tokens)):
            central_word = tokens[i]
            context = [tokens[i + delta] for delta in range(-window_size, window_size + 1)
                       if delta != 0 and i + delta >= 0 and i + delta < len(tokens)]

            contexts.append((central_word, context))

    return contexts

contexts = build_contexts(tokenized_texts, window_size=2)

Check, what you got:

contexts[:5]

[('what', ['is', 'the']),
 ('is', ['what', 'the', 'step']),
 ('the', ['what', 'is', 'step', 'by']),
 ('step', ['is', 'the', 'by', 'step']),
 ('by', ['the', 'step', 'step', 'guide'])]

1. Convert to indices

Let's convert words to indices:

contexts = [(word2index.get(central_word, 0), [word2index.get(word, 0) for word in context])
            for central_word, context in contexts]

1.2 Skip-Gram Word2vec

Word2vec is actually a set of models used to build word embeddings. We are going to start with the skip-gram model.

It's a very simple neural network with just two layers. It aims to build word vectors that encode information about the co-occurring words:

More precisely, it models the probabilities $\{P(w_{c+j}|w_c): j = c-k, ..., c+k, j \neq c\}$, where $k$ is the context window size, $c$ is index of the central word (which embedding we are trying to optimize).

The learnable parameters of the model are following: matrix $U$ (embeddings' matrix that is used in all downstream tasks. In gensim it's called syn0) and matrix $V$ - output layer of the model (in gensim it's called syn1).

Two vectors correspond to each word: a row in $U$ and a column in $V$. That is $U \in \mathbb{R}^{|V|, d}$ and $V \in \mathbb{R}^{d, |V|}$, where $d$ is embedding size and $|V|$ is the vocabulary size.

As a result, the neural network looks this way:

What's going on and how it is connected to probability and word context?

Well, the word is mapped to its embedding $u_c$. Then this embedding is multiplied to matrix $V$.

As a result, we obtain the set of scores $\{v_j^T u_c : j \in {0, \ldots, |V|}\}$. Each corresponds to the similarity between the word $w_j$ vector and our word vector. It's very similar to the cosine similarity we calculated in the previous lesson, but without normalization.

This similarities show how likely $w_j$ can be in context of word $w_c$. That means, that they can be converted to probability using the softmax function: $$P(w_j | w_c) = \frac{\exp(v_{j}^T u_c)}{\sum_{i=1}^{|V|} \exp(v_i^T u_c)}.$$

So for each word we calculate such probability distribution over our vocabulary. It's shown in using blue bars in the picture above. More likely word - bluer is the corresponding cell.

The model learns to distribute the probabilities between the co-occuring words for the given one. We'll use cross-entropy loss for it: $$-\sum_{-k \leq j \leq k, j \neq 0} \log \frac{\exp(v_{c+j}^T u_c)}{\sum_{i=1}^{|V|} \exp(v_i^T u_c)} \to \min_{U, V}.$$

For instance, for the sample from the picture model will be penalized if it outputs a low probability of word over.

Please, notice that we calculate the similarity between vectors from different vector spaces. $u_c$ is the vector from the input embeddings and $v_j$ is the vector from the output embeddings. A high similarity between them means that they co-occur frequently, not that they are similar in the syntactic role or their semantics.

On the other hand, the similarity between $u_k$ and $u_m$ means that their output distributions are similar. And that means exactly that the similarity of the count vectors we discussed before and also most probably means their syntactic or semantic similarity.

Check this demo to understand what's going on in more depth: https://ronxin.github.io/wevi/.

Let's implement it now!

1. Batches Generations

Neural networks are optimized using stochastic gradient descent methods. This requires a batch generator, a function that produces samples for optimizing the neural network.

Implementation of a Batch Generator:

import random

def make_skip_gram_batches_iter(contexts, window_size, num_skips, batch_size):
    assert batch_size % num_skips == 0
    assert num_skips <= 2 * window_size

    central_words = [word for word, context in contexts if len(context) == 2 * window_size and word != 0]
    contexts = [context for word, context in contexts if len(context) == 2 * window_size and word != 0]

    batch_size = int(batch_size / num_skips)
    batches_count = int(math.ceil(len(contexts) / batch_size))

    print('Initializing batches generator with {} batches per epoch'.format(batches_count))

    indices = np.arange(len(contexts))
    np.random.shuffle(indices)

    for i in range(batches_count):
        batch_begin, batch_end = i * batch_size, min((i + 1) * batch_size, len(contexts))
        batch_indices = indices[batch_begin: batch_end]

        batch_data, batch_labels = [], []

        for data_ind in batch_indices:
            central_word, context = central_words[data_ind], contexts[data_ind]

            words_to_use = random.sample(context, num_skips)
            batch_data.extend([central_word] * num_skips)
            batch_labels.extend(words_to_use)
        yield {
            'tokens': torch.LongTensor(batch_data),
            'labels': torch.LongTensor(batch_labels)
        }

Check it:

batch = next(make_skip_gram_batches_iter(contexts, window_size=2, num_skips=2, batch_size=32))

batch

Initializing batches generator with 24578 batches per epoch

{'tokens': tensor([ 23,  23,   2,   2,  10,  10,   3,   3, 966, 966,  35,  35,  85,  85,
         281, 281,   6,   6,   6,   6,  63,  63, 195, 195,  19,  19,  73,  73,
           6,   6,  96,  96]),
 'labels': tensor([   0,    1,   20,    3, 6353, 1066,  328,   76,  661,  745,   15,   43,
           85,   85,    2, 1340,    5,   65,   10,  695, 5461,   12,   10,    8,
           43,    3,  174,    1,   29,   10,   27, 1591])}

1. Model

Here is the model implementation of skip-gram model, which has only two layers of neural networks.

class Skip_Gram_Model(nn.Module):
    def __init__(self, vocab_size, embedding_dim):
        super().__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.out_layer = nn.Linear(embedding_dim, vocab_size)

    def forward(self, inputs):
        embedded = self.embeddings(inputs)
        out = self.out_layer(embedded)
        return out

1. Training

# Here are the hyperparameters you can adjust
embedding_dim = 32
learning_rate = 0.001
epoch_num = 4
batch_size = 128

# Initialization Model
model = Skip_Gram_Model(len(word2index),embedding_dim)
# Getting model to GPU
model.cuda()
# Define the loss function
criterion = nn.CrossEntropyLoss()
# use Adam optimizer
optimizer = optim.Adam(model.parameters(), lr=learning_rate)


loss_every_nsteps = 3000
total_loss = 0
start_time = time.time()
global_step = 0

for ep in range(epoch_num):
  for step, batch in enumerate(make_skip_gram_batches_iter(contexts, window_size=2, num_skips=4, batch_size=batch_size)):
      global_step += 1

      # Getting data to the GPU.
      tokens, labels = batch['tokens'].cuda(), batch['labels'].cuda()

      # make forward pass
      logits = model(tokens)

      # make backward pass
      loss = criterion(logits, labels)
      loss.backward()

      # apply optimizer
      optimizer.step()

      # zero grads
      optimizer.zero_grad()

      total_loss += loss.item()

      if global_step != 0 and global_step % loss_every_nsteps == 0:
          print("Epoch = {}, Step = {}, Avg Loss = {:.4f}, Time = {:.2f}s".format(ep, step, total_loss / loss_every_nsteps,
                                                                      time.time() - start_time))
          total_loss = 0
          start_time = time.time()

Initializing batches generator with 12289 batches per epoch
Epoch = 0, Step = 2999, Avg Loss = 7.2088, Time = 10.70s
Epoch = 0, Step = 5999, Avg Loss = 6.3028, Time = 7.49s
Epoch = 0, Step = 8999, Avg Loss = 6.1167, Time = 7.02s
Epoch = 0, Step = 11999, Avg Loss = 6.0377, Time = 9.27s
Initializing batches generator with 12289 batches per epoch
Epoch = 1, Step = 2710, Avg Loss = 5.9449, Time = 6.66s
Epoch = 1, Step = 5710, Avg Loss = 5.9208, Time = 6.38s
Epoch = 1, Step = 8710, Avg Loss = 5.8974, Time = 4.60s
Epoch = 1, Step = 11710, Avg Loss = 5.8994, Time = 5.07s
Initializing batches generator with 12289 batches per epoch
Epoch = 2, Step = 2421, Avg Loss = 5.8313, Time = 4.89s
Epoch = 2, Step = 5421, Avg Loss = 5.8260, Time = 4.60s
Epoch = 2, Step = 8421, Avg Loss = 5.8277, Time = 4.92s
Epoch = 2, Step = 11421, Avg Loss = 5.8241, Time = 4.64s
Initializing batches generator with 12289 batches per epoch
Epoch = 3, Step = 2132, Avg Loss = 5.7731, Time = 4.93s
Epoch = 3, Step = 5132, Avg Loss = 5.7685, Time = 4.93s
Epoch = 3, Step = 8132, Avg Loss = 5.7706, Time = 4.61s
Epoch = 3, Step = 11132, Avg Loss = 5.7783, Time = 5.31s

Obtaining word embeddings

Word embeddings are contained within the embeddings layer of the model. We just need to move them from the GPU to the CPU and convert them to a numpy array.

embeddings_skipgram = model.embeddings.weight.data.cpu().numpy()

Implementing a word similarity search algorithm

Let's check how adequate are similarities that the model learnt.

from sklearn.metrics.pairwise import cosine_similarity

def most_similar(embeddings, index2word, word2index, word):
    word_emb = embeddings[word2index[word]]

    similarities = cosine_similarity([word_emb], embeddings)[0]
    top10 = np.argsort(similarities)[-10:]

    return [index2word[index] for index in reversed(top10)]

most_similar(embeddings_skipgram, index2word, word2index, 'my')

['my', 'our', 'your', 'joe', 'his', 'ebay', 'her', 'phone.', 'myself', 'hell']

Visualization of word embeddings

import bokeh.models as bm, bokeh.plotting as pl
from bokeh.io import output_notebook

from sklearn.manifold import TSNE
from sklearn.preprocessing import scale


def draw_vectors(x, y, radius=10, alpha=0.25, color='blue',
                 width=600, height=400, show=True, **kwargs):
    """ draws an interactive plot for data points with auxilirary info on hover """
    output_notebook()

    if isinstance(color, str):
        color = [color] * len(x)
    data_source = bm.ColumnDataSource({ 'x' : x, 'y' : y, 'color': color, **kwargs })

    fig = pl.figure(active_scroll='wheel_zoom', width=width, height=height)
    fig.scatter('x', 'y', size=radius, color='color', alpha=alpha, source=data_source)

    fig.add_tools(bm.HoverTool(tooltips=[(key, "@" + key) for key in kwargs.keys()]))
    if show:
        pl.show(fig)
    return fig


def get_tsne_projection(word_vectors):
    tsne = TSNE(n_components=2, verbose=1)
    return scale(tsne.fit_transform(word_vectors))


def visualize_embeddings(embeddings, index2word, word_count):
    word_vectors = embeddings[1: word_count + 1]
    words = index2word[1: word_count + 1]

    word_tsne = get_tsne_projection(word_vectors)
    draw_vectors(word_tsne[:, 0], word_tsne[:, 1], color='blue', token=words)


visualize_embeddings(embeddings_skipgram, index2word, 100)

[t-SNE] Computing 91 nearest neighbors...
[t-SNE] Indexed 100 samples in 0.000s...
[t-SNE] Computed neighbors for 100 samples in 0.043s...
[t-SNE] Computed conditional probabilities for sample 100 / 100
[t-SNE] Mean sigma: 3.058861
[t-SNE] KL divergence after 250 iterations with early exaggeration: 61.748016
[t-SNE] KL divergence after 950 iterations: 0.601050

/usr/local/lib/python3.13/dist-packages/sklearn/preprocessing/_data.py:258: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.
  warnings.warn(
/usr/local/lib/python3.13/dist-packages/sklearn/preprocessing/_data.py:277: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0. 
  warnings.warn(

Advantages:

• Captures semantic relationships effectively.
• Efficient for large datasets.
• Provides meaningful word representations.

Disadvantages:

• May struggle with rare words.
• Ignores word order.

1.3 GloVe (Global Vectors for Word Representation)

Global Vectors for Word Representation (GloVe) is a powerful word embedding technique that captures the semantic relationships between words by considering their co-occurrence probabilities within a corpus. The key to GloVe’s effectiveness lies in the construction of a word-context matrix and the subsequent factorization process.

Let’s consider a simplified example. Assume we have the following sentences in our corpus:

“Word embeddings capture semantic meanings.”
“GloVe is an impactful word embedding model.”

The word-context matrix might look like this:

Here, each row and column corresponds to a unique word in the corpus, and the values in the cells represent how often these words appear together within a certain context window.

Advantages:

• Efficiently captures global statistics of the corpus.
• Good at representing both semantic and syntactic relationships.
• Effective in capturing word analogies.

Disadvantages:

• Requires more memory for storing co-occurrence matrices.
• Less effective with very small corpora.

import gensim.downloader as api
#GloVe
glove_model = api.load("glove-wiki-gigaword-300")

[==================================================] 100.0% 376.1/376.1MB downloaded

1.4 FastText

FastText is an advanced word embedding technique developed by Facebook AI Research (FAIR) that extends the Word2Vec model. Unlike Word2Vec, FastText not only considers whole words but also incorporates subword information — parts of words like n-grams. This approach enables the handling of morphologically rich languages and captures information about word structure more effectively.

FastText represents each word as a bag of character n-grams in addition to the whole word itself. This means that the word “apple” is represented by the word itself and its constituent n-grams like “ap”, “pp”, “pl”, “le”, etc. This approach helps capture the meanings of shorter words and affords a better understanding of suffixes and prefixes.

# FastText
fasttext_model = api.load("fasttext-wiki-news-subwords-300")

[==================================================] 100.0% 958.5/958.4MB downloaded

Advantages:

• Better representation of rare words.
• Capable of handling out-of-vocabulary words.
• Richer word representations due to subword information.

Disadvantages:

• Increased model size due to n-gram information.
• Longer training times compared to Word2Vec.

1.5 The comparison between Word2Vec,GloVe and FastText.

import numpy as np
import gensim.downloader as api
from sklearn.metrics.pairwise import cosine_similarity
import torch

This text includes:

• Common words (such as “quick”, “king”)
• Low-frequency words (such as “serendipity”)
• Out-of-vocabulary words (assuming “serendipity” has not appeared in the model before)
• Polysemous words (such as “king”, which can refer to a monarch or a chess piece)

# Example text
text = "The quick brown fox jumps over the lazy dog. The king rules the kingdom with wisdom. The word 'serendipity' is hard to spell."
words = ["quick", "king", "serendipity", "rules", "wisdom"]

# Get the word embeddings dictionary for SkipGram model
# Assuming embedding_skipgram is a torch tensor of shape (vocab_size, embedding_dim)
# word2index is a dictionary {word: index}, we can use it to get the word embeddings
skipgram_model = {word: embeddings_skipgram[idx] for word, idx in word2index.items()}

# Function to get word vectors
def get_vectors(model, words, model_type="skipgram"):
    vectors = []
    for word in words:
        if model_type == "skipgram":
            # For the SkipGram model, fetch the vector from the dictionary
            vector = model.get(word, None)  # Returns None if the word is not in the model
        else:
            try:
                vector = model[word]  # For GloVe and FastText
            except KeyError:
                vector = None  # Returns None if the word is not in the model
        vectors.append(vector)
    return vectors

# Function to compare cosine similarity between two vectors
def compare_similarity(vector1, vector2):
    if vector1 is None or vector2 is None:
        return None
    return cosine_similarity([vector1], [vector2])[0][0]

# Get word vectors for SkipGram, GloVe, and FastText models
skipgram_vectors = get_vectors(skipgram_model, words, model_type="skipgram")
glove_vectors = get_vectors(glove_model, words, model_type="glove")
fasttext_vectors = get_vectors(fasttext_model, words, model_type="fasttext")

Skip-gram return None because they are unable to handle out-of-vocabulary (OOV) words.

FastText, on the other hand,return a vector because it can handle OOV words by leveraging subword information.

# Compare the handling of low-frequency and out-of-vocabulary words：serendipity
serendipity_skipgram = skipgram_vectors[2]
serendipity_glove = glove_vectors[2]
serendipity_fasttext = fasttext_vectors[2]
print("Skip-gram (serendipity):", serendipity_skipgram)
print("GloVe (serendipity):", serendipity_glove)
print("FastText (serendipity):", serendipity_fasttext)

Skip-gram (serendipity): None
GloVe (serendipity): [ 0.18632   -0.34564    0.08389   -0.0088622  0.29216   -0.30842
  0.1446    -0.28246   -0.3814     0.17972    0.56072   -0.061665
 -0.14916    0.5461    -0.18552    0.63468   -0.09079   -0.47448
 -0.090303   0.77264   -0.39216    0.21172   -0.64588   -0.35832
  0.1424     0.68727    0.094841   0.10264   -0.20581    0.13369
  0.1335    -0.32181   -0.18035    0.060461   0.22884    0.25557
 -0.34548    0.18665    0.41418   -0.31108    0.3044     0.046528
 -0.021619   0.029792  -0.1386     0.016738   0.76186    0.082857
 -0.52825   -0.088613   0.0039659 -0.16441    0.049685   0.32602
 -0.13568   -0.0074606  0.22342    0.0017904  0.26966   -0.056631
  0.095477   0.15218    0.13187   -0.55669    0.55745    0.25169
  0.33585   -0.38694   -0.44972    0.3091    -0.069153   0.14882
  0.38556    0.1067    -0.31081    0.17787    0.23894    0.0087298
  0.57065   -0.29811   -0.2857     0.38242    0.056624   0.35403
  0.32367    0.23742    0.14528    0.29852   -0.0033725 -0.026211
 -0.17575   -0.29027    0.20888   -0.13644    0.44249   -0.0013981
 -0.11947   -0.097379  -0.17901    0.32461    0.85651   -0.16494
  0.21082   -0.029599  -0.28011    0.11902   -0.2108    -0.29511
 -0.22094   -0.10521    0.10531    0.37363    0.18809   -0.26492
  0.15593    0.14356   -0.17867   -0.41704   -0.31694    0.51266
 -0.23342    0.18465    0.26372   -0.0042298  0.15957   -0.10954
  0.42556    0.47028   -0.24178    0.80037    0.22366    0.0012861
  0.049398  -0.36584   -0.29655    0.14908    0.27978    0.55654
 -0.54622   -0.1923     0.060863   0.017136  -0.31743    0.36479
  0.011946  -0.4183    -0.42232    0.30339   -0.23194   -0.039143
 -1.283     -0.043825   0.37935   -0.14484   -0.40778   -0.2479
  0.44472    0.97023   -0.007969  -0.28284    0.1909    -0.29943
 -0.22702    0.29905    0.60438    0.27322    0.26682    0.33175
  0.35046    0.023653  -0.23109    0.27642   -0.12867    0.14209
 -0.23383   -0.50974   -0.61034   -0.081647  -0.29914   -0.30302
  0.67571   -0.075578  -0.18453   -0.32562   -0.071593   0.010142
  0.093097  -0.34611    0.61285   -0.16049   -0.54084   -0.3363
  0.53979   -0.4669     0.1916    -0.20776   -0.13912    0.048965
 -0.14741   -0.11732   -0.0094399 -0.25253    0.12036    0.33808
  0.09632    0.28489    0.11688    0.043861   0.44266   -0.36999
 -0.42889    0.45218    0.32458    0.33865   -0.28339    0.096947
  0.15864   -0.2372    -0.10074   -0.042115  -0.74153    0.46197
 -0.40608    0.55082   -0.32483   -0.47397   -0.014787  -0.20286
 -0.0070767  0.12768    0.19346   -0.038147  -0.18549   -0.3642
 -0.0040809  0.22498    0.39486    0.082756  -0.022859  -0.75962
  0.48698    0.033705  -0.049857   0.058969  -0.3512    -0.77932
 -0.49243    0.49367   -0.022767   0.10698   -0.17816   -0.026324
  0.32856   -0.30507   -0.4945    -0.1067     0.32769   -0.086502
  0.34027    0.040302   0.26741   -0.32661    0.60733    0.093418
  0.37058    0.0028669 -0.070005  -0.38477    0.35507   -0.39972
 -0.2046    -0.3511    -0.13235   -0.049713   0.18461    0.54833
  0.94674   -0.44362   -0.17485   -0.15367   -0.56011    0.34371
  0.21185    0.24543    0.19474   -0.56808   -0.4943    -0.77071
  0.1845    -0.04606    0.21294   -0.13378   -0.0036589 -0.02757
  0.02645   -0.10409   -0.16088    0.33948   -0.013567   0.24813  ]
FastText (serendipity): [-2.9805e-02  2.6854e-02 -3.1424e-02  7.5323e-03 -1.1384e-01 -2.7078e-02
 -4.7518e-03 -6.6480e-02  3.4808e-02  9.1730e-04 -6.0305e-02  3.1930e-02
  1.0609e-02 -1.0736e-02 -1.0970e-02  3.4149e-02  5.1842e-02  5.4079e-02
 -8.7730e-03  2.2972e-03 -2.0631e-02  7.7932e-02 -3.2296e-02  2.5283e-02
 -2.7547e-02 -3.4044e-02  2.4507e-02  4.0076e-02  4.2797e-02  2.0118e-02
  3.7017e-02 -1.1477e-02  2.0691e-02 -3.1354e-02  1.2515e-02  9.5467e-03
 -6.2899e-03 -1.3610e-02  4.1874e-02  1.1588e-04 -1.8748e-02 -4.4494e-02
 -1.0173e-01  3.3002e-02  4.5566e-02 -4.5788e-03  7.3325e-02 -3.9758e-02
 -1.0988e-02 -8.0551e-03  1.8704e-02 -3.5747e-02  4.9400e-03 -3.4726e-02
  8.6243e-03 -8.4493e-02  7.9489e-02  4.9694e-02 -4.1716e-02  2.1004e-03
 -2.1612e-03  4.6965e-03 -9.1081e-03 -1.3131e-02  4.2662e-02 -4.9487e-03
 -1.2876e-02 -1.0012e-03 -2.9265e-02  3.4636e-02  6.0778e-02  6.5794e-02
 -1.5603e-02 -2.8355e-02  6.6601e-02  1.1085e-02 -5.9065e-02 -3.1554e-02
  2.2983e-02  6.7906e-03 -8.8510e-02 -1.7605e-02  4.4317e-02  8.4607e-02
  3.5877e-02 -5.1026e-02  4.0443e-02  1.8851e-02  7.5502e-02 -4.3072e-03
  1.6037e-02 -2.0483e-02 -7.2615e-02 -9.3846e-02 -8.7600e-03  2.6943e-02
  4.2688e-02  3.6502e-02 -2.9784e-02  3.2130e-02  1.6471e-02  4.0318e-02
 -4.6387e-02 -1.4238e-02 -1.9918e-03 -1.7839e-02  1.9834e-02  5.4533e-02
  5.9133e-02 -3.2239e-02 -1.6680e-02  6.6877e-02 -8.8362e-03  5.6069e-02
 -2.8538e-02  9.4626e-02 -7.0414e-05  3.2430e-02 -7.4438e-02  1.1245e-02
  6.6492e-03 -5.6852e-02 -4.7480e-02 -3.7671e-04  5.1336e-02 -2.6837e-03
 -1.1159e-02 -4.1757e-02 -2.7279e-02  7.9218e-02  6.2733e-02 -4.2663e-02
 -4.7548e-02  6.1073e-06 -6.9286e-03 -1.5168e-04  5.2172e-02  3.4323e-02
  1.0259e-02 -3.5228e-02  3.7493e-02 -2.4893e-02 -1.6201e-02  5.3851e-02
 -5.0565e-02 -1.2669e-02 -4.6908e-02 -2.3780e-02  1.6584e-03  1.8918e-02
  3.3723e-02  1.3831e-02  4.2392e-02 -2.0058e-02  4.3174e-02  2.2994e-02
  2.8830e-02  1.7093e-02 -6.4822e-03 -5.9819e-03  3.9156e-02  5.5234e-02
 -5.8772e-02 -3.1360e-02 -3.3978e-03  3.8582e-02  2.5142e-02 -2.9289e-02
  9.6138e-03  2.7690e-02  6.5373e-02  3.6202e-02 -4.3323e-02 -8.5680e-03
 -3.0314e-03  3.9565e-02  1.7750e-03  4.3692e-02 -3.9007e-02 -8.9581e-03
 -2.0762e-02 -6.3132e-02 -1.8234e-02 -7.9662e-02 -7.5792e-02 -3.1938e-02
  7.7712e-02  7.4359e-02 -1.6705e-02  1.0577e-02  7.7386e-03  1.0373e-02
 -3.8770e-02 -1.4657e-02 -8.5282e-02  3.0172e-02  9.8768e-03  3.7279e-02
 -8.7546e-03  6.1621e-02 -8.5347e-02  7.1000e-02  1.8729e-02  1.6303e-02
 -4.0717e-02  7.5443e-02 -5.3836e-02 -1.7919e-02 -1.1262e-03 -2.0924e-02
  2.3430e-02  2.0931e-02  6.0641e-03 -6.1802e-03  1.0194e-01  7.0412e-03
 -3.5549e-02 -1.7181e-02  7.6885e-03 -3.9319e-02  2.2079e-02 -1.7141e-02
 -4.0551e-02  7.7523e-02  7.5541e-02  5.4512e-03  8.8621e-02  6.5679e-02
  1.8234e-02 -2.1421e-02 -4.9822e-02 -1.0807e-01 -5.1674e-02  1.7589e-02
  7.8029e-02  1.8037e-02 -1.9322e-02 -4.7413e-02 -3.1870e-02 -4.4834e-02
  7.2774e-02 -6.6183e-03  2.9034e-02 -1.8229e-03 -5.7142e-02 -5.4412e-02
 -5.0543e-02 -5.3607e-02  1.7995e-02  2.2765e-02  3.2847e-02 -1.8328e-02
 -3.0570e-04 -4.5240e-02  1.4371e-02  2.3070e-02 -3.0135e-02  1.2707e-02
  5.5695e-02  6.1089e-02  2.2758e-02 -1.0575e-02 -3.6754e-02  2.8981e-02
 -3.8409e-02 -3.8485e-02 -1.7823e-02  6.9334e-03  1.5735e-02  4.3667e-02
 -2.8411e-02  6.2190e-02 -8.0278e-02 -2.5576e-02  1.4574e-02 -7.5652e-02
  1.4334e-02  8.1623e-04 -6.2756e-02 -2.1008e-02  1.6251e-02 -1.0859e-01
  3.5296e-02 -2.7259e-02 -1.2488e-02 -4.4157e-02 -3.2562e-02  4.5759e-02
 -6.5838e-02 -7.5107e-03  1.8531e-02  3.0088e-02 -2.9809e-03  1.7422e-02
  4.6868e-02  7.7602e-03  8.7317e-02 -4.0683e-02  8.7127e-03  6.0970e-02]

# Compare the representation of polysemy (e.g., "king")
king_skipgram = skipgram_vectors[1]
king_glove = glove_vectors[1]
king_fasttext = fasttext_vectors[1]

print("Skip-gram (king):", king_skipgram)
print("GloVe (king):", king_glove)
print("FastText (king):", king_fasttext)

Skip-gram (king): [-2.2188125  -1.0111594   1.0707227   0.4085443   0.14322577 -0.64025384
 -0.4076173   0.90832573 -0.45077094  1.1945928  -0.04054052 -0.6113147
  0.1833278   1.5405859   1.3805898  -0.503673    0.05243981  1.0274988
 -1.2985619   0.5218746   2.6140766  -0.17210904 -0.00406189  0.74075514
 -0.9066404   0.6351083   0.05806823 -1.6418552   1.6289414   1.1448437
  1.1084406  -2.0645926 ]
GloVe (king): [ 0.0033901 -0.34614    0.28144    0.48382    0.59469    0.012965
  0.53982    0.48233    0.21463   -1.0249    -0.34788   -0.79001
 -0.15084    0.61374    0.042811   0.19323    0.25462    0.32528
  0.05698    0.063253  -0.49439    0.47337   -0.16761    0.045594
  0.30451   -0.35416   -0.34583   -0.20118    0.25511    0.091111
  0.014651  -0.017541  -0.23854    0.48215   -0.9145    -0.36235
  0.34736    0.028639  -0.027065  -0.036481  -0.067391  -0.23452
 -0.13772    0.33951    0.13415   -0.1342     0.47856   -0.1842
  0.10705   -0.45834   -0.36085   -0.22595    0.32881   -0.13643
  0.23128    0.34269    0.42344    0.47057    0.479      0.074639
  0.3344     0.10714   -0.13289    0.58734    0.38616   -0.52238
 -0.22028   -0.072322   0.32269    0.44226   -0.037382   0.18324
  0.058082   0.26938    0.36202    0.13983    0.016815  -0.34426
  0.4827     0.2108     0.75618   -0.13092   -0.025741   0.43391
  0.33893   -0.16438    0.26817    0.68774    0.311     -0.2509
  0.0027749 -0.39809   -0.43399    0.049531  -0.42686   -0.094679
  0.56925    0.28742   -0.015721  -0.059162   0.1912    -0.59814
  0.65486   -0.31363    0.16881    0.10862    0.075316   0.34093
 -0.14706    0.8359     0.39697    0.52358   -0.0096367 -0.14406
  0.37783   -0.596     -0.063192  -0.85297   -0.3098    -1.0587
 -1.025      0.4508    -0.73324   -1.2461    -0.028488   0.20299
  0.00259    0.31995    0.35744    0.28533    0.228      0.50956
 -0.35942    0.32683    0.046264  -0.86896   -0.2707    -0.15454
 -0.32152    0.31121    0.44134    0.85189    0.21065   -0.13741
 -0.15359   -0.059722   0.027375   0.23724   -0.39197   -0.66065
  0.23587    0.032384  -0.64043    0.55004    0.29597    0.14989
  0.46079   -0.26561   -0.1607    -0.36328    1.0782     0.31375
  0.1149     0.20248    0.032748   0.41082   -0.082536   0.36606
  0.18771    0.75415    0.079648   0.24181   -0.60319   -0.37296
 -0.047767   0.45008   -0.21135    0.022251  -0.084325   0.18644
 -0.14682    0.56571   -0.30995    0.17423   -0.41122   -0.84772
 -0.71114    0.69895   -0.13008   -0.34195   -0.30501   -0.12646
  0.29957   -0.43488    0.31935    0.2817    -0.20631   -0.48877
  0.34477    0.03907    1.6198    -0.6352    -0.0037675 -0.41271
  0.30704   -0.50486    0.036385  -0.046386  -0.12004    0.010029
 -0.49116    0.041486   0.002979  -0.57694   -0.42088   -0.063218
  0.0034244 -0.25093   -0.39689   -0.36984    0.32689    0.01385
  0.23634   -0.055199  -0.58453    0.13211    0.50943    0.25198
 -0.0088309 -0.21273   -0.48423    0.5234    -0.32832   -0.013821
  0.15812    0.46696    0.036822  -0.090878   0.18854    0.20794
 -0.42682    0.59705    0.53109    0.19185   -0.16392    0.064956
 -0.36009   -0.59882   -0.28134    0.1017     0.02601    0.44298
 -0.31922   -0.22432    0.7828     0.041307   0.1742     0.27777
  0.43792   -0.84324    0.27012   -0.21547    0.52408   -0.19426
 -0.21878   -0.20713    0.092994  -0.15804    0.28716   -0.11911
 -0.20688   -0.36482    0.68548   -0.10394   -0.49974   -0.47038
 -1.2953    -0.46236    0.44467    0.13337    0.88762   -0.26494
  0.080676  -0.20625   -0.51232    0.31112    0.062035   0.30302
 -0.33344   -0.20924   -0.17348   -0.43434   -0.45743   -0.077803
 -0.33248   -0.078633   0.82182    0.082088  -0.68795    0.30266  ]
FastText (king): [-1.2063e-01  5.1695e-03 -1.2447e-02 -7.8528e-03 -2.3738e-02 -8.2595e-02
  4.5790e-02 -1.5382e-01  6.4550e-02  1.2893e-01  2.7643e-02  1.5958e-02
  7.7559e-02  6.0516e-02  1.2737e-01  8.4766e-02  6.3890e-02 -1.7687e-01
  4.3017e-02 -1.8031e-02 -3.3041e-02  2.1930e-02 -1.1328e-02  6.6453e-02
  1.5826e-01 -2.3008e-02 -4.3616e-03 -2.2379e-02  4.4891e-02  3.0103e-03
 -1.5565e-02 -7.6785e-02 -9.2186e-02  5.7907e-02 -2.7658e-02  5.4500e-03
  1.8975e-02  4.2939e-02  3.4704e-03  4.0449e-02 -4.0245e-03 -1.1594e-01
 -5.8337e-03  3.2509e-02 -8.6535e-02  7.2000e-02 -2.2299e-02  1.3079e-02
 -3.9515e-02  6.8996e-02  9.2300e-02 -7.5371e-02  5.9412e-03 -3.4945e-02
 -3.3417e-02 -9.9982e-02  1.6438e-02  6.3739e-02 -6.2391e-02  7.8285e-04
 -2.9210e-02 -9.6416e-02  7.2910e-02  4.5905e-02 -8.3387e-02  7.1969e-02
  4.0932e-02 -5.6454e-03  1.3709e-01 -1.1793e-01 -7.1011e-02 -7.1963e-02
  6.5600e-02 -4.6315e-02 -1.7200e-02  3.4434e-02  4.4218e-02 -9.6354e-03
 -6.8105e-02  3.0810e-02  1.5424e-02  5.6398e-02  4.4225e-02  8.0547e-02
 -5.2413e-02 -3.6509e-02  2.6141e-02  2.5574e-02 -3.4346e-02 -4.5879e-02
 -1.7031e-02  5.1450e-02 -1.2766e-01 -8.6838e-02  1.1084e-02  1.3282e-01
  2.0850e-02  7.0881e-02 -5.9277e-03  2.2612e-02  4.8919e-02 -1.2490e-02
  1.5460e-01 -6.1251e-03 -8.9369e-02 -2.3707e-01  2.0696e-02 -3.7604e-02
 -8.3793e-02 -2.5512e-03 -4.0426e-02  1.0575e-01  9.7514e-02  4.4101e-02
  4.1732e-02  7.4080e-02  6.3560e-02  3.1801e-02 -1.4961e-02 -4.3675e-03
 -1.4893e-02  8.6208e-02 -2.0204e-02 -2.0797e-03  7.7648e-02 -1.9620e-03
  3.2115e-02 -1.5615e-01 -3.6702e-02  1.2009e-01 -8.0633e-02  4.2894e-02
 -3.5265e-02  2.2693e-02 -3.3743e-02  1.7573e-02 -7.5089e-02  9.8873e-02
  2.7042e-02 -1.7185e-02  1.7489e-02 -1.1096e-01  7.5456e-02 -4.2234e-02
 -3.7115e-02 -1.2356e-02  1.1243e-02 -4.6907e-02 -5.5681e-02 -6.5216e-02
  5.4923e-02  3.7514e-02  5.0259e-02 -7.4453e-02 -2.0440e-02 -8.3293e-02
 -2.3010e-02 -4.2105e-02 -2.8792e-02 -1.9139e-02  3.6758e-02  7.7620e-02
 -6.3909e-02 -2.9304e-02  3.1128e-02 -1.2056e-02 -3.0854e-02 -2.3162e-02
 -4.4762e-02  1.2797e-01 -7.7709e-03 -7.7466e-02 -2.7976e-02  5.1038e-02
 -5.5217e-02  7.5312e-02  3.4093e-02 -3.4833e-03  9.7360e-03  5.8273e-02
  9.3454e-02 -4.3781e-02 -4.5870e-02 -7.3544e-02 -4.1269e-02 -9.1712e-02
 -1.5840e-01  1.1790e-01  3.4210e-02 -2.4719e-02  6.1251e-02  8.2068e-02
 -1.1710e-01  2.9949e-02 -7.1442e-02  2.2185e-02 -2.4418e-02 -2.5316e-02
 -5.3970e-02  1.1615e-01 -1.9979e-01  6.8714e-02 -6.1776e-03 -3.9478e-02
 -1.8856e-02  7.8819e-02  3.0709e-02 -4.7448e-02 -5.0356e-02 -4.0706e-02
  1.4722e-01 -4.6420e-02  1.1976e-05  9.2290e-02 -6.1358e-02  6.0161e-05
  1.4491e-02 -2.4847e-02  5.6051e-02  1.9206e-02  3.2446e-02  5.0245e-03
  1.9242e-02  1.3482e-01  7.3311e-03 -1.0219e-01  7.6724e-02  9.7512e-02
 -4.9655e-02 -7.2788e-03 -1.1748e-01 -3.5783e-02 -6.9954e-02 -8.8086e-03
 -1.5677e-02  6.4489e-02 -7.2463e-02 -5.0428e-03  7.5461e-02 -6.0999e-02
  9.2653e-02 -5.3002e-02 -9.8853e-02  4.4468e-02  1.5699e-03  1.0594e-02
  5.4306e-02  2.1943e-02 -1.4941e-02 -2.9272e-02  1.0173e-01 -2.7459e-02
 -1.7016e-02  3.7454e-02  8.5015e-02  8.6834e-02 -7.6342e-02  9.5069e-02
  4.6912e-02 -2.2718e-02 -7.9839e-02  6.6125e-02  6.2540e-02  2.5836e-02
  2.4580e-02  5.1879e-02 -1.8032e-04  4.8657e-02 -1.1875e-01 -2.4103e-02
  1.5130e-03  8.0515e-02 -1.0280e-01 -1.3489e-02  7.1108e-02 -6.0643e-02
 -2.3006e-02 -9.8232e-03 -8.7159e-02  8.5388e-02  5.3778e-02 -8.4714e-02
  5.4218e-02 -4.1406e-02  1.0716e-02  6.9728e-02 -8.9833e-03 -8.0539e-02
 -3.0566e-02  1.0912e-01 -3.9061e-02 -6.3893e-02 -3.3986e-02 -2.0095e-02
 -6.0904e-02  1.5957e-02 -1.0371e-02  6.7261e-02 -3.0458e-02 -3.1992e-02]

# Compare the ability to capture contextual relationships
queen_skipgram = skipgram_model.get("queen", None)
queen_glove = glove_model["queen"]
queen_fasttext = fasttext_model["queen"]

king_queen_skipgram = compare_similarity(king_skipgram, queen_skipgram)
king_queen_glove = compare_similarity(king_glove, queen_glove)
king_queen_fasttext = compare_similarity(king_fasttext, queen_fasttext)

print("Skip-gram (king vs queen):", king_queen_skipgram)
print("GloVe (king vs queen):", king_queen_glove)
print("FastText (king vs queen):", king_queen_fasttext)

from sklearn.metrics.pairwise import cosine_similarity

Skip-gram (king vs queen): 0.032229178
GloVe (king vs queen): 0.63364697
FastText (king vs queen): 0.7704246

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA

# List of words to analyze
words = ['king', 'queen', 'man', 'woman', 'prince', 'princess']

# Get word vectors from GloVe model
glove_vectors = [glove_model[word] for word in words]

# Get word vectors from FastText model
fasttext_vectors = [fasttext_model[word] for word in words]

# Get word vectors from Skip-gram model, skipping words not found in the model
skipgram_vectors = []
for word in words:
    if word in skipgram_model:
        skipgram_vectors.append(skipgram_model[word])
    else:
        skipgram_vectors.append(np.zeros(300))  # Use a zero vector if word not found

# Normalize all vectors to the same size for PCA (e.g., by truncating or padding)
max_len = max([len(vec) for vec in glove_vectors + fasttext_vectors + skipgram_vectors])
glove_vectors = [np.pad(vec, (0, max_len - len(vec))) for vec in glove_vectors]
fasttext_vectors = [np.pad(vec, (0, max_len - len(vec))) for vec in fasttext_vectors]
skipgram_vectors = [np.pad(vec, (0, max_len - len(vec))) for vec in skipgram_vectors]

# Reduce the dimensionality of word vectors to 2D using PCA
pca = PCA(n_components=2)
glove_pca = pca.fit_transform(glove_vectors)
fasttext_pca = pca.fit_transform(fasttext_vectors)
skipgram_pca = pca.fit_transform(skipgram_vectors)

# Visualize the word embeddings
plt.figure(figsize=(10, 8))

# Plot GloVe word vectors
plt.scatter(glove_pca[:, 0], glove_pca[:, 1], label='GloVe', color='blue')
for i, word in enumerate(words):
    plt.text(glove_pca[i, 0], glove_pca[i, 1], word, color='blue', fontsize=12)

# Plot FastText word vectors
plt.scatter(fasttext_pca[:, 0], fasttext_pca[:, 1], label='FastText', color='green')
for i, word in enumerate(words):
    plt.text(fasttext_pca[i, 0], fasttext_pca[i, 1], word, color='green', fontsize=12)

# Plot Skip-gram word vectors
plt.scatter(skipgram_pca[:, 0], skipgram_pca[:, 1], label='Skip-gram', color='red')
for i, word in enumerate(words):
    plt.text(skipgram_pca[i, 0], skipgram_pca[i, 1], word, color='red', fontsize=12)

# Set plot attributes
plt.legend()
plt.title('Word Embeddings Comparison (GloVe, FastText, Skip-gram)')
plt.grid(True)
plt.show()

GloVe (in blue): The word vectors from the GloVe model are more dispersed in the diagram but still show some semantic relevance. For instance, "man" and "woman" are relatively close in space, indicating a semantic relationship based on gender. "King" and "queen" as well as "prince" and "princess" also show proximity, suggesting their semantic relatedness.

FastText (in green): The word vectors from the FastText model are clustered more tightly together in the diagram, demonstrating closer semantic relationships. For example, "man" and "woman" as well as "king" and "queen" are much closer in distance, indicating stronger semantic proximity. This tight clustering may suggest that FastText performs better at capturing subtle differences between words since it considers subword information.

Skip-gram (in red): The word vectors from the Skip-gram model are more dispersed but still exhibit some semantic relevance. "Man" and "woman" as well as "king" and "queen" are relatively close in space. Compared to GloVe and FastText, the distribution of Skip-gram's word vectors might rely more on contextual information, and thus, it might not perform as well as the other models in capturing semantic relationships in some cases.

Overall, all three models have the capability to capture semantic relationships between words, but they differ in the distribution and clustering of word vectors. GloVe and Skip-gram have more dispersed word vector distributions, while FastText's vectors are more tightly clustered, showing stronger semantic relevance. These differences may be related to the training methods and the way word vectors are generated by each model.

1.6 Comparison of Memory and Computational Overheads: Word2Vec, GloVe, and FastText

---

1. Memory Overhead

(1) Word2Vec (Skip-Gram)

• Parameters: Only the embedding matrix.
  $\text{Memory} = V \times d$
  • $V$: Vocabulary size, $d$: Embedding dimension.

(2) GloVe

• Parameters: Embeddings + biases + co-occurrence matrix (preprocessing).
  $\text{Memory} = 2 \times V \times d + 2 \times V + \text{Co-occurrence Matrix}$
  • Co-occurrence matrix uses sparse storage (e.g., hash tables).

(3) FastText

• Parameters: Subword embeddings + word embeddings.
  $\text{Memory} = (V + S) \times d$
  • $S$: Number of subwords (e.g., 3-6 character n-grams).

---

2. Computational Overhead

(1) Word2Vec (Skip-Gram)

• Complexity:
  $O(T \times (C + K) \times d)$
  • $T$: Total tokens, $C$: Window size, $K$: Negative samples.

(2) GloVe

• Complexity:
  $O(N \times d)$
  • $N$: Non-zero co-occurrence pairs.

(3) FastText

• Complexity:
  $O(T \times (B + C) \times d)$
  • $B$: Subwords per word (typically 3-6).

---

3. Example with Parameters

Assume:

• $V = 10^4$, $d = 300$, $T = 10^6$, $C = 5$, $K = 5$, $S = 10^5$, $N = 10^7$.

Memory Overhead

Model	Calculation	Memory (float32)
Word2Vec	$10^4 \times 300$	11.4 MB
GloVe	$2 \times 10^4 \times 300 + 2 \times 10^4 + 10^7$	23.6 MB
FastText	$(10^4 + 10^5) \times 300$	125.8 MB

Computational Overhead

Model	Calculation	Operations
Word2Vec	$10^6 \times (5 + 5) \times 300$	$3 \times 10^9$
GloVe	$10^7 \times 300$	$3 \times 10^9$
FastText	$10^6 \times (5 + 5) \times 300$	$3 \times 10^9$

---

4. Summary

Metric	Word2Vec	GloVe	FastText
Memory	Lowest (11.4 MB)	Moderate (23.6 MB)	Highest (125.8 MB)
Computation	Moderate	Moderate	Highest
Use Case	General semantics	Co-occurrence	Morphology-rich languages

• Memory-sensitive: Prefer Word2Vec or GloVe.
• Morphology-rich languages: FastText excels.
• Balanced choice: GloVe for stable performance.