Large Language Model

Map > Data Science > Predicting the Future > Modeling > Classification > Large Language Model

Large Language Model

At its simplest, a Large Language Model (LLM) does this: "given the words so far, predict what comes next". For example:

The patient has high blood

The model might assign probabilities:

pressure 0.72
sugar 0.08
cholesterol 0.04
count 0.01
...

It selects pressure, producing:

The patient has high blood pressure

Then it predicts the next token again. So fundamentally:

LLM = next-token prediction + a very large neural network trained on enormous amounts of text

Our first model will be deliberately a tiny LLM.

Vocabulary
↓
Token IDs
↓
Embedding
↓
Neural Network
↓
Next-token probabilities

For example, suppose our vocabulary is:

["I", "am", "a", "doctor", "scientist", "."]

We convert:

"I am a doctor"

into:

[0, 1, 2, 3]

The numbers themselves don't mean anything. They are simply IDs.

1- Create a training data

Suppose our training text is:

I am a scientist.
I am a doctor.
I like data.
I like science.

The model learns examples such as:

Input	Target
I	am
I am	a
I am a	scientist
I am a scientist	.
I am a	doctor
I am a doctor	.
I like	data
I like data	.

The important concept is: The target is the next token.

2- Create a tokenizer

For our first model, we'll use a very simple word tokenizer.

Python

text = """
I am a scientist .
I am a doctor .
I like data .
I like science .
"""
words = text.split()
vocab = sorted(set(words))
word_to_id = {word: i for i, word in enumerate(vocab)}
id_to_word = {i: word for word, i in word_to_id.items()}
print(vocab)
print(word_to_id)

You might get:

['.', 'I', 'a', 'am', 'data', 'doctor', 'like', 'science', 'scientist']
{'.': 0, 'I': 1, 'a': 2, 'am': 3, 'data': 4, 'doctor': 5, 'like': 6, 'science': 7, 'scientist': 8}

Now:

Python

sentence = "I am a doctor ."
tokens = sentence.split()
ids = [word_to_id[word] for word in tokens]
print(ids)

For example:

[1, 3, 2, 5, 0]

3- Create input/target pairs

Python

import torch

encoded = torch.tensor([word_to_id[word] for word in words], dtype=torch.long)
X = encoded[:-1]
y = encoded[1:]
print("Input:", X)
print("Target:", y)

If the text is:

I am a scientist .

We have:

X:
I am a scientist

y:
am a scientist .

So the model learns:

I → am
am → a
a → scientist
scientist → .

4- The embedding layer

This is one of the most important ideas. Instead of representing:

doctor = 5
scientist = 8
data = 4

as meaningless numbers, we represent each word with a vector. For example:

doctor → [0.21, -0.43, 0.82, 0.17]
scientist → [0.19, -0.39, 0.77, 0.22]
data → [-0.51, 0.18, 0.32, 0.71]

The model learns these vectors during training.

Python

embedding = torch.nn.Embedding(num_embeddings=len(vocab), embedding_dim=16)

x = torch.tensor([word_to_id["doctor"]])
vector = embedding(x)
print(vector)

Now every word (e.g., doctor) becomes a 16-dimensional vector.

[-0.3165, -2.1291, 0.2706, 0.3934, -0.3485, 1.9182, 0.8745, 0.2270,
-0.7873, -0.9977, -1.1800, -0.0366, 0.2482, -0.8141, -0.0971, -0.5478]

5- Our first language model

Let's create a very small neural network.

Token
↓
Embedding
↓
Linear layer
↓
Vocabulary probabilities

Python

import torch
import torch.nn as nn

class TinyLLM(nn.Module):

   def __init__(self, vocab_size, embedding_dim):
      super().__init__()
      self.embedding = nn.Embedding(vocab_size, embedding_dim)
      self.linear = nn.Linear(embedding_dim, vocab_size)

   def forward(self, x):
      x = self.embedding(x)
      x = self.linear(x)
      return x

6- Train the model

We need a loss function. The standard loss for language modeling is Cross-Entropy Loss.

Python

model = TinyLLM(vocab_size=len(vocab), embedding_dim=32)

loss_function = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

Now train:

Python

for epoch in range(1000):
   optimizer.zero_grad()
   output = model(X)
   loss = loss_function(output, y)
   loss.backward()
   optimizer.step()

   if epoch % 100 == 0:
      print("Epoch:",epoch,"Loss:",loss.item())

You should see the loss generally decrease.

Epoch: 0 Loss: 2.492607355117798
Epoch: 100 Loss: 0.32899045944213867
Epoch: 200 Loss: 0.32743334770202637
...

7- Ask model to predict

Now give it a word.

Python

word = "I"

x = torch.tensor([word_to_id[word]])

with torch.no_grad():
output = model(x)
probabilities = torch.softmax( output, dim=-1)

print(probabilities)

Find the most likely word:

Python

prediction = torch.argmax(probabilities, dim=-1)
print(id_to_word[prediction.item()])

Hopefully:

You have built a tiny language model.

8- Give the model a context window

A real LLM needs to understand a sequence. For example:

The dog chased the cat because it was hungry.

The meaning of it depends on the surrounding context.

This is where the architecture becomes much more interesting. Instead of:

I → am

We train:

I am → a

am a → scientist

a scientist → .

For a context length of 3:

Input	Target
I am a	scientist
am a scientist	.
I am a	doctor
am a doctor	.
I like data	.

9- The missing part: Attention

This is the breakthrough behind modern LLMs.

Suppose the model receives:

The patient took aspirin because he had chest pain

When predicting something later, the model needs to decide: Which previous words are important?

Attention allows the model to calculate relationships such as:

patient ←→ he
aspirin ←→ took
chest ←→ pain

Instead of treating every previous token equally. Conceptually, The model learns which tokens should pay attention to which other tokens.

10- Transformers

Modern LLMs are essentially stacks of Transformer blocks. A simplified LLM looks like:

TEXT
↓
Tokenizer
↓
Token IDs
↓
Embeddings
↓
┌─────────────────────┐
Transformer Block

Self-Attention
↓
Feed Forward
↓
Layer Normalization
└─────────────────────┘
↓
repeat N times
↓
Linear layer
↓
Vocabulary logits
↓
Next-token prediction

GPT-type models are essentially very large versions of this idea, with many engineering improvements.

Let's build a complete tiny GPT-style Transformer in Python, train it on a small English text, and then generate new text. The goal is educational: you'll be able to see exactly how tokenization → embeddings → attention → transformer → prediction → generation works.