Map > Data Science > Predicting the Future > Modeling > Classification > Large Language Model > Embeddings
 

Embeddings: How LLMs Represent Meaning
An embedding converts an object (text, image, sound) into a vector of numbers that represents its meaning or characteristics. For example:
Text Embedding (simplified)
"cat" [0.82, 0.15, -0.32, 0.71]
"kitten" [0.79, 0.18, -0.30, 0.69]
"dog" [0.76, 0.21, -0.25, 0.65]
"car" [-0.12, 0.84, 0.71, -0.20]
These numbers are not manually assigned. A neural network learns them from large amounts of data.

 The important idea is: Similar meanings → vectors that are close together
Why do we need embeddings?
Consider this search: "automobile"

"I purchased a car."
"The vehicle was expensive."
"I bought a new automobile."
A traditional keyword search may only find:

"I bought a new automobile."
because it contains the word automobile. However, an embedding-based search can recognize that:

automobile ≈ car ≈ vehicle
and therefore return all three. This is called semantic search.
Embeddings and distance
Once text has been converted to vectors, we can calculate how similar two vectors are. A common measure is cosine similarity. For two vectors:

A = [1, 2, 3]
B = [1, 2, 2]
cosine similarity is:
   
The result is between approximately:

-1 → completely opposite
0 → unrelated
+1 → very similar
For text embeddings, we generally want semantically similar sentences to have a high cosine similarity.
 
Python example � no neural network yet
For our first model, we'll use a very simple word tokenizer.
Python

import numpy as np

texts = {
   "dog": np.array([0.9, 0.1, 0.2]),
   "puppy": np.array([0.8, 0.2, 0.2]),
   "cat": np.array([0.8, 0.1, 0.3]),
   "car": np.array([0.1, 0.9, 0.8]),
   "truck": np.array([0.2, 0.8, 0.7])
}
Now define cosine similarity:

Python

def cosine_similarity(a, b):
   return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)
)
Compare dog and puppy::
Python

print(cosine_similarity(texts["dog"], texts["puppy"]))

print( cosine_similarity(texts["dog"], texts["car"] ) )

You should get something conceptually like:

dog vs puppy → cosine=0.99 → high similarity
dog vs car → cosine=0.33 → lower similarity

This demonstrates the mathematical principle of embeddings.
 
Real embeddings with Python
Now let's use an actual embedding model. A very convenient library is Sentence Transformers.

pip install sentence-transformers
Then:

Python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

sentences = [
   "I love my dog.",
   "My puppy is very cute.",
   "Cats are wonderful pets.",
   "I bought a new automobile.",
   "The vehicle is very fast."
]

embeddings = model.encode(sentences)
print(embeddings.shape)
You will get approximately:

(5, 384)
Each sentence has been converted into a 384-dimensional vector.
 
Calculate similarity
Now we use cosine to calculate similarity.

Python

from sklearn.metrics.pairwise import cosine_similarity

similarity = cosine_similarity(embeddings)

print(similarity)
You'll get a 5 � 5 matrix:

[[1.0000001 0.59701616 0.54455847 0.27561283 0.06691159]
[0.59701616 1. 0.4266082 0.15517783 0.08879378]
[0.54455847 0.4266082 1. 0.1321229 0.13605565]
[0.27561283 0.15517783 0.1321229 0.9999999 0.34935087]
[0.06691159 0.08879378 0.13605565 0.34935087 1. ]]

The model learns these vectors during training.
Embedding search
This becomes particularly powerful when we have thousands or millions of record. 
Python

documents = [
   "Dogs are popular household pets.",
   "Cats are commonly kept as pets.",
   "Toyota produces automobiles.",
   "Electric vehicles are becoming popular.",
   "Puppies require special care.",
   "Machine learning is used for medical diagnosis."
]
Our user searches:

"How should I take care of my puppy?"

    
We create an embedding for the query:
Python

query = "How should I take care of my puppy?"
query_embedding = model.encode([query])
Then calculate similarity:
Python

document_embeddings = model.encode(documents)

scores = cosine_similarity(query_embedding, document_embeddings)[0]
Sort them:.

Python

results = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)

for document, score in results:
   print(f"{score:.3f} {document}")
You might get:

0.719 Puppies require special care.
0.390 Dogs are popular household pets.
0.227 Cats are commonly kept as pets.
0.060 Machine learning is used for medical diagnosis.
-0.024 Toyota produces automobiles.
-0.052 Electric vehicles are becoming popular.
Notice something important. The query doesn't necessarily contain the exact words from the document.. 

Query:
"How should I take care of my puppy?"

Document:
"Puppies require special care."
Yet the embedding model recognizes their semantic relationship.
Embedding vs. LLM
An important distinction:
 

Input

Output
Embedding model "What is Alzheimer's disease?" [0.023, -0.145, 0.081, ..., 0.034]
Generative LLM "What is Alzheimer's disease?" "Alzheimer's disease is a neurodegenerative disorder..."
So, embedding primarily represents meaning but LLM generates language.
Embedding can represent different types of data, including text, images, audio, video, and speech, by converting them into numerical vectors that capture their meaningful features.
This is why embeddings are so powerful for AI systems.
 
This is a small neural network that learns embeddings from text, so you can see exactly what is happening rather than using a pre-trained embedding library. The example uses a simple word-context prediction task, which is conceptually related to Word2Vec.