AI Large Language Models

What Is a Large Language Model?

A large language model (LLM) is a neural network - almost always built on the transformer architecture - trained on a huge collection of text (books, websites, articles, code, and more) to perform one core task extremely well: given some text so far, predict what token is most likely to come next. 'Large' refers to both the size of the training data and the number of internal parameters (weights) the model has, often numbering in the billions.

The Core Training Idea: Predict the Next Token

Training works by repeatedly showing the model a snippet of real text with the last token hidden, asking it to guess that token, and adjusting its weights (via the backpropagation and gradient descent process described earlier in this tutorial) whenever it guesses wrong. Because the correct next word can be pulled directly from the training text itself, no person needs to manually label each example - the text supervises itself. After this base training, many LLMs go through additional rounds of fine-tuning on curated examples of good responses, and sometimes further adjustment based on human feedback, to make them more helpful, better at following instructions, and less likely to produce harmful output.

# A miniature, hugely simplified illustration of "predict the next token" -
# real LLMs use billions of parameters and far more sophisticated methods,
# but the underlying idea of learning what tends to follow what is the same.

corpus = "the cat sat on the mat the cat ran away the dog sat on the rug".split()

# Count what word follows each word in the training text
next_word_counts = {}
for current, nxt in zip(corpus, corpus[1:]):
    next_word_counts.setdefault(current, {}).setdefault(nxt, 0)
    next_word_counts[current][nxt] += 1

def predict_next(word):
    options = next_word_counts.get(word, {})
    return max(options, key=options.get) if options else None

print("After 'cat', likely next word:", predict_next("cat"))
print("After 'on', likely next word:", predict_next("on"))

Why Scale Changes What's Possible

As LLMs have grown in training data, parameter count, and compute, they have picked up abilities that were never directly targeted during training - answering questions, writing code, holding a conversation, following multi-step instructions - simply by learning statistical patterns of language at enormous scale. Researchers call these emergent abilities: capabilities that show up in larger models but were barely present in smaller ones trained the same way, even though the training objective (predict the next token) never changed.

What LLMs Are Actually Doing (and Not Doing)

It's worth being precise about what's happening under the hood: an LLM generates text by repeatedly predicting a statistically plausible next token given everything so far. It does not have beliefs, a persistent memory of what is true, or a built-in fact-checking process - it produces the continuation its training makes most likely, which is usually accurate on well-represented topics, but is not the same thing as verified reasoning or guaranteed correctness.

• Hallucination - the model can state incorrect information confidently and fluently, with no signal to the reader that it's unsure
• No guaranteed factual accuracy - correctness is a byproduct of good training data and pattern-matching, not a checked property
• Knowledge cutoff - the model only 'knows' about events and information from its training data, unless connected to external tools
• Sensitivity to phrasing - rewording a prompt can change the answer, sometimes significantly
• Inherited bias - the model can reproduce skewed or unfair patterns present in its training text