What’s Actually Happening When You Talk to ChatGPT?: How LLMs (Large Language Models) Actually Work, Explained Simply

Okay so you’ve used ChatGPT, or Claude, or whatever chatbot your company just made you download. And at some point you’ve probably wondered…what IS this thing actually doing?? Is it thinking? Is it googling stuff behind the scenes? (Unless you’re using a search-enabled feature, nope!) Is there some giant database somewhere with every answer already written down? Nope. Let’s break this down.

---

What is an LLM, really?

A large language model predicts the next word. One word at a time. Based on patterns it picked up from a truly absurd amount of text. That’s it. That’s the whole trick!!

Everything else you’ve seen it do — writing paragraphs, spitting out code, holding a conversation — is that same simple move, just running over and over, at a speed and a scale that’s honestly kind of hard to wrap your head around.

---

What’s the difference between training and inference?

Two completely different phases here, and mixing them up is where most of the confusion starts.

• Training happens once, way before you ever type a message to it. The model reads a mountain of text — books, articles, half the internet — and slowly tunes billions of internal settings (called “weights”) so it gets better and better at guessing what word comes next. This part is slow. Expensive. Months-of-computing-power expensive.
• Inference is what happens every single time you hit send. By this point the model’s done learning — it’s locked in. It’s not picking up anything new in real time, it’s just using everything it already absorbed to predict, word by word, what a decent answer looks like.

Think of it like this: training is culinary school (years of it). Inference is cooking dinner tonight. The learning already happened — tonight you’re just executing.

---

What’s a “token,” and why does it matter?

Here’s something that trips people up: LLMs don’t process “words” the way you’d picture it. They process tokens — chunks of text that might be a full word, part of a word, or even just punctuation. “Strawberry” might get chopped into two or three tokens depending on the model.

Why does this matter? A couple reasons:

1. It’s part of why models sometimes botch simple stuff like counting letters in a word — they’re working with chunks, not letters, unlike you and me.
2. It’s literally how you get billed on the API side of things. Costs are quoted in tokens, not words.

(So next time you see a pricing page full of “$X per million tokens,” now you know what you’re actually paying for.)

---

What does “context window” actually mean?

The context window is just how much text the model can look at, at once — basically, your current conversation. People love calling this “memory,” but it’s really more like short-term working memory that wipes itself clean. Unless a product is specifically built to remember you across sessions, the model has zero idea who you are the second a new conversation starts.

And yeah — long conversations eventually fill that window up, which is part of why a really long back-and-forth sometimes seems to forget something you said an hour ago. It didn’t forget, exactly. It just scrolled out of view.

Kind of like trying to remember page 3 of a book while you’re on page 300. Cut it some slack.

---

Why do LLMs “hallucinate”?

This is the part that really messes with people, especially if you’re used to computers being either right or throwing an error at you. There’s no fact-checker running quietly in the background here!! The model generates text through pattern-completion — it’s predicting what a plausible, fluent-sounding answer looks like, based on everything it’s read.

Most of the time that lines up fine with reality, because most of what it learned from was true to begin with. But here’s the catch, and it’s a big one: the exact same mechanism that gives you a correct, well-reasoned answer is producing a completely made-up one right next to it, with the exact same confident tone. There’s no switch flipping to “actually, I don’t know this one.”

So — fact-check the specific stuff. Numbers, quotes, dates, citations. No matter how sure it sounds. Especially when it sounds sure.

---

What do fine-tuning and RLHF actually change?

After the big training pass, most models go through more rounds meant to shape how they behave, not what they know. Being more helpful, saying no to harmful requests, matching a certain tone — that kind of thing.

One popular method here is RLHF — Reinforcement Learning from Human Feedback. Basically, humans rate a bunch of the model’s possible responses, and the model gets nudged toward producing more of whatever got rated highly.

Important part: this doesn’t add new facts. It’s adjusting behavior on top of everything already learned in training. Same brain, different manners, if you will.

---

Why does any of this matter beyond curiosity?

If you work anywhere near security, training, or customer-facing tech — which is where I come from — this isn’t just cool trivia.

• Prompt injection attacks work because a model can’t always cleanly tell the difference between “instructions from the developer” and “text that just happens to look like instructions,” buried in whatever content it’s chewing through.
• Data leakage shows up when people paste sensitive info into a chat window without really thinking about where that context goes, or where it might pop back up later.

Understanding the actual mechanism at play — prediction based on patterns, not retrieval, not reasoning the way a human does — is the foundation for explaining these risks to someone who isn’t technical. And honestly? That’s really the whole point of this site: taking something technical and making it make sense to the person who needs to get it, not just the person who already does.

---

I still have questions about where all this goes next — how much better hallucination detection gets, whether context windows keep growing or whether that’s the wrong problem to solve, what happens to prompt injection defenses as these things get plugged into more of our daily tools. I don’t have clean answers to any of that yet. But that’s a post for another day.