co-founder palma.ai
A Practical Guide to How ChatGPT Works
Many of you are using ChatGPT by now and may be wondering—how does it actually work? Before we get into what we (and OpenAI) don’t really know, let’s use an analogy. Disclaimer: this analogy is provided by ChatGPT itself! But I did think it was quite a good way to understand the different bits and pieces that make the AI work.
Imagine you're learning to become a chef. You start by watching lots of cooking shows (training), observing how different chefs prepare various dishes. Over time, you begin to understand recipes (data) and cooking techniques (algorithms). Now, suppose someone asks you to cook a meal (input prompt) based on what you've seen. You'd combine all the techniques and recipes you've learned to prepare that dish (generate a response). This process of learning and creating is similar to how ChatGPT works, but instead of cooking, it's about generating text.
Learning from a Cookbook (Training on a Dataset)
Just as a chef learns to cook by studying recipes, ChatGPT learns language patterns from a vast dataset of text. This dataset is like a gigantic cookbook with recipes for every conceivable topic.
Practicing Recipes (Training the Model)
A chef practices recipes to get better. Similarly, during its training, ChatGPT practices by predicting the next word in a sentence, again and again, improving its understanding of language over time. This is akin to practicing recipes until they're memorized.
Taking a Cooking Request (Receiving a Prompt)
When you ask ChatGPT a question or give it a statement (the prompt), it's like asking a chef to make a specific dish for you. The request tells the chef (ChatGPT) what you're in the mood to eat (the kind of response you're seeking).
Preparing the Dish (Generating a Response)
The chef decides what ingredients (words) and cooking techniques (language rules) to use based on what they've learned. ChatGPT does something similar by selecting words based on the patterns it has seen in the data it was trained on, preparing a "dish" (response) tailored to your request.
Using Special Ingredients (Attention Mechanism)
Suppose the dish needs to focus on a specific ingredient to taste right. ChatGPT uses something called an "attention mechanism" to focus on important words in your prompt, ensuring the "flavor" (context) of your request comes through in the response.
Adjusting the Recipe (Fine-Tuning)
Even after learning from the cookbook, a chef might adjust a recipe based on the diner's preferences. Similarly, ChatGPT can be fine-tuned with additional information or data to better suit specific tasks, much like tweaking a recipe to get the flavor just right.
Serving the Meal (Delivering the Response)
Finally, the chef serves you the dish, which is the culmination of their training, practice, and your specific request. Similarly, ChatGPT presents you with a piece of text, a direct result of its training and your prompt.
Final Tidbit
Just an interesting tidbit and the reason why you shouldn't post anything into ChatGPT is that even OpenAI itself doesn’t quite understand how the AI learns from the cookbooks (in the analogy) and uses that information to output new recipes (generate content). As Sam Bowman, professor at NYU, said:
"If we open up ChatGPT or a system like it and look inside, you just see millions of numbers flipping around a few hundred times a second, and we just have no idea what any of it means. With only the tiniest of exceptions, we can’t look inside these things and say, 'Oh, here’s what concepts it’s using, here’s what kind of rules of reasoning it’s using. Here’s what it does and doesn’t know in any deep way.' We just don’t understand what’s going on here. We built it, we trained it, but we don’t know what it’s doing."
What this means practically is that anything you post into public ChatGPT instances is used to train the models, and no one can predict how the information you put into it via prompts may end up on someone else's screen again. You can read more about this in my earlier blog post.
You can completely avoid the risk of your sensitive data being leaked via ChatGPT by using palma.ai's extension. We monitor every single prompt and ensure that no secret or sensitive information ever reaches ChatGPT servers (or any other public AI for that matter).