Lesson Plan: How Do Large Language Models Predict?
What is this lesson about ?
This lesson explores how large language models (LLMs) work by linking their probability-based predictions to real-world applications. We start with the basics of next-word prediction, then extend the idea to forecasting stock prices using news sentiment. Along the way, we explain how AI models can be powerful but also imperfect predictors.
The goal is to understand how these tools fit into broader questions about data and economics, not to make you a machine learning expert.
Learning Objectives :
By the end of this lesson, students will:
- Understand how large language models (LLMs) generate text through next-word prediction and probability distributions.
- Describe the role of temperature in shaping outputs.
- Connect these concepts to other forms of prediction, such as time-series forecasting.
- Apply sentiment analysis to real-world news headlines and interpret the results.
- Combine sentiment with stock price data to test whether public mood helps forecast financial markets.
Materials and Technical Requirements :
The lesson will be administered through a Jupyter Notebook llm_distributions.ipynb, hosted on the prAxIs UBC website. Students will need access to a device connected to the internet, preferably a laptop. No previous coding experience is required, though some familiarity with Python is an asset. If students do not have access to a capable device, it is acceptable to pair up into groups of two or three.
Pre-lesson Checklist
- You should have read the short introduction on probability and prediction (5–10 minutes).
- The instructor has loaded and tested the notebook before class.
- Bring your laptop or be ready to pair up.
Agenda
- Pre-discussion & mini-lecture (10 min)
- Warm-up example: “The cat sat on the ___.”
- Discuss how prediction works in language and how it might connect to forecasting in other contexts.
- Quick question : Can you think of where else we use probability in everyday life?
- Section 1 : Next-word prediction (5 min)
- Students run toy examples showing probability distributions across a small vocabulary.
- Quick question : Why does context matter?
- Instructor explains chain rule and conditional probability.
- Section 2 : Sampling and temperature (10 min)
- Students experiment with temperature values (0.5, 1.0, 1.5).
- Compare “sharper” vs. “flatter” distributions.
- Quick question : When might you want more creativity versus more focus in an AI system?
- Section 3 : From words to stocks (5 min)
- Transition from language to financial data: show how LLM ideas generalize.
- Instructor ties this to econometric forecasting models.
- Section 4 : Sentiment analysis of news headlines (10 min)
- Apply RoBERTa classifier to Starbucks news.
- Students classify a few headlines/sentences themselves, then compare with the model’s output.
- Students try using the classifier, inputting different sentences.
- Quick question : Did the model get it right ?
- Section 5 : Sentiment + stock prices (5 min)
- Explore combined dataset of sentiment and daily stock returns.
- Instructor introduces SARIMAX and forecast visualization.
- Students interpret forecast plots: does sentiment lead stock prices, or not?
- Section 6 : AI on AI (5 min)
- Sentiment analysis for top 10 AI companies.
- Quick question : Is the news optimistic or pessimistic about AI?
- Wrap-up and reflection (5–10 minutes)
- Key takeaways: probability, sentiment, prediction, evaluation.
Activity Materials
Discussion Post (2-3 sentences)
- How does changing temperature alter model predictions?
- Why might news sentiment affect stock prices? Does it cause changes, or just reflect them?
- What are the benefits and risks of using LLMs in economic or social research?
- If you built a forecasting model like this, what else would you include to improve it?