Chapter 5 `train_model_simple` should control randomness

[zzehli]

Bug description

In chapter 5, the book trains a toy GPT for 10 steps and produces a loss curve plot. In the loss curve, the train loss goes down while the validation loss stays flat. However, the code provided generates very different plots every time. Is this expected? I tried several ways to fix the random to no avail. Any suggestions to make the training loop produce the desired loss curves every time?

What operating system are you using?

macOS

Where do you run your code?

Local (laptop, desktop)

Environment

[rasbt]

Hi there, that's a good question and not expected. When I run the code in my environment multiple times, it looks the same. Have you tried using the notebooks from this repo directly without modifying them?

[zzehli]

I was using my own notes, which is a combination of book code and code in this repo. But I tried to run ch05 notebook in this repo and also got different results each time as well. Turns out it's an apple mps issue. If I use cpu instead the loss will be the same.

[d-kleine]

This might be related to #625, tagging for reference. I hope this helps 🙂