The most interesting thing that I’ve learned about Language Learning Models(LLMs) this past week was that they don’t actually try to predict the right answer for the prompt, but that they purely go off of statistics with a little bit of context. I found the term ‘Stochastic Parrot‘ from Emily Bender’s lecture to be a good nickname for the process that LLMs use to answer any given prompt; similar to real parrots, they just repeat and string together words and/or sentences without knowing the meaning behind them.
I found the video above from 3Blue1Brown to be a very understandable overview(for someone who hates computer science) of a bit of the background and process of transformers to LLM’s to the generative AI that we see now. And it really helps to illustrate not just how they work but also alludes to how they can go wrong and create negative feedback loops and hallucinations mentioned in the article by Jon Cheung, ‘How AI Image Generators Make Bias Worse.’ With how AI datasets are bound to have some form of bias in them, and then as generative creates more content representative of that bias, it then feeds back into itself, amplifying the biases even more.
Citations:
- 3Blue1Brown. (n.d.). Transformers, the tech behind LLMs | Deep Learning Chapter 5 [Video]. YouTube. https://www.youtube.com/watch?v=wjZofJX0v4M
- Lis, & Lis. (n.d.). How AI image generators make bias Worse. London Interdisciplinary School. https://www.lis.ac.uk/stories/how-ai-image-generators-make-bias-worse
- Harvey Mudd College. (2024, November 20). Emily Bender: “Don’t Try to Get Answers from a Stochastic Parrot” | 2024 Nelson Speaker Series [Video]. YouTube. https://www.youtube.com/watch?v=5m0ZolIb2hA
- Declaration of AI Use: I acknowledge the use of Generative AI tools in writing this post: I used Grammarly to refine the language after drafting on my own, I declare that I reviewed and edited the contents as needed and take full responsibility for the content of my submission. All of the information I presented is complete and accurate.