Recently, I asked readers here to share how they’re using large language models (LLMs) like ChatGPT to learn and study. Today, I’m rounding up some of those suggestions and trying to synthesize the advice for when (and when not) to use these tools for learning.
Strengths and Weaknesses of LLMs
Using applications like ChatGPT requires some care.
Part of the difficulty is that ChatGPT’s human-like conversation abilities can be deceptive. Feeling like you’re talking to a real person encourages you to rely on conversational expectations that may not hold with a machine.
For instance, we generally expect that most people do not make up facts. Large language models, however, routinely violate this expectation by providing fluent answers that may be totally wrong. The metacognitive ability to know what you don’t know is underdeveloped in these applications.
Another expectation we have is that verbal fluency tracks other aspects of intelligence. We expect that someone who can spout lines from Shakespeare, explain quantum computing, and give a proof of the prime number theorem in rhyming verse would also be able to count. Thus, naively treating LLMs like a really smart and knowledgeable person is likely to backfire.
Those caveats aside, ChatGPT is clearly helpful for a range of tasks. Simon Willison suggests thinking of LLMs as a “calculator for words”—something that can do useful things with text—rather than as a general-purpose intelligence or smart person.
I tend to agree. The more we can distinguish the cases where LLMs work well from where they don’t (yet), the more we’ll be able to take advantage of the new capabilities without falling into unexpected traps.
Ten Useful Learning Strategies with ChatGPT
After receiving dozens of emails from my audience on how they have been personally using ChatGPT to learn, I’ve compiled some tips with some of the most common suggestions.
1. Create your own Socratic tutor.
By far, the most common use readers reported was using an LLM as a personal tutor.
Asking ChatGPT to explain tricky concepts, unfamiliar code or problems seems like an area where LLMs might do alright. And the only reasonable substitute (a human expert) is notoriously expensive and in short supply.
If you do this in conjunction with a class or textbook, the risks of mistakes also seem attenuated since you still have a primary source to compare against. Challenge explanations that don’t jive with what you’ve read in the book rather than taking everything the AI says at face value.
2. Practice chatting in new languages.
The next most common way people used LLMs to learn better was as a language tutor. This seems like a task LLMs are well-equipped for. Whatever their other flaws, they can produce grammatically correct text.
Many people set up their conversations with ChatGPT so that the AI could go back and forth between the language they are learning and English explanations when they got confused. Likely, those explanations may be imperfect, but human tutors also often give incorrect accounts of the grammar and vocabulary they manage to use proficiently.
Another use is rewriting texts to be at a more beginner-friendly level of reading comprehension. Graded readers and comprehensive input are great strategies for learning to read in another language. Unfortunately, learner materials are often sparse or uninteresting. You can use an LLM to transform a text you want to read that is written at a fluent native level into something appropriate for your current ability.
Duolingo seems to be getting in on the LLM game as well. I’ve been harsh on the drag-and-drop style of language learning used in earlier versions, but these new advances may force me to revise my opinion.
3. Generate summaries of longer texts.
Summaries are another area where LLMs seem to excel. Consumer applications already exist for generating summaries of journal articles or research topics.
Several readers said they were using these AI tools to provide digests of their substantial reading material, helping them keep atop new developments in their field.
Good summaries, especially those fine-tuned to your particular needs, might be a good way of navigating the large information loads we often face in knowledge work. You could use it to help prioritize which documents to read in-depth or do a first pass organizing unfamiliar material.
It looks like ChatGPT is not able to open links to summarize directly, but it often does a reasonable job hallucinating with famous texts like the one I used above. Until this functionality is standard in the consumer application, it’s better to copy+paste.
4. Dialog with long documents.
LLMs also can help you “ask questions” of longer texts. For instance, when reading a scientific paper you could quickly query the sample size or ask for the methodology or results. Consensus does this while offering references, so the risk of mistakes seems reduced when you can easily double-check the LLM’s work.
While there are more fanciful usages here, such as people asking ChatGPT to impersonate a given writer and dialog with them, I suspect the ability to ask natural-language questions of documents and receive replies with references is a useful tool for dealing with large texts.
That said, you must be prepared to fact-check the LLM’s answers. For instance, in the dialog captured below, I asked ChatGPT to list evidence supporting strongly-guided instruction, and it cited a review article by Mayer. But it falsely claimed Mayer’s work was a meta-analysis, which it isn’t. In fact, the paper isn’t even a comprehensive literature review but simply looks at three prominent cases of the failure of discovery learning. If taken at face value, this response can potentially mislead, but it’s relatively easy to “check” the AI’s work if you know what text it is transforming.
5. Rewrite texts at different levels of explanatory depth.
A major difficulty in following expert thinking is that most expert-level text is written for other experts. Concepts are unexplained, context is lacking and jargon abounds. This means that most people must rely on translators, such as general-market nonfiction or science writers, who present what experts think in a more readable format.
There seem to be two approaches to using AI tools here. One is simply asking an LLM to explain a popular concept in simpler terms, such as: “Explain quantum computing like I’m an eighth-grader.” The other is to provide ChatGPT with a text or explanation and ask the AI to rewrite it in a more digestible manner.
I tend to think the latter is a little more reliable since you have the source material to compare with rather than taking ChatGPT’s word for it.
6. Clear up unfamiliar jargon.
Several years ago, I remember reading Tyler Cowen’s Marginal Revolution blog and being perplexed by his frequent, unexplained use of the term “Straussian” to describe ideas or other thinkers. I Googled for an explanation, but none was forthcoming.
After a lot of research, I understood the term as meaning, roughly, “closely reading between the lines in prominent thinkers’ ideas, looking for what they really meant but couldn’t always express because of prevailing censorship and intellectual orthodoxy.”
Had ChatGPT existed during my confusion, I could have just gotten this:
Many readers found similar benefits in using LLMs to figure out jargon and terms used within a particular community in a way that dictionary definitions often fail to elucidate.
7. Create study plans and agendas.
This usage surprised me, but it showed up enough times among reader replies that I include it here. People seem to like using AI to tell them how and when to learn.
For instance, some readers asked ChatGPT to break down a complex learning goal and give them a curriculum. Others preferred to go even further, asking for ChatGPT to create a studying schedule for them, given their constraints for the day.
I probably wouldn’t trust LLMs to give me a well-designed curriculum for a subject. But if I was learning something completely new, it might be a decent starting point. Sometimes the hardest part of approaching a new field is breaking down what appears to be an insurmountable goal. Similarly, sometimes being told when to study can help overcome the inertia of getting started.
While skill breakdowns might be alright, ChatGPT still struggles with creating reading lists, confabulating books and references. Thus, while it might do well for decomposing an ambiguous learning task, I wouldn’t trust it to give me good resources (yet).
8. Provide refreshers on forgotten or infrequently used tools.
Programmers were the biggest professional group to reply to my query. I can’t say whether this is because programming is uniquely well-suited to LLMs or because programmers, as a group, are more likely to adopt novel software tools.
The productivity advantages for programmers seem obvious. I don’t write much code these days, so I haven’t made much use of this well-publicized feature of LLMs. But since a lot of coding is relatively routine, the ability to have a machine create the first draft for an algorithm clearly saves a lot of time.
While there are cases of people with zero programming knowledge relying on AI output to build applications, I suspect these might be tricky to debug and maintain. In contrast, an expert programmer can override ChatGPT’s output for a language he or she knows particularly well.
The place LLMs seem to work really well is at the fringes of a programmer’s expertise. Many programmers told me that they found AI helpful in getting starting hints in unfamiliar languages or tools. Their base of programming experience allowed them to make sense of and implement the output, but their unfamiliarity with the underlying language meant the AI saved them a lot of time.
9. Generate flashcards based on text. (Tentative)
Flashcards are a powerful learning tool. They’re also a pain in the butt to make.
Some readers said they were using ChatGPT to generate flashcards for subjects they’re studying. This seems well within the LLM abilities as a “calculator for words.” Thus, with the correct prompts, you could get fairly good results here—provided you’re inputting the material you wish to see transformed into flashcards and not expecting the LLM to get the facts on its own (see below).
However, given the difficulty of making “good” flashcards, I wouldn’t enter any into my Anki without reviewing them first. Nonetheless, making flashcards is tedious, so getting a first draft that I later review might speed up the process considerably. The risks seem relatively limited if you confirm the cards’ correctness before putting them in your deck.
10. Use it to organize your notes. (Advanced)
As someone who does a lot of research, I often waste a lot of time trying to locate my notes. Robert Martin finds the same problem. Searching via keywords is fraught because sometimes you can’t remember the exact term you used, even if the meaning is roughly the same.
Martin solves this problem by using the embedding feature of LLMs. While not strictly ChatGPT, this tool from the same family of natural language processing techniques allows you to find semantically-related notes rather than exact keyword matches.
Personalized LLMs that live on your hard drive and access your existing data may be a valuable application. I’d love to be able to search things I know I’ve seen but can’t quite recall where.
Some Things NOT To Do
1. Don’t expect AI to get facts right.
LLMs frequently make stuff up. These hallucinations are problematic if you depend on ChatGPT to give correct answers. The prevalence of these mistakes is difficult to say right now. When Wikipedia was released, for instance, “experts” were in an uproar about how the user-generated nature of the website meant it couldn’t ever be relied upon as a source. Except Wikipedia actually does fairly well, and some of these knee-jerk reactions were misplaced.
LLMs haven’t reached Wikipedia’s quality in terms of facts, and we still don’t know much about when they’re likely to get an answer right and when they’re likely to make stuff up. For now, it seems best to use them for situations where the cost of an incorrect answer is minimal, either because you can look it up in a verified source, or because your use of the AI for isn’t factual in nature.
2. Don’t expect AI to get citations right.
While LLMs sometimes mess up facts, they seem abysmal at getting citations right. They frequently invent authors, papers, studies and research.
I wouldn’t use a LLM for any research that I needed to cite, and I would always double-check the sources it does provide.
Similarly, I wouldn’t ask an LLM to give me a reading list or references to specific books or authors (unless, possibly, the authors were quite famous and likely to be well-represented in the data set).
3. Don’t expect AI to get the math right.
I believe it’s a mistake to attribute general intelligence to LLMs based on their ability to do many tasks at a human level or beyond. As with chess bots and image classifiers, the technology behind LLMs is extremely narrow compared to what we would expect of a human who scored similarly on verbal tests.
One finding from psychology is that much of reasoning is performed by different subsystems in the brain than those devoted to language. This paper argues that LLMs seem to match the neurological evidence from double-dissociation studies: you can have fluent verbal abilities with seriously impaired reasoning and vice versa.
As such, LLMs are really bad at math. And not just higher-level math that humans struggle with. LLMs often fail at basic counting tasks. Thus I suspect LLMs would be uniquely bad at a task like providing practice problems for a math class and grading the answers. ChatGPT might be able to explain a math concept well, but be unreliable at actually using it.
Have you used ChatGPT or other AI tools to further your learning? Share your own observations and suggestions in the comments below!
