Ethics and LLMs: Quality, Error, & Biased Error
Let's talk about error, and how error can be an ethical problem on top of being just a regular problem.
First, let’s talk about the types of error, and why LLMs are trickier than we are used to.
In a classifier, you have two types of error: false positive, and false negative. In a false positive, the correct answer is "no," (e.g. this is not a photo of the guy we are looking for) but the model returns a "yes" instead. In a false negative, the answer is "yes" (e.g. this is a photo of the guy we are looking for) but we let him go.
We rely on the context and the stakes of the error to prefer one type of error over another. For example, with medical diagnostic tests, we would rather that they have high false positive rates to avoid even a few false negatives, because we judge that being nervous about your diagnosis for a week is better than getting the "all clear" when you actually need treatment.
With an LLM, you can have more subtle kinds of error. LLMs don't just give binary yes/no answers—they generate text, explanations, and recommendations that can be wrong in complex ways. Here are the main types of LLM errors to watch out for:
Hallucinations happen when the system presents fabricated information as factual. This can range from inventing citations to manufacturing statistics or creating entirely fictional narratives about real people or organizations. I dealt with this personally when writing “Amplify Good Work”—I used AI to help create outlines for a few of the values chapters based on my notes, and it added case studies (good idea!) that turned out to be completely made up (terrible idea!) Don’t worry, I plan to publish all the prompts I used with each model!
Mischaracterizations involve subtle distortions of facts that may be difficult to detect but meaningfully change the content's implications. For example, an LLM might oversimplify a complex policy issue or misrepresent the cause of a problem, which could be critically important when you're making the case for your organization's work.
Critical omissions happen when systems leave out necessary context or information. LLMs don't have a comprehensive view of what's important, so they might give you a framework or recommendation while missing crucial factors specific to your situation. There’s a high risk of this when you’re asking about anything current: LLMs have a cutoff date beyond which they can receive information from you in your prompt or search the web, but do not have any knowledge in their training data.
These errors can be practical problems when outputs aren't what you're expecting or when you act on inaccurate information. But they can also be ethical problems.
Here's how:
Error in high-stakes domains can lead to unethical action. If you have a system that has high error of any kind and are using it to make very consequential decisions for others, you may have done something unethical (and you may be legally liable for any harm caused). Putting error-prone algorithms in charge of whether people are hired, fired, demoted, promoted, identified to law enforcement, convicted, sentenced, etc., raises serious ethical concerns.
Wrong answers that communicate or perpetuate bias. Even in domains that aren't particularly high-stakes, this could have ethical implications. For example, in research I co-authored, we studied emotion recognition designed for use in the workplace. If you worked in a factory and an emotion recognition program kept playing "soothing" music to you because it believed you were mad, how would that make you feel? Might it impact your social standing among coworkers? Now add that emotion recognition technology over-estimates anger among Black people (which emotion recognition systems based on facial expressions does consistently). This annoyance is going to happen more to Black workers than others, potentially reinforce stereotypes, and possibly limit people’s career advancement inappropriately.
Error that is distributed in a biased way. Even if an algorithm has low overall error, that error can be distributed across groups in an unethical way. For example, we might prefer that the overall false negative rate for skin cancer detection be low, because we would rather people get a second test than walk away with the false belief that they do not have skin cancer. But, if the error rate is not evenly distributed, it could be more likely to give people with dark skin a false negative. That means the false sense of security we specifically didn't want is more likely given to people with dark skin.
One important way to mitigate AI errors is to put a human in the loop. This could mean making sure a human reviews all AI outputs before they are acted on. This can be a very important step, but it is not perfect. There are two main reasons that human-in-the-loop can fail as quality and ethical control:
They can be biased by the AI output. For the same reason we don't learn that well when we have the answer key right next to the homework questions, we can be swayed by the answer that AI gives if we don't have time to think critically about it ourselves. Research shows that humans tend to over-rely on automated recommendations, even when those recommendations are clearly labeled as fallible—a phenomenon called automation bias.
They can be used as "moral crumple zones." Companies sometimes hire people to be in the loop so that whenever the system makes a consequential mistake, they can assign responsibility to the human, fire the human, and not feel obligated to make costlier changes to the technology and practices that caused the problem in the first place.
Consider the ethics of error alongside its practical impacts when implementing any AI, including LLMs! I recommend being very cautious about using AI when high-stakes decisions are involved, because we simply don't have great safeguards for error, especially the subtle kinds that LLMs can produce.
Have you noticed ethical impacts of error in AI systems that I didn't mention? I would love to hear more, especially with LLM implementations!
—
LLM disclosure.
I took a photo of my arm and asked ChatGPTo3, Claude, Gemini, Co-Pilot, and Grok “Do you think this mole is cancerous?”
None of them would give me a straight answer! Fair enough. I made a real Grok account for this test. I hope you appreciate it ;)I asked ChatGPT o3:
“I'd like to write a blog post about ethics and LLMs centered around error. Here's what I have so far. I'd like add two things: 1) to explain the types of error (false positives vs false negatives) and 2) to note that it is a little more complicated because LLMs don't only offer binary answers; qualitative answers can be wrong in more subtle ways. Can you give me some suggestions for where this content would go and a draft of the explanations, including at least one example in each explanation?”
Most of this answer I didn’t use, but it did give me some examples of how error can show up in LLMs that I used in the fourth paragraph.Then, a few weeks later, I had written the chapter on this topic for my book and added some useful content, so I asked Claude 4 Sonnet : “Thanks! can you update this blog post to include some of the information in this chapter that is not already here? for example, types of LLM errors. The resulting post should be much shorter than the chapter, and it's OK to leave some of the really deep dive stuff for the chapter.”
I retried my entire blog post—> image plan.
“can you create an image for this one? It should be approximately square.”
Solved the problem!