AI and the News

When I read Simon Wardley's post on LinkedIn from a couple of days ago, I was immediately struct by its resemblance to criticisms of another medium: the news (you can read Wardley's post here on LinkedIn if the subscription wall lets you, but no matter, I'll be quoting most of it anyways).

Here's the premise. Wardley reports that he's "going through another AI horror story" and reminds us that "these tools are great but please remember the following when you are using LLMs/LMMs," after which is a list of things to remember.

Now I don't disagree with any of these in any particular way, but as I said, they reminded me of critiques of news media. By 'the news' what I mean is the coverage in print, radio and television (and more recently, the web) by organizations (and sometimes individuals) specifically dedicated to reporting current events and issues. When I say 'the news', I mean basically what you think I mean.

My point here is that the things we need to remember are not things unique to AI. They're things we need to keep in mind when people are telling you things generally. So let's go through Wardley's points:

First, "All outputs are hallucinations i.e. fabricated and ungrounded. Many of these outputs happen to match reality when there’s abundant training data and repetition, so they look useful on common tasks. But they cannot do research. These machines are stochastic parrots (Bender et al), they are pattern matchers and not reasoning engines." 

There's a sense in which this is true. As was noted elsewhere today, a generative AI engine is a "library that retrieves and recombines information based on patterns." But so is, strictly speaking, a news reporter. They do not directly experience the events they describe, they rely on second hand reports, and weave these together to create a picture. It's often accurate, but the process does not guarantee accuracy.

When Wardley writes of AI engines, "they cannot do research", he is, strictly speaking, wrong. An AI can go through the same steps as a news reporter, which includes research into background documents and publications, data analysis, and even (if access is granted) questioning participants. Ultimately, both news reporters and AI engines depend on the data they're given.

(It's worth noting that we used to have AI 'reasoning engines' (also known as 'Good Old Fashioned AI' or GOFAI). They didn't work. The demand for AI that 'reasoned' was responsible for the long 'AI winter'. In the end, AI that did not 'reason' turned out to be a lot more reliable than AI that did). 

Second, "These systems will happily invent plausible seeming but unverified detail. That’s a design feature not a bug, they are optimised for coherence, not truth."

News reporters do the same thing. It is rare that every assertion in a news report is a strict restatement of something seen and verified by the reporter. Let me offer an example from the news today:

"Ottawa’s night mayor has launched a new website to raise awareness about the capital’s nightlife and help residents and visitors explore Ottawa at night, as part of the plan to shake the image as the “town that fun forgot."

Strictly speaking, the 'night mayor' (not his actual title; he is officially the "nightlife commissioner") did not 'launch' the website. The website was actually turned on by a City of Ottawa staffer responsible for the website. The website is not 'about the capital’s nightlife' but instead focuses on commercial music and entertainment offerings. Nor is "the plan to shake the image as the 'town that fun forgot'" referenced anywhere in the nightlife commissioner's job description or mandate.

These inaccuracies are not bugs. They are features that help present the news as part of a coherent whole and to fit into a context that has been developing over time. Almost any news story will contain instances of this, and in more significant cases the reinvention of core features of the event being reported will be significantly misleading, as for example this story on Labor Day (in the U.S.) describing "What the U.S. Department of Labor Does".

Third, "These systems do not understand what they are creating. The use of tools and guardrails is mostly to convince you of their correctness and to hide their inner workings, they are about shaping perception and behaviour, not true comprehension. Yes, guardrails also reduce some classes of harm."

There are some very distinct points being made here; to say "these systems do not understand what they are creating" is one thing, while to say "the use of tools and guardrails is mostly to convince you of their correctness and to hide their inner workings" is to say something very different, and strictly speaking, they cannot both be true, because you can't hide the fact that you're doing something if you don't actually know what you're doing.

But we get the point, right? It's like when we say "they don't want you to see how the sausage is made", on the one hand, we're pretty sure it's not just meat, but on the other hand, nobody knows exactly what it is. But what they're really doing isn't selling you 'meat', it's selling you 'deliciousness on a bun' and encouraging you to buy more.

Similarly, while the news industry likes to portray a certain image of itself, it downplays the role of advertising and political pressure in story selection, the broad dependence on press releases and official statements, the limitations of 'access journalism', the ravages of 'churnalism', and the limits on perspective that often serve to dilute the product of 'truth'. And just as AI developers have some sense of exactly what they are creating, so do journalists, but the whole of the story is masked in any account of it.

I would add, in many cases, reporters do not have much, if any, understanding of what they're reporting on. To them, the topic or issue at hand is just 'the beat' they're assigned, and they do not enter the field with the requisite background or experience to know what's plausible and what's not (as an aside: economists suffer from the same deficiency).

Fourth, "These problems are not with the user and their prompting. Stop blaming users for what are design flaws and systematic issues."

Similarly, the problem with the news isn't the fault of the viewers of news and 'what viewers want'. Viewers didn't create "if it bleeds, it leads". Advertisers did.  

Fifth, "You cannot 'swarm' your way out of these problems. Orchestration doesn’t solve fundamental epistemic limits. However, these systems (including agentic swarms) are extremely useful in the right context and are excellent for creating hypotheses (which then need to be tested)."

Similarly, more reporters doesn't necessarily mean better news. Even orchestration and optimization (for example, through the use of news agencies) does not solve the epistemic limitations of journalism, where failure is the inevitable result of not having direct access to the story and being driven by external pressures such as publishers and advertisers. 

That said, large numbers of journalists provide fertile ground for 'hypotheses' whether based on predicting what the next election result will be or speculating on whether the president is dead (compounded by speculation on why people thought the president is dead) (I love the story that states, with utterly no evidence, that "Donald Trump is feeling fine"). 

Sixth, "These systems can output long, convincing 'scientific' documents full of fabricated metrics, invented methods, and impossible conditions without flagging uncertainty. They cannot be trusted for policy, healthcare, or serious research, because they are far too willing to blur fact and fiction."

In news media we have two major sources of "scientific documents full of fabricated metrics": polls, and rankings. The creation and use of polls as 'science' by news media is well known, as are the flaws with the methodology. In a similar vein, rankings (such as the U.S. News and World Report or Times Higher Education educational rankings). In both cases, the news media has invented a 'fact' on which it will now report.

It would be a very bad idea to base policy, health care, or any other decision of significant importance on poll data or rankings created by news services. And polls and rankings are only the most obvious ways the news industry attempts to project scientific accuracy.Governments and industry do their own research for the precise reason the news media does not offer sufficient accuracy or rigor.

Seventh, "These systems can and should be used only as a drafting assistant (structuring notes, summarising papers) with all outputs fact-checked by humans that are capable in the field. Think of these systems as a calculator that sometimes 'hallucinates' numbers - it should never be blindly trusted to do your tax return."

Attach an AI engine to a calculator (using model control protocol) and you can be pretty sure about the financial calculations. The same cannot be said about news media, which often misrepresents the scale or value of the things it reports on, even though the reporters all have calculators.

What should happen in media is that all statements of fact should be 'fact-checked'. This used to be a common practice in newsrooms, though the impact on the bottom line and political pressure has made it less common. Now statements - even those that are obviously false on their face - are reported without qualification or qualms by media under pressure by the political figures that made them.

Neither news media nor artificial intelligence should be fully trusted. There are too many ways they could misrepresent the facts. Fact-checking by readers is essential.

Eighth, "The persuasive but false outputs can cause real harm. These systems are highly persuasive and are designed to be this - hence coherence, the appearance of 'helpfulness' and the use of authoritative language."

Arguably, the harmful effects of AI have been rather less than the harmful effects of news media, through there is still room for change. News media, after all, have started wars on the whim of their owners, propagated medical misinformation, whipped up public unrest, and ruined reputations. News media carry an implied credibility - an image they very carefully cultivate - and this magnifies the impact they have.

Ninth, "Being trained on market data, these systems exhibit large biases towards market benefit rather than societal benefit. Think of it like a little Ayn Rand on your shoulder whispering sovereign individual Kool-aid. In other words, the optimisation leans toward market benefit, not necessarily public good."

News media are in many cases the producer and the conveyor of the market data on which AI were trained. If AI acts like a little Ayn Rand, it's because it was trained on the real Ayn Rand. Market values and market fundamentalism are hallmarks of news media coverage, from the prominence of the 'business' sector, to the representation of stock market increases as 'good', to the description of tax as a 'burden', to regulation as 'red tape', and government actions as 'artificial' and 'interventions' in what would otherwise be the natural order of things.

---

My point here is not to defend generative AI from any particular criticisms. Simon Wardley's advice is sound, and people really should keep all of these things in mind when they use AI. 

Rather, what I am challenging is the supposition that human-generated news (and other 'factual' representations) are any better. It is astonishing that inveterate AI-critics are at the same time so willing to jump on some magazine or news article as though it were gospel. Human-generated media has a long history of getting it wrong, for a wide variety of reasons. Readers, listeners and viewers should never take any assertion for granted just based on some news report.