Steven Rosenbaum started writing his book The Future of Truth: How AI Reshapes Reality in 2022, around when ChatGPT launched. Initially he didn’t use it at all, “But as the writing moved forward into 2023, 2024, it got better and I got better at using it,” he said. “To be clear, it never wrote a page of the book,” he added. “But it became a research partner. I would ask it for quotes on certain things, and it would deliver them. They would occasionally be spectacular, often serviceable, and then, in very odd ways, just staggeringly wrong.”

“I kept thinking, I’ll be really careful, and I’ll double-check everything,” he said.

In May, the New York Times reported that Rosenbaum had included “more than a half-dozen misattributed or fake quotes” in the book seemingly generated by AI. Rosenbaum, a media entrepreneur, had previously acknowledged that he’d used AI tools during the research, writing, and editing process, but the Times investigation was nevertheless mortifying — for both Rosenbaum and his publisher, Simon & Schuster. The book-publishing industry had already been wrestling with the prospect of a flood of AI-authored texts in the fiction market, and now the Rosenbaum scandal was showing the way AI could blow a hole in the nonfiction sector, too.

Nonfiction publishing is uniquely vulnerable to AI because the industry has long neglected to do anything to ensure the books it publishes are factually accurate. “People outside of the industry don’t understand that, contractually, publishers are not obligated to fact-check,” said Paul Bogaards, the longtime marketing and publicity executive at Knopf who now has his own PR firm. Fact-checking is not a service publishers will pay for, though they sometimes encourage authors to seek it out on their own dime. But fact-checking is expensive: Hiring an outside checker can cost between $7,000 to $10,000 per book, or even more depending on its length, which might not be feasible for an author with a modest advance.

Worse, it seems publishers have no idea what to do about this glaring vulnerability. “We don’t have systems in place,” said literary agent Alia Hanna Habib. “For every contract, there is a conversation, and it never really feels like anyone has the right answer,” said one editor at a major publishing house.

Editors, writers, and agents say the problem is likely already rampant. “I feel like everyone is passing off AI work as their own and most of the time don’t say anything about it,” said a senior nonfiction editor at a major publishing house. “I brought this up last year because I had been finding so many errors in the books that I was editing. I have just been told over and over again that the publisher can’t take on the responsibility of fact-checking or hiring a freelance fact-checker because that shifts the responsibility onto us.”

Asserted Rosenbaum, “Anyone who is a working writer today who sits in front of a computer, either doing longform or on deadline or at magazines, whatever the cadence of your work is, you’re using AI one way or another at least in part because it is not only seductive as hell but it’s really incredibly valuable.”

A lot of writers would beg to differ with that statement. But it does point to the fact that there is no industrywide standard on what AI usage, if any, is acceptable even as everyone I spoke to seems to agree that guardrails are necessary. “A lot of authors are well intentioned in their use of AI and don’t want to rely on AI to generate work that they would then present as their own,” said Todd Shuster, co-founder of the literary agency Aevitas. “But they might rely on AI for some research or ideas around the structure of the book or outline. And the author then sort of forgets or denies or suppresses the extent to which they relied on the AI for such research. And before you know it it’s not only that they’ve looked to AI for assistance, they’ve actually generated texts that they’re including in their proposals or manuscripts.” (New York does not allow writers or editors to use AI tools for drafting, outlining, or editing.)

Contracts contain clauses in which the author warrants, among other things, that they are the sole author of the work and that the work is original, indemnifying the publisher. “But as of now, most publishers don’t have contractual language that’s specific to AI concerns,” said Habib. “Therefore, when AI allegations are levied, you’re in a kind of difficult spot. There’s some contractual language around saying that something has to be an original work and against plagiarism, and I would say AI work would fail that test, but others might disagree.”

“I’ve always felt that, despite my ethical holdups with AI, there should be a more in-depth conversation on what things it absolutely shouldn’t be allowed to be involved with,” said one editor at a major publishing house. “Nobody seems willing to have that conversation.” An editor at another publishing house said their company sends authors a guide about not using AI in certain ways, “But it’s not legally binding. Once you’re at the editing stage, this is basically a business of trust.”

Agents, too, are in an ambiguous position. “People are saying, ‘Oh, what if we just have the writer declare that they haven’t used AI?’ That’s a slippery slope, because what is the legal recourse for that?” said one literary agent. “And if they used it just for research, is that fine? Theoretically, that’s how Rosenbaum used it, but that still led to hallucinations and inaccuracies.”

At the Gernert Company, literary agent Chris Parris-Lamb said they have “talked about putting something on the submissions page of our website that we do not represent work in which AI has been used to generate the text, in whole or in part. But we’re not asking people to sign an affidavit when they submit their manuscript, and, unfortunately, shame is not the deterrent it should be.” Some agents and editors are keeping this front of mind as they sign new clients, spending more time vetting writers for their human expertise. “If I’m getting you a six-figure book advance, I don’t want you to be putting it into ChatGPT,” said a top literary agent. “Go to the fucking library.”

Multiple people noted that a fact-checker would almost certainly have caught the fabricated and misattributed quotes in Rosenbaum’s book, some only a Google search away. Rosenbaum said his book had three rounds of proofreading but that he did not hire a fact-checker, something that “was never suggested to me by the publisher,” he said. (He has since personally hired two fact-checkers, who are working on a new version of the book.) He added that the Times article “sounds like the book is rife with errors and in fact if you count them, it’s five, and of the five, one is a citation error. So it’s four, which my guess in nonfiction is below the average. But I wrote a book about truth, and I got four things wrong, and the Times wrote about it, so I’m going to be the poster boy for now.”

Even full transparency about AI usage could pose a problem. “My belief is that unless there’s full disclosure of this, it’s improper. And even when there’s full disclosure, there’s the risk that you as an author, if you’re including AI-generated content, are actually engaging in copyright infringement, because the LLMs often spew out significant amounts of prose that are verbatim copies of prior published work,” said Shuster. “If those works aren’t in the public domain, the defense of fair use is flimsy if not inapposite. These are the challenges facing us.”

AI-detection tools have become more effective over the years but are still not entirely reliable in their assessments, sometimes mistaking human writing for computer-generated work. “As a representative for authors, I really worry about the prospect of false positives,” said Parris-Lamb. Still, “There’s increased openness” to these tools, said Shuster, who has been consulting for Pangram, a start-up whose namesake AI-detection tool purports to be 99.98 percent accurate. Shuster has been introducing publishers and other companies to the software. “I don’t know how feasible it is to just run every manuscript through one of these programs,” said Parris-Lamb. “It’s probably still going to be something that’s done when suspicions are raised.”

And “anything that costs money, publishing is allergic to,” one literary agent noted. “As usual, I think the onus is going to fall on the authors and agents.”

• Bari Weiss Chooses War
• Scott Pelley Accuses CBS Execs of Incompetence, Bias in Fiery 60 Minutes Exit
• Who Would Want to Buy BuzzFeed?

See All