There are a few issues if it’s AI-written/edited/drafted:

1. AI generated text works off of whatever model it came from, whatever prompt was used, and probabilities. It’s been described as a “next token predictor” and that, for the purposes of this answer, is how I’m describing it as well. First problem: it’s only as good as the above.

2. Because of the above, its accuracy relies on probabilities. It can confidently give me wrong information, and as someone who would read this book to learn a topic I don't already know, I have no frame of reference to know whether I’m being fed accurate information.

3. Since it greatly speeds up the time it takes to physically write material, there is a possibility the author didn’t closely vet the information provided by the LLM as a next token predictor. That means it could be wrong in subtle ways, and the only way for me to know is to rely on knowledge I don’t have.

4. Had AI not been used, I could conceivably go on the past work of the author, rely on the person who has the expertise who wrote the forward, or the fact that physically writing a book takes prohibitively long enough that the dopamine from having written a book doesn’t outweigh the amount of time it takes, and if you are going to take months to years to write a book, you’re going to not want that effort to be in vain. You’re going to do your diligence and not just “wing it”.

AI makes it relatively trivial to appear knowledgeable and to get that dopamine rush from saying “I’m an author” and appearing to be an expert in a field without being an expert in a field. For some, that temptation is too great and there’s no way to know without the passage of time. Time for folks to review the book who know the subject matter. Time for reviews. Time for folks to put signs up that say “don’t trust this material.” In the meantime we still have that content out there, further infecting new AI models and people into believing it’s accurate.

That’s why the use of AI is dangerous, and is especially dangeorus without disclosure.

Sure, but you can build an agentic system which verifies outputs and fact corrects, with probably better accuracy than a human would.

As well, if people were a bit less anti-AI then maybe we can get disclaimers that things are written by AI that don't immediately torpedo any value the writing has.

As a professional mathematician it seems that AI is already at the point where, when well guided, it both does better research and writes better exposition than run if the mill mathematicians.

Well guided is a qualifier that matters.

In my opinion, doing good mathematics is very different from writing good exposition.

I have observed that models are not able to effectively come down to a human level. They are mediocre at approximating what a first-time reader would want to know. If you will forgive the anthropomorphization, they are not very empathetic. In this book for example, the author pauses the flow of thought with countless little anecdotes, anticipating where the reader might have a question or want to know more.

Of course, a well-guided AI is capable of anything a human is capable of. But is that not tautological?

That's not a fair comparison: I choose what I read not from the middle of humanity. I choose skilled writers. They write far better than AI. I'm through the opening volley of this book, and it is far better than the best AI prose I've read.

Tangent: I use AI for code, but I've had to ban it from writing readmes and doc comments; the AI models I've used produce text I'm unable to parse, and assume my readers would not be able to either.

I think a lot of it actually comes down to the perception of where the content came from. I brainstorm and riff with AI all the time, but I know what I am communicating with, I don't get the same sense of ickyness. Blogs and books and all these things that are claimed be to written by a human, but are actually just AI, is frustrating. Honestly, the content may be perfectly fine, but I feel something irritates humans when they think they have been deceived.

I’d be surprised to see current LLMs write as well (and as non-annoying) as the writing in this book.

Because it will absolutely have hidden errors.

Thank goodness human books never have hidden errors.

> What is the problem if it is AI written? Properly guided AI writes better than most human expositors.

I don't understand why HN cannot ever say the first sentence without the second one.

Most people agree that using AI for research or finding better words is fine, but then can't upvote or have a meaningful conversation because of the performative politics and wild delusions about a scifi future.

Dude, all we have with LLMs is an NLP search engine. Get over it. It's been around half a decade of this nonsense. There is no "step change". There is no "hockey stick" curve. The models have not "dramatically improved compared to last year" regardless of what year it currently is. There is no big bad overlord coming for you. Nobody is getting replaced by AI. Even frontier AI can't replace local AI. It's just the logical next step towards better search that assumes you want the "I'm feeling lucky" button a little too often.