> There's an idea in this book, that when using a mouse, smaller buttons are harder to click

That would be Fitt’s law, which - if you can believe it - dates back to 1954!

https://en.wikipedia.org/wiki/Fitts%27s_law