Its no more wishful than expecting a root of trust to stay trustworthy or expecting a trusted name to consistently resolve to the same party.
And there's a lack of motivated attackers for a reason, the attack scales poorly. Maybe you can bamboozle a target or two, but if you go for hundreds, somebody's gonna start wondering what's up with the keys. If you're trying to take over the world, or sell to somebody who is, you're better off focusing on compromising a root of trust or tampering with name resolution, because attacks like that are sustainable at scale.
Maybe it's the same probability of being a problem, but it would be a problem introduced by different kinds of threat.
If you're worried that you'll be targeted as an individual by somebody who cannot coerce a CA but can tamper with the channel you're using to get the public key, use https.
If you're worried that you'll be targeted along with many others by somebody who can coerce a CA, use ssh and ensure that some folks are comparing key fingerprints such that at least one is close to the source.
And there's a lack of motivated attackers for a reason, the attack scales poorly. Maybe you can bamboozle a target or two, but if you go for hundreds, somebody's gonna start wondering what's up with the keys. If you're trying to take over the world, or sell to somebody who is, you're better off focusing on compromising a root of trust or tampering with name resolution, because attacks like that are sustainable at scale.