Most GPU drivers are an order of magnitude more complex than a CPU kernel.

There is also the fact most GPU are a bodged on architecture, and fundamentally impossible to really secure in the case of CUDA rootkits.

Fast, but a bad design. This is why we can't have nice things. =3