With current hardware, it's not feasible to "let the CPU access the GPU directly" for a few reasons:
1. There are approximately a zillion different GPU chips in widespread use, and the whole point of drivers is to avoid having to rewrite the renderers in games for each GPU family and all the variants. Effectively this means that game developers can never hit the metal directly on PC, because there are simply too many chips that would have to be supported. That work would have to be repeated for every single game (engine) developed, which would be a colossal waste of manpower. This is why we have drivers.
2. Letting applications (games) access hardware directly would open up a giant security hole that hackers and other deviants would try to exploit to take over a machine. This would not go down well with business users.
Some sort of virtualisation scheme might one day make it possible, but you won't see it any time soon.