32bit applications are limited to 2^31 bytes (2GB) normally. Under a 32bit OS, they can use up to 3GB with some tricks. Under a 64bit OS, they can use up to 4GB with some tricks. In other words, only a 64bit OS should be required, a 64bit executable should not be necessary since the game won't use more than 4GB at a time anyway.
VRAM has nothing to do with this limit. Although, many games use managed DirectX resources, which means that all textures and models are copied to both main memory and video memory, to allow for quick reloading (after an alt-tab for example) and swapping.