I often have issues with Steam-games that use the UT3 engine - Borderlands, Bioshock, and now The Ball as well.
Couldn't get FSAA to work, no matter what I tried in my nVidia-CP.
For Bioshock, I set my desktop-settings to the ones I wanted ingame (was the only way for me). Downside was, that I had to do it everytime I wanted to play the game, and then reverse back to original desktop-settings, once I had finished playing.
For Borderlands, however, I used the suggestion in this post:
http://gbxforums.gearboxsoftware.com/showpost.php?p=1654054&postcount=1
And it also works for The Ball.
DX9, on a WinXP-Pro machine, btw.