I had the same experience, where I went from ultra, kept lowering settings and fps did not rise at all and stayed below 30 on some maps. It helped that I deleted roengine.ini and re-did my settings from default.
ALso, world detail has a surprisingly large influence (then perhaps light beams, then shadows). Try changing those first.
There is definitely somthing wrong somewhere though in the sense that it seems changing settings should make more of a difference. I personally think that in my case it is not caused by a bottleneck on the pc so much, but some problems in the code when changing settings.