They've basically self-funded this project. Yes, the engine was "free" (if you count spending working unpaid for 2 years on the mod as "free") but there's so much more required than just the engine. Art assets, code, marketing, etc. - all that comes at a price.
IMHO it's a good decision they made, releasing RO:O to cover their expenses, raise money, and establish a foothold. Now, when it comes time to produce for UE3, they'll have a much better shot of securing distribution and capital since they already have a successfully released game on their corporate resume.
And if you're asking about UE3, then you've likely seen the images and models that UE3 can display. Those assets are lightyears ahead in complexity and technology. Normal maps, higher poly counts, more complex animations, etc. The higher the complexity, the more time consuming the production of these assets. All that comes at a price in terms of time and labor.