two engines i´d recommend and why:
1. DXStudio - it´s a british engine built for general purpose game development - it´s not free nor open source, but the license is *relatively* cheap and it can be freely extended with an C++ SDK...
it also makes a LOT of things VERY easy, DXS is built with the SpiderMonkey javascript1.8 engine, which makes for some pretty easy programming (specially for addons) - it also features a layer-based approach which allows 3d renders to be overlapped, which would enable one to use different layers for rendering the VC, near objects and far-away things...
also, making 3d things interactive in DXS is as easy as if they were 2d... it has a built-in raycasting system which automatically detects mouse events over any object
unlike most other platforms at it´s price range, DXS can be greatly addon-friendly - and it can deal with meshes in .x format, through a easily integratable commandline importer, or meshes can be exported straight into it´s own zip-compatible format via the DXMESH C++ API
also, ALL proprietary formats employed in DXS are ZIP-compliant, and can be freely edited (and even created) even by those who don´t have the DXSTUDIO package installed
another option, is that DXS can be embedded in another app as a COM component, which make it a great option to use as a powerful rendering and scripting engine while tucked inside a larger platform (like the current Orbiter engine)
usage of DXS is royalty-free, so only one license is required... which should go for about 100GBP, since orbiter is non-commercial - there is also a freeware license, but it´s a tad limited....
being a beta-tester myself, i can pretty much guarantee that in one way or another, Orbiter could be successfully ported over to DXS
yet a suggest we wait a bit... V4 is coming soon with shader model 3 support and a million other goodies
DXS has already been used successfuly for large-scale simulations, so it´s definitely worth checking out:
www.dxstudio.com
--
second choice: IrrLicht3d - an open-source C++ 3d engine, very powerful and unbelievably fast... i haven´t yet had much experience with it, but i see it can be a very powerful tool in the right hands... it supports pretty much all popular mesh formats and has a plug-in library for sounds...
it also has a built-in raycasting system, although not as worry-free as the one in DXS
IrrLicht is free and open source.... i don´t know how well it would suit something in the likes of Orbiter, as most of the demos seem to feature dungeons and that type of stuff...