Generally you need some rendering trickery (merging the meshes and using texture atlases, instancing, intelligent LOD management...) to make a whole city composed of individual buildings work performance-wise.
Flightgear has a toolchain to convert OSM data to 3d meshes, and there's currently lots of effort going into making the rendering efficient (the current wisdom is that simple buildings get instanced and only complicated buildings get an explicit mesh,...)
In principle you can learn the toolchain (ask google for OSM2City ) and might end up with the 3d models automagically - though converting them to an Orbiter-suitable format may sill require work - but the raw meshes without any accelerating trickery will have poor framerate.