I'm not quite sure what you mean by "average out". If your vessel starts out with zero angular momentum, it will have a small non-zero angular momentum after one orbit, so they don't average out in that way. This is because over the course of the orbit a torque will be generated on the vessel as the axis with the highest PMI rotates away from the local horizontal plane.I don't think the ISS does this, though, because it uses gyros to store angular momentum, which from gravity gradient should average to zero over an orbit, someone correct me if I'm wrong. In Orbiter, as far as I know, there is no way to model gyroscopes yet, at least not without a way-cool custom dll.
The ISS also has to deal with some aerodynamic forces from the thermosphere. This is why it flies in the torque equilibrium attitude I mentioned earlier so that the aerodynamic and gravity gradient forces almost cancel out, so the CMGs don't have too much work to do, even over the course of a single orbit.
And yeah, am I currently wading through a document titled "A Geometric Study of Single Gimbal Control Moment Gyros - Singularity Problems and Steering Law" as I plan to shortly start work on a CMG solution for ISS Ultra. This document is mostly about single gimbal CMGs but it also contains some useful info on dual gimbal CMGs as used on the ISS. If anyone can point me in the direction of some reference info on dual gimbal CMGs, particularly their steering law, it would be most appreciated.
Just to clarify, you shouldn't be aiming to "killrot", ie, kill all rotation, since that will be too unstable. What you should be aiming for is to tidally lock the rotation of your station to the earth. In other words, based on the PMIs you posted earlier, you want to have the Z-axis always pointing parallel to the R-vector. This means you should have the Y axis perpendicular to the orbit plane and hence a rotation around that axis of 360/(orbit period) °/s.And no matter how finessed, your gonna need to killrot sometimes no matter what as you suggested.