60 km deviation for a LMO to Phobos trajectory sounds like a lot. Normally I get an oscilation of ~20 km for targets that are within the same SOI as the spacecraft.
I am not going to pretend that I know exactly what's going on or the math behind it, but I can share what I have found to work reasonably well.
The map program's predictions work best if you have non-spherical gravity turned on.
Eventhough the inclusion of non-spherical gravity in the map program is automatic by default, I have found that predictions with non-spherical gravity turned on are more stable than the ones without.
So, make sure that you have non-spherical gravity turned on in the Orbiter Launchpad→Parameters and "NonSpherical 2" in Config\IMFD5.cfg
Use a small step size and max out the legs per frame in IMFD5.cfg
In IMFD5.cfg "LegSize 1" and "LegsPerFrame 64".
This will have a small impact in your frame-rate, especially if you have an old computer, but it's worth it. One more thing it does is that it somewhat limits how far "out" the prediction will go, but it's only noticable when you are going on long journeys, thus need a long prediction of the trajectory. If the prediction doesn't extend as far as you want it to, you can always change the "Accuracy", "Legs/Frame" and/or change the "Hyper Limit" to "No". on the fly from the config page of the map in IMFD.
Set the error tolerance to a lower value that the default one.
In IMFD5.cfg change the "AdapTol" to 4 or 3 (from 8).
The manual suggests that a lower value = higher accuracy for this variable, but I have found that changing it to 1 impacts the time it takes for a change to be shown in the map program's plan. For example when you are planning a burn with the Delta velocity program on one side and watching the outcome of that burn in a shared IMFD map, it takes longer to update the prediction with an AdapTol value of 1. Setting it to 4 works best for me. YMMV.
Other than the non-spherical gravity, all these changes can be made on the "fly" by pressing the MOD button 3 times in the Map program, but I highly recommend to simply edit the Config\IMFD5.cfg once and save it.
Here are the contents of my IMFD5.cfg
Code:
// Configuration file for BaseSyncMFD, I-MFD, AeroBrakeMFD
Color_01 0x0000DD00 // Current Ship orbit, Source Orbit [Bright green]
Color_02 0x00008800 // Additional Map lines [Dark green]
Color_03 0x0000BB00 // Base Text color [green]
Color_04 0x0000EEAA // Hilighted text
Color_05 0x0000CCFF // Target orbit, Some Warnings [Orange]
Color_06 0x00EEEEEE // Selected Item [White]
Color_07 0x00FF5555 // Planned trajectory [Blue]
Color_08 0x00666666 // Planets [Dark Grey]
Color_09 0x000000DD // Warning lights [Red]
Color_10 0x00EEEEEE // [White]
Color_11 0x00999999 // Headlines, Some planets [Grey]
Color_12 0x0000BB00 // Adjustable Items
// Multibody predictor configurations
Rectification 0.005 // Rectification constant for trajectory engine
NonSpherical 2 // Use Non spherical gravity on low orbit prediction in Map Program, 0=Disabled, 1=Auto, 2=Always ON
LegSize 1 // Legsize factor used in Multibody predictor
LegsPerFrame 64 // Number of legs calculated each time step
Celbody 1 // Use Orbiter's Celbody Interface to improve accuracy of the map (1=On, 0=Off)
Adaptive 0 // Use Adaptive step-size control (1=On, 0=Off)
AdapTol 4 // Error tolerance (Lower value = higher accuracy)
Integrator 0 // Default Integrator 0=RK5(6), 1=RK6(7), 2=RK7(8)
// General Configurations
RefAltitude 65e3 // Reference altitude for atmospheric entry interface
ReEntryAngle 5.5 // Default ReEntry in degrees
DateFormat 0 // Default Date Format 0=MJD, 1=GET
UseRegres 1 // Use Nodel Regression Estimation (1=On, 0=Off)
IgnState 1 // Use numerical ignition state propagation
FastUpdate 1 // Use Fast Update of MFD Screens
// Key Setup
BaseSyncKey 0x30 // StartUp key used by BaseSyncMFD
IMFDKey 0x17 // StartUp key used by IMFD
RotationKey1 0x26 // Rotation Key "L"
RotationKey2 0x2C // Rotation Key "Z"
// Auto-burn options
AB_Rate 10.0 // AutoBurn MaxRate (degrees/s)
AB_RCS_Th 2.0 // RCS Level Treshold m/s
AB_Thr_Th 20.0 // Throttle Down Level Treshold m/s
Hope this helps
