In the tutorial, it says that I need to put Retro and boost main engine since the PeA is too high when you're making orbit transfer to the moon.
I can empathize with your situation. I'm still new enough at Orbiter that I can understand the real problem you are facing.
The real problem you are facing is that you are trying to follow a set of directions like 1, 2, 3 or A. then B. then C.
Tutorials are excellent, but they often fail to mention that what they are doing in the tutorial may not match the situation that you find yourself in.
I have made numerous trips to the moon and I don't think any two trips have gone exactly the same way twice. So instead of thinking in terms of A. then B. then C., I had to learn to think in terms of IF X occurs THEN do Y, ELSE if N occurs, THEN do O.
When you reach the mid-course correction point while going to the moon, there are probably two situations you will find yourself in. You will either be on a course that "overshoots" the moon (your PeA is high), or you will come up short. (In which case you are either on a collision course with the lunar surface, or your initial velocity was so low that you aren't even going to reach the moon at all ... without an additional Delta-V boost.)
Each of those situations requires its own solution.
I had watched the excellent "DG to the moon" tutorial probably a dozen or more times before I finally got myself from Earth to lunar orbit. (Landing on the moon was another issue altogether.)
One thing I remember not understanding was when the guy in the tutorial sets up the Transfer MFD, he presses DV+ until it gets to 3.121k. I thought that 3.121k was the "right number" needed to get to the moon. Period. So on my subsequent attempts, I always set it to 3.121k not realizing that in some cases, you may need a "higher number", and in other cases a "lower number."
It is completely obvious to me now what is going on with the Transfer MFD. You are calculating how much Delta-V you need in order to bring your apogee out to a point that is close enough to the moon that you can "transfer" from Earth orbit to the moon's orbit.
But when I was completely new to Orbiter, I had NO IDEA what "DV+" and "DV-" were for. For example, if I tell you that a restaurant is 3.1 miles from your house, that number is always going to be 3.1. It's never going to be 3.8 or 2.1 or 8.9. So when I was new to Orbiter, I thought 3.121k was some kind of "fixed" number that would never change.
When making my first few trips to the moon, all the MFD's just seemed like these "magic boxes" where you dial a few numbers and "it" just magically gets you to the moon. Now I understand the fundamentals behind the numbers ... and that, in turn, helps me understand when to deviate from what a tutorial says. (But I am definitely still learning. There is A LOT I still don't know how to do. e.g. Going to Mars.
It's still a mystery to me.)
Anyway...
In your case, the tutorial you are reading is using a scenario where the author of the tutorial was "overshooting" the moon but your actual scenario is one where you are coming up short. So if you follow the directions given in the tutorial, they are exactly the opposite of what you need to do. (But on some subsequent trip to the moon, you may find yourself in the same situation as the tutorial.)
Does all that make sense?
Here are some notes I made a while back when learning to get to the moon. I like reading what other people think, or seeing videos other people have made. Each new thing I read (or watch) helps me understand things a little more. So maybe my notes will resonate with you on some level?
(These notes pick up at the point you are having your problem.)
Next we have to adjust our projected orbit around the moon. This particular burn is going to be very situation dependent. I've never had the same exact conditions at the mid-course correction come up twice. (Probably indicative of my inexperience.) But nevertheless I've found that the more I understand about orbits, the easier it is to adapt to the variety of situations I've encountered at mid-course.
In the tutorial, he adjusts the periapsis so he doesn't crash into the moon. This is probably the most common situation to encounter. (Which probably means it's the 'right' situation to be in.) But you can also find that you're on a slingshot trajectory around the moon ... in which case you wouldn't want to follow what he does in the tutorial ... rather you would want to go retro grade to slow yourself down so the moon's gravity could pull you into an orbit.
But there is something to consider here. If you slow yourself down, it's going to take that much longer to get to the destination. While this doesn't really matter in a simulation experience ... because you can accelerate time ... it would matter in real life. So in the event of a slingshot trajectory, I think it's better to wait until you're closer to the moon before making the burn.
In one of my moon shots, my PeA was something like 1.5M ... I wanted to bring that down to about 150km ... but at the same time I didn't want to make the trip take an extra long time. So I waited until I was about 10M away from the moon before I corrected. (And I'm sure I could have waited even longer. The moon's gravity is very weak ... so corrections are relatively easy to make.)
In the tutorial, he fixes his crash course situation by going to PRO GRD and then adjusting to -50 degrees off the flight path. He says -50 degrees is just a "wild guess for a reasonable burn direction"
It looks like he uses PRO GRD to orient himself, then he levels out his horizon manually. He uses KILL ROT to help. Then he manually yaws to the right to 50 degrees ... again using KILL ROT to help.
He does a small main engine burn to bring the PeA "up" to about 140-150km. Prior to the burn, his PeA was -747.8k. His projected orbit prior to the burn was a wide V. It was almost a U shape around the moon ... but not quite. His projected orbit was definitely a collision course with the surface. Having an "almost orbit" is a bit like having a parachute that "almost worked."
For the fun of it, I interrupted the playback of the tutorial and sped ahead to the moon without the course correction. Sure enough, I collided with the moon right on schedule. Though what I found interesting was that this was easily preventable. The speed only reaches about 2500 meters/second ... it would have been easy to go retrograde, fire the main engines to slow down, and then go into a horizontal attidue and use the hover engine to land at whatever random point the collision trajectory was aligned with.
In one of my flights, my orbit around the moon was not an orbit at all. It was a very steep V. My projected PeA was -1.689M. I took a "wild guess" to try and fix it. I went into the prograde position and then yawed to the right to 100 degrees. I did small CTRL+ burns to test ... and it was having the desired affect so I was able to bring up the PeA where I wanted.
At this point, he goes back to PRO GRD just to improve the view.
Next you wait to reach the perilune (a.k.a. periapsis). This part is fairly "standard" orbital stuff. At periapsis, you do a retograde burn to lower the apoapsis. In the case of going from the Earth to the moon, this is a CRITICAL burn because without it, you'll "slingshot" around the moon and head off in some undesired direction.
Reading these kinds of "internal monologues" (what a person is thinking) helps me ... hopefully it helps you too?
Also, here is something I made recently... I took a bunch of screenshots while going to the moon and added captions to each picture as if I were a real pilot making a real trip to the moon.
http://www.facebook.com/album.php?aid=86496&id=1636496947
