Here is a little more on the math for choosing gains for a very simple plant model. The math is pretty simple, but the concepts are more complex and are covered early in an undergraduate course
in control theory. One would need undergraduate engineering mathematics including calculus, linear algebra, rigid body dynamics, Laplace transforms, etc. to really make use of an undergraduate
control theory text book. I'm pointing this post at someone with serious high school+home study or early undergraduate mathematics knowledge who might like to have a cook book (but math based)
method to design controller gains, even if all the ideas aren't perfectly clear. If someone were to pick up an introductory control theory book, this might show you the "good parts"

to study
heavily. You can then go back to using all the real "cook book" methods out on the internet, but perhaps starting a lot closer to a useful set of gains, making the searching around for good values
less annoying.
It is also very hard to do it in words and one picture instead of lots of equations. Sorry about the length of this post.
The way I'm doing it here is "classical control theory" in the "frequency domain" (you are concerned at controlling the way things oscillate.) The other more modern technique is called "Modern
Control Theory" ') and involves creating linear models and lots of matrix multiplications and solving these matrix equations. The MatLab tool (for example) applied to control problems can attack
the problem using either paradigm.
The very simple plant model is a "double integrator". This means that the vehicle's attitude accelerates proportional to some command from the controller. (That is, the vehicle physical systems
effectively double integrates some control moment's (torque) attitude acceleration to get back to the way the attitude changes.) An example would be an RCS attitude control system in vacumn. A
thrust level command from the controller becomes attitude rocket thrust that creates a moment, and the moment accelerates the vehicle's attitude. Put another way, the controller commands thrust
level scaled from zero to 1, and the rest of the physical system turns it into an attitude acceleration (radians/second/second). The resulting attitude profile is two integrations of the attitude
acceleration.
See the figure for a Block Diagram and some equations:
Xc, the commanded attitude (for example, an Euler angle) enters from the left. The feed back position measurement is subtracted from it to get the error signal delta-X which is fed into the
controller.
The Laplace Transform of the controller Martin mentioned in the original post is: Gc(s) = a*s + b (This is one of those "trust me" things, see below)
The 'a' gain is the derivative gain. Multiplying by s in this domain is just differentiation. So 'a' multiplies the derivative of the error signal delta-X (that is, the attitude rate), and the
proportional gain 'b' multiplies the error signal itself. So this is just a Laplace transform model of the controller Martin mentioned up above. We are going to multiply another simple Laplace
transform model of the whole vehicle's position response to this farther down, then feed back the position to get the model of the closed loop system. The "closed loop" system response is the way
the vehicle responds with the attitude controller active, which is what we are concerned with improving.
The Laplace transform of the plant is Ka / s^2 (that is, s squared). When you "take the Laplace transform of a function" you get an equation in the complex variable 's' (just take this on faith
for now if you haven't studied it.) Dividing by s in this domain is integration, so this means the plant is simply a "double integrator": the control output of the controller (greek delta in the
figure) is some control effector state: elevator position, RCS thrust level, etc.
The Ka is an acceleration gain: the control effector state is multiplied by Ka to get the resulting attitude acceleration (radians per second per second). For an RCS system in vacumn, (this
matches this simplistic model most closely) Ka would equal the full thrust moment of the RCS system in the axis considered, divided by the inertia about that axis.
So the whole Orbiter vehicle is modeled just by Ka/s^2 ! Very simplistic, and only really fitting for something like a vacumn RCS controller, or a hover yaw controller with not very much
aerodynamic damping, etc. But it is a useful approximation for many other situations. More complex plant models obviously make the math harder to use, but the concept is the same. (One would tend
to use some modeling program like MatLab for a case any more complex, actually.)
H(s) is an optional feedback model. In this case, set H(s) = 1 (no effect here. You could use this for a complex sensor model.)
I included H(s) in the drawing because the equation for the Closed Loop Transfer Function (CLTF) of the model form shown in this figure is straight out of control theory texts. (I won't try to
derive it here.)
The characteristic equation is the denominator of the CLTF. The solutions of the characteristic equation (complex number 's' values for which the characteristic equation = 0 is true) are where
the "poles" of the closed loop system are in the complex (complex numbers, that is) frequency plane. The location of the poles of the model is a "Big Deal".

The pole positions summarize how the
whole systems acts. (See a controls text. Sorry, some concepts just need to descend into jargon and mathematics

)
Since the highest power of s in the characteristic equation is 2, this is a second order model.
This means we can easily physically interpret the systems dynamics just from the characteristic equation. The Laplce transform equation of a simple oscillator is known to be (see a controls text,
again):
s^2 + 2 zeta omega s + omega^2 = 0
Where omega is the frequency in radians per second and zeta is the damping ratio. You use these to solve for equations for the frequency and damping as functions of the gains and Ka. One way to
use this information is to pick 'b' to obtain a (conservative,and probably unfortunately slow) frequency you want the system to respond at, and then set the 'a' gain to get a useful damping ratio
(0.7, for instance). The higher your 'b' proportional gain, teh faster it will act: but it might go unstable if you try and control it too fast.
Notice that because the omega and zeta are also functions of Ka, that changing the moment capability or the inertia will change the closed loop system's dynamics, unless you adjust your gains to
compensate in some way (this is "Gain Scheduling"). For the simple vacumn RCS, you might (for example) allow a faster frequency at empty tank inertia (Ka is higher), but adjust the 'a' gain to
keep a good damping ratio.
Phyically, in the complex plane the plant model starts out with two poles at the origin: it is on the border between stable and unstable. If you disturb it, it will just drift off.
Adding the 'a' and 'b' gains in the controller causes the system response to change. Ideally, the gains will be chosen to move the poles left (more stable so disturbances damp out quickly) and
they also will likely move off of the real axis (meaning the system becomes oscillatory: There Ain't No Such Thing As A Free Lunch.) The "closed loop" response to a disturbance (or a change in the
commanded attitude) with the controller operating becomes a second order oscillator, as shown. You would like this to have a fast frequency (acts fast to disturbances or command changes) and well
damped (any oscillations dies out after a few wiggles so the passengers don't get sick.)

Unfortunately, if you make the gains too high, you will start commanding beyond the RCS system's
maximum thrust. ("Saturate" the controls.) If this happens, you will command so much rotation rate as you speed up your maneuver that you can't slow down in time as you approach the desired
attitude. You will oscillate forever, or worse be actually unstable with each oscillation being bigger than the previous. This happens because this very real physical limit of the maximum thrust
isn't included in this simplistic analysis

, so don't get too near to the limit, or else. (In practice, this means your controller gains will be set such that the system won't act as fast as you
might otherwise prefer.)
The last paragraph went into jargon again, sorry. Hopefully it was sort of useful anyway.