OK, I think I'm legitimately sailing this time.
The sail properties are of a flat board, so it's drag dominated and can't go faster than wind speed, but I think with a better airfoil model that can mimic a fabric sail that can be remedied. I need to verify my wind triangles as I keep forgetting whether the calculated wind speed is coming FROM a direction or heading IN a direction. I am thinking of making an animated weathervane so the user can visually see the direction of the relative wind over the deck to help set the sail angle. I have text output now which helps, but is still rather hard to grok. The Diesel engine is set to idle and does not contribute thrust and the throttle is locked out when the sail is up. I can take down the sail and switch to engine power as desired.
This took more doing than may be apparent. I basically had to eliminate all vessel drag functions in the airfoil definitions in Orbiter as they all assume air is the driving fluid, which creates havoc when atmospheric wind is enabled for the sails. I had to calculate air and water drag forces on the hull independently. I defined a vessel bounding box with its length, beam, draft and freeboard dimensions to estimate projected areas for drag both in the water and in the air, and calculated the drag forces based on those areas, drag coefficients (purely a function of geometry), and using the correct respective density of the fluid (air or water).
I also simplified the touchdown buoyancy model using the bounding box dimensions of the bottom of the hull (area = length x beam). That area was populated with a uniform distribution of touchdown points, with the stiffness of each prescribed by (water density) x (gravity) x area / n_points. I can specify a desired density and the code will automatically generate the touchdown points symmetrically about the center of gravity of the vessel.
I am seeing why the number of touchdown points matters for stability - if only a small number of points are used, their stiffness must be very high to support the vessel, and so applied forces become sensitive to small displacements, causing instability. It seems that 33 touchdown points works nicely for the skiff, and is computationally very light. I applied the correct displacement (weight) of this vessel, and even though the stiffness of the touchdown points is based solely on the hydrostatic model with no information about the vessel weight, when the touchdown points support the vessel, the vessel rests perfectly level and the draft is nearly exactly that specified for the vessel. It's a crude model but it seems to work nicely.
I also realized that I had to take care to not define drag forces in the same axes as the touchdown points or to the keel. The damping of the touchdown points effectively IS the drag force in the Y vessel direction, so setting a drag force in that direction caused conflict and instability. Similarly, I have a "perfect" keel where no side slip is permitted, and so drag in the X vessel direction would not have any effect. I am not satisfied with this keel model as it doesn't allow lateral drift, but it is very stable and allows you to keep direction while moving under sail or power. I am thinking I might have to invoke some "skid" like my VW Thing in order to allow such sideways drifting, but that is a lot of coding for that effect. I'm trying to see if something plausible and stable can be produced with a minimum of code.