1. 660 is right
2a. Agree with twinboat - you want to start charging before you get to 11.3v. However, that's not totally "dead" because the battery is still under load when the AGS voltage is detected. If the load was removed, that voltage would probably bounce up to 11.6 or so, which is about 25%. Still very low, though. I would suggest setting the AGS start voltage to at least 11.6v-11.8v.
2b. I think the AGS function is actually part of the remote, not the RS3000 itself. Do you have the RC7GS remote control panel? Or maybe the Link 2000 panel instead? There is a generator stop parameter for the RC7GS that is a percentage of SOC (state of charge), 0-99%. For practical purposes, choose a value somewhere between 80% and 99%. 80% is a decent level of charge, but obviously 100% is the goal. Use that unless you want to limit the run time for noise reasons.
If you have the RC7GS, get the manual to learn about the settings.
Xantrex also has a separate AGS system with its own panel. It doesn't seem to have as many charge paramters to control as the RC7GS, but does a lot with time of day settings and such. It has a different manual to study!