I had a similar situation. When plugged into shore power, my AC would work and the house batteries would charge most of the time, but randomly, the AC would go off and nothing would work. I started by going through all of the wiring to the transfer switch (ATS). I discovered that the white wire going to the ATS was loose. It was not only loose, but about 6 inches of the wire insulation was burned. My transfer switch was an old mechanical model, so I replace it with a digital switch and installed all new wiring. Everything works great now (knock on wood!).
When going from shore power to generator, the older models wouldn't always switch automatically. You could burn them up if you were on shore power and started the generator. The newer models switch over automatically whenever you start the generator.