First, clean every battery connection you can find. Both negative and positive. Make sure the terminals are crimped properly and not failing.
Corrosion on any of the connections will give you the intermittent, no crank and or the clicking noise when trying to crank. After a few tries, the current burns thru the corrosion and it cranks. Feel the cable ends, after cranking, for one that is warmer than the others. A warmer one is suspect.
As far as the starter running on, that can happen with low voltage cranking. The contacts weld themselve in the solenoid holding the bendix drive engaged. A rap on the starter should make it stop, when it happens.
Second, if it's a " bolt up " starter, not a bolt on style, you may be missing a shim between it and the engine. Without the shim, the bendix and flywheel gear spacing, is so close that the gears will mesh, but will not dis-engage. See if it originally had a shim.