You can buy a
butyl caulk tape that is made for sealing RV windows. Easy to work with and more reliable than tube caulk.
And yes, removing the windows is quite doable, but you'll need a second person on the outside to catch the window as you push it out.
Once you remove the screws on the outside frame, remove the screws from the trim ring on the inside; then, just push.
A lot of windows will have a bead of silicone on the top of the outside frame- this is to reduce black streaks, not for sealing- the butyl does that.
Scrape off any old caulk and outline the window with the butyl tape, making sure it covers all screw holes. After you've re-installed the windows, trim any excess butyl with a plastic knife; it will pick up bugs and crud if you leave it untrimmed.
The trim that runs along the top/sides seam also comes off easily once the screws/rivets are removed (careful not to bend it!). Use the same butyl tape (or upgrade to
Eternabond; best stuff on earth) underneath the trim ; again making sure it covers all screw holes and cracks.
If there are holes where the screws seem to have nothing to bite (quite likely) remove the screw and fill the hole with caulk. If its somewhere critical like at the very ends or a seam, you can try a longer screw and see if that finds something to catch hold of.