I would, but you can experiment and see what happens. On the larger GMC Acadia we used to tow, the power draw from the coach exceeded the 10A toad charge power feed. We also found that without pulling the fuse, various "body computer" functions operated, e.g. the Auto Traction Control, and we got warning lights on the dash after towing awhile.
IMO the best solution is to put a switch inline with the fuse and switch it off rather than removing the fuse each time. Usually not difficult to do. I mounted the switch under the hood on the Acadia, but you could route it to the dash area inside if you like.
2004 American Tradition; 2014 Buick LaCRosse
Homebase in the Ocala Nat'l Forest near Ocala, FL