Only you can judge for yourself. I wouldn't do it. I've driven it many times in the Toad. I've driven the RV towing down many of the "famous" tough passes, i.e. Wolfcreek Pass, Teton Pass, etc. But no, I'd go west and come down via 62/145 Placervile/Telluride/Rico/Delores. It's a beautiful trip too. There's even a great NFS campground at the top of the mountains on that side.
Stop in Delores for a few days and hit the National Park - then on 160 to Durango.
2017 Winnebago Adventurer 37F
2016 Lincoln MKX Toad