You are correct about highway #31 between Swan Valley (el. 5277ft) and Victor, it also climbs over Pine Creek Pass (el. 6764ft). "OK" if you take it easy though. The CG will be on your left as you get into Victor.
A better way is to go north from Idaho Falls on US 20 to north of Sugar City and take ID #33 to Victor. Better road with a few gradual climbs. From Tetonia and on south a fantastic Idaho view of the Tetons. Comming from the north into Victor turn right at the flashing light (intersection of #33 and #31) the CG will be on your right as you leave town. It is only about 8-10 miles farther that way. Never stopped at the CG but it looks "OK" though.
When in Victor there are two things that you must do/see.
* Take a drive to Swan Valley and the Rainey (sp?) Creek Store (NE corner of the #26 and #31 intersection and have an ice cream dipped with a "square" ice cream scoop. When our boys were small we had them convinced that ice cream dipped with the square scoop was better than the "regular" spherical scoop.
* The other thing is to go to the Bar J Chuckwagon dinner/show just outside of Jackson, WY. check out their site. http://www.barjchuckwagon.com
I guarantee it will be the best $17.00 you will ever spend.
Also go SOLO over Teton Pass (el. 8431ft) between Victor and Jackson. It is about 5 miles up and down at 10% grade.
Let us know about your trip when you finish it. PM me back or make another post if there is anything else that I can do for you.
May you have fair winds and following seas.