sonnyjim welcome to irv2.
If you look at this PDF
you will see how your fridge is installed in wall cavity page 23 on down. It may not be your fridge just want to show a installation.
At the top of your fridge at the rear you will see baffles that direct the hot air from your cooling coils up through the roof vent.
If the baffle has fall-in out of position more cold air can come into your living space.
When you look from outside at you fridge you may see insulation on both sides of fridge to stop cold air from entering your living space.
My fridge also has insulation blanket on top of fridge between inside wall and baffle.
Previous owner may have had work done and they never correctly replaced the baffle or any insulation that was in place.
You maybe able to check from your roof vent when you remove it and see whats up if not you may have to pull fridge.
The baffles have to be in place for proper ventilation of your fridge.