I go to my cabin for three to four days at a time, then go home for a week, then come back again. I take my food in a cooler with ice packs, turn on the fridge when I arrive and turn it off again when I leave, taking the perishables home with me. Is it more efficient for the longevity of the fridge to keep it running during my absences?

I don’t think you need to leave your fridge running all the time - you spend more time at home than away. Turning the fridge on and off every three or four days shouldn’t hurt the compressor, as, if left on constantly, it will turn itself on and off throughout the day anyway. So by turning the fridge off when you go away, you’re probably extending its life rather than shortening it.
To save even more energy, what about a smaller fridge for the cottage, just large enough to hold your four days’ worth of goods? If you want a cold fridge waiting for you when you arrive, you could hook a timer up to it, so it turns on a couple of hours before you get there.