Just get one of these:
Supra KeySafe Lock Box, Slimline Surface Mount
and you can leave a key in it securely so you don't have to take it with you at all.
I keep one bolted to the door frame between by storm and inside doors. I got mine at Home Depot. You can pick your own combination of however many numbers you want to use (though you can only use each number once). It's made of steel coated in ceramic and has 4 bolt holes in the back you can only access when it's unlocked and open. Just fine something that isn't going to move and mount it there.
In my case, you'd have to rip apart the door frame to get it off, then you still would have to use a cutting torch or bang on it an awful lot with something heavy to get into it.
Mine holds two keys. In my case, it holds the door knob key and deadbolt keys. There are ones that hold up to 5 keys, but you'd have to buy something like that online.