I suspect (but do not know for a fact) that if the key and the car are in close enough proximity they ping each other and wear the battery down. I did notice my Lexus manual has instructions for how to put the fob in a battery saving mode.
After having to replace the new fob from my second key that recently came but I have never even used it was clear that the fob uses battery unless it is locked or disabled. How else could a stored fob run down in so little time?