I did that at age 36 in 1988. My son was in the car in a car seat. Thankfully, it was near home and a neighbor called my DH to bring the extra keys. Only took 5 minutes for DH to get there but I was in a panic. I would have broken a window to get in if it came to that.
So no, you are not alone. And yes, it made me feel very STUPID. I really heard it from DH but I guess it frightened him too.