Am I a sex buddy or girlfriend?

I have been seeing this guy for almost 4 months. I am 31 and he is 36. We get along great! We both are confident, outgoing people. We text or chat when we aren't together.. We go the gym together everyday, go out to eat, movies, hiking, go out with his buddies, etc.. There has been no talk or physical actions showing that we are together as a couple besides we have had sex a few times (which was great and he was definitely not shy!). There has been no traditional hand holding or kissing or cuddling, that Im use to when Im with a guy. Not even a kiss or hug good bye when we go our own ways.. I have joked about being the aggressor and he said he wouldn't mind it one bit, but.. I dunno.. So.. This has me wondering (at the old age of 31) Am I just a sex buddy to him or does this seem like a guy who is more interested sense its definitely not just sex.. And yes I could ask him but I really don't want to make it feel awkward.. Its not that big of a deal, Im just curious..