I went through a golfing phase about 15 years ago and pretty much ended up at the same conclusion that you mention here. I played quite often for about 10 years. I would have a good round followed by several infuriating or downright embarrassing play. Seems like I just didn't have the natural timing, hand/eye coordination, or rhythm that is needed for consistency.
It all ended one day in Myrtle Beach, back around 2007, after a humiliating round where I shot over 100, playing with a group of pretty decent golfers, I had an epiphany and realized the following:
-Golf is difficult for me
-Golf can be very expensive and time consuming
-I pretty much suck at the game
-Playing really crappy golf is not fun for me
-It's foolish for me to pay lots of money and spend lots of time only to suck at the game, and as a result of crappy play, be miserable, not enjoying the game.
-I am done playing golf
I have not swung a golf club since then