I spent most of my late 20's and early 30's being an avid golfer who worked like hell at it and when I calculate what I spent during that run I just have to laugh. It's a stupid expensive hobby, especially if you want to be pretty good and play nice courses. You are right about both issues....its expensive and people don't like slow games anymore. Also we live in a time crunch now and taking 4-5 hours to drive to, warm up, play a round and get home seems like it doesn't fit the life flow for a lot of people.
Once the 2nd kid came along I knew my time was up. The oldest was getting into youth sports and me being gone all day Saturday was going to be unapproved behavior. Then I started coaching the youth teams and now I haven't played a round in 3 years. Honestly don't miss it much. Hope one of my sons picks it up so that I can get back into it eventually though.