Re: A mistress?
I think another issue here is that many people just don't understand having activities beyond home/work. That is what they were taught to focus on and they simply cannot understand "hobbies". They are seen as wasting time and money and I often sense a bit of jealousy. My wife and I are totally on the same page with hobbies as we see them as enhancing our lives as well as setting a better example to our girls.
When you have a hobby person together with a no hobby person, things get very messy. Seen it too many times to count. It is often said that money is a key cause of divorce, but in my personal experience when you look under the hood in those cases it is often a disagreement about priorities.
To the OP, do everything you can to communicate. Sounds like the channels may have some static.