I used to live aboard a 36 Marine Trader trawler style boat. I know it's not a sailboat, but many of thing things I noticed will still apply.
First I would start by saying that I was crazy, I had no experience owning a larger boat with things like electrical systems, plumbing systems, heating or cooling etc. I was in my early 20's and really had no idea how to fix anything and maintenance was not something I was keen on at all. But the moment I bought the boat I was hooked. I loved it. I used to work in Manhattan NY, but I hated city life, although I did enjoy the social aspects. So I kept the boat right across the Hudson River in Weehawken NJ. I loved waking up and looking at the NY skyline. It was beautiful. I loved that I had a marina with an awesome steam shower, work out room, and entertainment room. Right across the street was a Houlihans restaraunt, a Ruth's Chris steakhouse and a short cab ride or ferry ride away was all the fun either Hoboken NJ or Manhattan could provide.
Living on the dock with all the "interesting" people that decide to live on boats was great also. Coming home from work on a Friday evening always resulted in 4 or 5 offers to stop by for a beer or some food as I passed the other boaters. Usually there would be a group just hanging out and enjoying themselves and it was always just kind of nice knowing that there was always company to enjoy.
Now to the nitty gritty, I was woefully unprepared for all things I would eventually find out I knew nothing about.
I'm running short on time so here is a list of some of the things I can think of off the top of my head.
Never enough electricity, always popping circuits one way or another.
Everything was always much more expensive on a bigger boat ( i was used to my 17' Boston Whaler)
Humidity, humidity, humidity in the winter the condensation on the windows was ridiculous. The rest of the time it was just unbearable.
Laundry-I don't know but I always seemed to use more clothes living on the boat.
Space- there were always so many things I thought I needed and never really did, and also so many things I could have used but didn't realize.
How little I would use the boat because I never wanted to stow all the things that just living on board accumulated.
All that said, I loved it and lived aboard for over 2 years until I decided to sell the boat and buy an even larger one. I actually sold the boat for $5k more than I paid for it, but never did manage to buy the bigger boat. I found a cool apartment and bought that instead. Now almost 20 years later I'm looking to do it again. I loved that time of my life. All the people I met were great and they all helped me whenever I needed help. I certainly took more from others than I could give, but I don't think any of them minded. They were great people.
As far as picking the smallest you could deal with, I'm not sure I agree. The biggest thing I would recommend is make a real decision about the kind of person you are. Are you really planning on getting out and sailing a lot or are you more interested in the lifestyle of living aboard. The two are somewhat at odds with each other and finding the right compromise is the most important thing in my mind. If you are not going to leave the dock that often, and in my experience that was much less than I thought I would, than bigger is better. Comfort is key and space and storage are super important.