I generally don't follow other blogs, or Twitter accounts or whatever just because they happen to follow me... and I don't expect it in return.
Usually I start because they've written or posted something that touched me or they've been recommended by someone I respect.
Some I follow as a trial. Just to see if that little hint continues. Sometimes it does, sometimes it doesn't. Every now and then I clean out the list. It's not a bad thing.
Be authentic and don't worry about numbers. It's much better to have fewer followers that feel a connection with you than a long list of people who don't.