It sounds like you've had an interesting and exciting life, and I'm guessing that many of the people you don't fit in with envy your ability to escape social pressures to conform.
You seem to have very early in your life had a strong sense of who you were and where you wanted to go with your life. Some people feel less confident about their choices.
With maturity, many of us begin to care a lot less about seeking approval from others. We wonder why we tried to fit in with a group that we don't even like very much.
Here's a link to something I wrote about that: https://medium.com/age-of-awareness/aging-allows-us-to-say-fuck-off-7be31948e172