Ever since Mary-Kate and Ashley Olsen appeared on TV screens while shouting an ecstatic, “You got it, dude!” on Full House, people have had a weird fascination with them. Even today, with the twins now 31 and running huge, profitable fashion houses, their privacy and frequent media silences have led many fans to obsess over the women and their lives. What happened to them? How come they don’t look alike anymore? Why do they look like they’re always drowning in couture?

While the Olsens’ current lives continue to baffle fans, perhaps the twin’s super reclusive lifestyle is a response to the relentless media scrutiny they’ve received for most of their lives. Since they were born, the twins have been thrust into the spotlight without much say on the matter. They were millionaires by the time they were seven, and magazines had a field day when Mary-Kate checked into rehab for anorexia in 2004. The Olsens never had a chance to be “normal” children, no matter how much they said they were in interviews.

In 2010, Mary-Kate admitted to Marie Claire that she and her sister felt like “little monkey performers.” She said, “I look at old photos of me, and I don’t feel connected to them at all. I would never wish my upbringing on anyone.” Her remarks shocked fans who now saw their warm feelings of childhood nostalgia taunted by darkness. In fact, once you look back at old interviews and read between the lines you start to see that the twins’ successful juggernaut of a career wasn’t always so perfect. Below, 11 things about the Olsen twins’ childhood that explain why they’ve rejected fame.

Love J-14? Be sure to subscribe to our YouTube channel for fun, exclusive videos with your favorite stars.