Chwarae Teg said attitudes towards gender significantly affect young woman.
Helen Antoniazzi, of the charity, said its own research on career aspirations suggested young women felt pressured into gender-stereotype roles.
Cafe owner Bethan Roberts said she had experienced her “fair share of sexism”, even when she was her own boss.
Ms Roberts, 22, has been running The Haystack in Swansea for the last six months with her boyfriend.
She said she had to sack someone because he would “not take her seriously”.
“We had a chef but we had to get rid of him because he just wouldn’t listen to me. He listened to whatever Liam said because he was a man, but me? No, nothing.”
Ms Antoniazzi said: “We’ve all got a responsibility as a wider society to tackle it.
“I think that a lot of people think that we’re worrying about something that doesn’t matter, but actually the effect on young women can be quite serious.
“I think that we need to stop thinking about this as something that the woman facing it should tackle alone.”
She added: “The main thing that people can do is just challenge these things when they see them happening, and also challenge ourselves as well and think of what we’re saying and the effect that might have on someone.”
Ms Roberts said: “I think it’s definitely an issue, especially being young as well. Suppliers and even people in general just don’t take you as seriously just because you are a young woman in business.
“When we first opened we’d have suppliers come in and try and sell us products and stuff, and they would often come in and ask to speak to someone who was in charge and they just wouldn’t believe that it was me.”