The problem is that I am a man and have never been a woman. I've taken the time to sit down and read books about what it is to be a woman in society. I've been talking to women and more importantly listened to them. It turns out that the experience of living in this society as a woman is completely different than the experience of living in this society as a man. I know that's an almost trivial and totally obvious statement, but the sheer mass and volume of that difference is far from trivial.
Epistemic privilege is simply the idea that I have a special kind of access or knowledge about myself. I have direct and unfettered access to my own thoughts, feelings, and memories. No one else has as much information about my inner-world than I do, which makes me the #1 foremost expert on me. Other people, my parents or my wife, might have insight into me or my behavior and may be able to understand things about myself that I might not, but no amount of external observation can match the intimate knowledge of myself that I have that comes from existing as me.
The notion of epistemic privilege does is not restricted to simply just being me, but reaches out to my categorical experiences. For instance, as a sighted person I have a special kind of knowledge about what it is like to see the color red. Contrast that with someone who lacks sight. Imagine describing the experience of seeing red to someone who has never seen anything before. You could go on and on about light waves and photo receptors and so on, but there is an ineffable quality to seeing colors, the experience just can't be reduced to words.
Imagine the difference of beliefs about the world that a sighted person would have compared to a blind person. The sighted person would "know" that the world is colored and that there is an extensive and nearly endless variety of colors to be had. Colors are a fundamental and almost bedrock part of experiences for sighted people: the world just is colored. For a blind person, someone who has never seen anything, the world is devoid of color altogether. The notion of color bears absolutely no weight in their view of the world, it's not even that the world is colorless to them, the very idea and reality of color or lackthereof never even gets meaningfully included in their overall view of how the world is.
As a sighted person it would be nearly impossible to truly imagine what it would be like to have a worldview that lacked color. To try to simulate that, I would have to strip away my entire system of beliefs, even the most basic and simple ones and rebuild them back up without incorporating any notions of color at all. This is probably impossible since I really can't rip the colors out of my head, they are there because of the particular type of experiences I've had.
Sadly, it wasn't until well into my adult life that I wondered what would it be like to exist in a society where I don't see my group sitting in every major seat of power. The first and most meaningful step in any philosophic endeavour is to discover what it is that you don't know. It turns out that I don't know what it's like to be a woman. I don't know what it's like to be black. I don't know what it's like to be Hispanic. In fact, it turns out that my epistemic privilege is limited to all and only my experiences as a white male. Like the blind person learning about color through physics and biology, sure, I can read all the books I want about race and gender and I can talk to myriad people about their experiences. I can struggle to gain insight, but there is an ineffable quality to the experience of being a woman in our society.
Ultimately, what I'm getting at is that women are the experts on being women. When it comes to issues relating to gender inequality: all eyes and ears should fall upon women to be the ones to explain to everyone what the nature and effects of that inequality is. This doesn't mean that men can't participate in the conversation or that it is futile to try and understand women's plight. All too often though, I see men pass judgments and make statements about women's issues, without realizing or respecting the full range and extent of their own ignorance. The notion that a man's opinion on women's issues is just as valid is foolish and, in a sense, almost childish.
Here's a quick example of the difference that epistemic privilege makes. Men: if a woman were to drive by in their car and holler something to the effect of "Nice ass!". You might welcome the attention, you might find it flattering or exciting, but for certain, you will not feel threatened. Even if you are a man who is offended that someone would have the audacity to publicly objectify you, you will not become worried that something bad might happen to you. This is because according to your experience, women have generally never posed a serious physical threat to you. You've grown up knowing that is most situations you would be able to physically over power a woman and that it is incredibly rare that a woman initiates a violent encounter with a man. I feel like I should put that last sentence in bold, italics, and at 30 point font just to emphasize that. So when a woman catcalls you, it is of little consequence.
Most men don't really see the problem in catcalling because it has never been a problem for them. This is the danger that ignoring one's own ignorance produces, the bias and rather pernicious view that "what is good for me is good for all."
So, ladies, if you ever feel inclined to say "You just don't know because you're not a woman." you have every right to say that because it is completely true.