No, that's a false equivalency. Demonstrably, mainstream norms of masculinity are what drive violence, including sexual violence, as well as other cultural issues. While not all men are perpetrators, pretty much universally all men are participating in and upholding at least some of these harmful norms. And, even if by some miracle some men aren't, unless they are challenging those harmful norms, they are complicit in the problems they create.

If you haven't already, read Elle Beau's recent story about that. As she likes to say, "We have the culture that we tolerate", and "I didn't do anything" is actually a huge part of the problem.

https://medium.com/inside-of-elle-beau/men-need-to-lead-the-way-91affad0b1c8