First of all, gender isn't a binary. There are intersex births all the time (1 in 1500) and chromosomal expressions that defy the confines of XX and XY. And if we're all a blend of Yin and Yang, then how is that going to fit into a binary outlook? Insisting that it's a binary, rather than a continuum is indeed a patriarchal construct. Is there such a thing as a male or a female essence that is more spiritual than physical? Maybe? But that's going to be pretty hard to substantiate as well.
We don't actually know what of our personalities or behaviors/interests are biological because there's no way to raise a child outside of a culture (without traumatizing them). Cutting edge neuroscience says that there is no such thing as a gendered brain, so those differences later in life are pretty much 100% due to socialization and the opportunities that one had to develop different area. Get Legos when you're a baby and play with them a lot and you'll probably develop spacial relations, etc. Don't get those, and you probably won't have the same kinds of skills. Gender indoctrination begins at birth and is omnipresent even when parents try to mitigate it.
Hormones may exert some impact but everything I've seen says that this too is incredibly diverse and not anything close to a binary. After all, Donald Trump, Ru Paul, Mikhail Baryshnikov, and Pee Wee Herman are all men. It's reductive to say that testosterone makes men be like XYZ in part because it doesn't do that in a uniform way, because it impacts a wide variety of things other than aggression, and because women have testosterone too, and in some cases fairly high levels of it. Men have estrogen and progesterone.
In other cultures and at different times, what is expected of men and women is very different than what we have in the modern West.
"In a 2005 meta-analysis of 46 studies conducted over the past 20 years, Janet Shibley Hyde, PhD concluded that males and females are more alike than different. “Furthermore, Hyde found that gender differences seem to depend on the context in which they were measured. In studies designed to eliminate gender norms, researchers demonstrated that gender roles and social context strongly determined a person’s actions. For example, after participants in one experiment were told that they would not be identified as male or female, nor did they wear any identification, none conformed to stereotypes about their sex when given the chance to be aggressive. In fact, they did the opposite of what would be expected — women were more aggressive and men were more passive.”
"The other day I wrote a story about the Na of China who have about as fundamentally different a culture from the West as is possible. It’s a society where family is centered around a matriarch, and everyone lives in the house of their mother or grandmother along with their brothers and sisters, as well as the children of the women. There is functionally no such thing as a father or a husband. Men and women have love relationships but never live together as spouses. Any children belong to the family of the woman and her matrilineal clan.In this culture, women do the farming and a lot of the heavy physical work. Men build houses and tend to livestock. They are quite involved with childcare, although men help to raise their siblings, and the children of their sisters and cousins, rather than the ones they have a biological connection to."
"As late as the 4th c BC many women in Egypt were the heads of their households, took part in business transactions, including contracts and litigation (sometimes against their own fathers or husbands). Men were reported to have stayed home to do the weaving. One of the earliest archeologists of the pyramids of Egypt, Sir William Flinders Petrie, wrote in 1925 that, “In Egypt, all property went in the female line, the woman was the mistress of the house; and in early tales she is represented as having entire control over herself and the place.”
So how do we know what is really innate when it's so varied from individual to individual, from culture to culture, and from era in time to era in time? I'm not saying there is no such thing as gender, but I do think it's a huge continuum and not a binary, and that we have no real way of know what part of that is biological.