Gender and age effect everything about almost every human interaction. This is not something you can get around, well unless your writing about people who are not human, or who have been altered or something of course. If that is what your doing feel free to ignore me in this case.
First I am not saying your can’t or should not have one gender doing things they normally don’t (say a women become infantry in the army, or a man become a daycare worker), or go against the norms in some fashion. You can and often should do that, characters in books tend not to be typical. It does effect things, same as if your characters was albino would effect things. But that is hardly a reason to not do it.
What I am talking about is how characters interrelate and interact. Even a man who is happily married and has no wish to cheat on his wife will treat attractive women differently then he would a man.
Think of a simple scenario, like a person stopping you on the street by gripping your arm tightly and asking directions. You would respond differently if the person was a old vs young, and male vs female. It may not be much differently, but it will be there.
Ideally you want things like age and gender to be ingrained into the characters such that you could remove all direct callbacks to either age or gender and it would still be clear (at least if you knew the POV character well enough). This in fact is one subtle way to character build, how they treat varius kinds of strangers.