There's a few findings that no one seems to have integrated, so this is my attempt. First off, women are higher in neuroticism than men. Neuroticism can generally be described as the tendency towards negative thoughts, so tendencies towards anxiety, depression, worrying and so on. A typical set of findings based on self-report data:
Facets are the sub-factors of the big 5 (OCEAN) traits usually used in personality measurement. They are somewhat but not entirely arbitrary, and there are 6 for each of the big 5. Overall, the neuroticism gap is 0.40 standard deviation (top right), but looking closer, we see the difference is mainly anxiety and vulnerability (the differences are larger when adjusted for measurement error). Basically, women are more scared of things that might happen. This has some plausible survival value considering that women are the physically weaker sex, and often targets of sexual aggression (rape) so it pays off being more cautious (parental investment is also much higher which is the root cause). Furthermore, given women's role in child care, it is wiser to be overly cautious because the cost of failing to detect and fix a problem with a child might result in its death, whereas spending a little more time unnecessarily is less of a cost.
Neuroticism comes with its downsides too. If we look at romantic relationships and their endings, we see that personality predicts dissolution.
The study is based on Household, Income, and Labour Dynamics in Australia (HILDA). There were 5 surveys of the same people over time, which included self-rated 40 items for personality, relationship status (marriage and otherwise). Personality was measured in wave 1, and in the follow-up waves, they then checked if those relationships still existed. The model above tries to predicts which relationships failed based on average of the couple's personalty level (I think). We see that neuroticism predicts dissolution positively with a beta of 0.31. The odds ratio is 1.35, meaning that a 1 standard deviation increase in couple neuroticism predicts 0.35 greater odds of dissolution in the study period. Interestingly, openness also predicts with about the same effect size. I interpret this as being due to 'grass is greener' and openness to new relationships, maybe infidelity, related to curiosity or proneness to boredom. So here we see an example of a positively valued trait not always having only positive effects. One could also imagine that some of this is related to political ideology, where leftism correlates with higher openness, and their independent causal effects may be difficult to estimate. Does openness cause leftism, and leftism causes dissolution? Or does openness cause both also? Who knows. There is a meta-analysis of these kinds of studies with similar findings, but their results are presented less well, so I opted to cover this study instead.
Can we find evidence of the benefits of neuroticism? While there is potentially some survival value to neuroticism, especially in earlier times, does it do anything good now? There's a clue from this Swedish study of fertility of mental diagnoses:
Participants In total, 2.3 million individuals among the 1950 to 1970 birth cohort in Sweden.
Main Outcome Measures Fertility ratio (FR), reflecting the mean number of children compared with that of the general population, accounting for age, sex, family size, and affected status.
Results Except for women with depression, affected patients had significantly fewer children (FR range for those with psychiatric disorder, 0.23-0.93; P < 10−10). This reduction was consistently greater among men than women, suggesting that male fitness was particularly sensitive. Although sisters of patients with schizophrenia and bipolar disorder had increased fecundity (FR range, 1.02-1.03; P < .01), this was too small on its own to counterbalance the reduced fitness of affected patients. Brothers of patients with schizophrenia and autism showed reduced fecundity (FR range, 0.94-0.97; P < .001). Siblings of patients with depression and substance abuse had significantly increased fecundity (FR range, 1.01-1.05; P < 10−10). In the case of depression, this more than compensated for the lower fecundity of affected individuals.
Conclusions Our results suggest that strong selection exists against schizophrenia, autism, and anorexia nervosa and that these variants may be maintained by new mutations or an as-yet unknown mechanism. Bipolar disorder did not seem to be under strong negative selection. Vulnerability to depression, and perhaps substance abuse, may be preserved by balancing selection, suggesting the involvement of common genetic variants in ways that depend on other genes and on environment.
In a figure:
The bars show relative fertility. 1 = average for that cohort and sex. Woe is to be schizophrenic or autistic male! Their fertility rate is reduced by almost 80%. This is almost the same effect size as being a homosexual male:
Overall, the fertility impact of diagnoses is much smaller for women than men. It seems that women strongly selected for mentally fit men, or that mentally ill men fail to do proper courtship, or don't want child if they succeed. Every diagnoses is associated with marked sub-fertility, except for one: depression in women. That's not a coincidence I think. The authors compared the fertility of the index cases (those with a diagnosis) with their siblings without diagnoses:
Here it gets very interesting. Many of the sibling comparisons show elevated fertility. The two largest 'benefits' are depression and substance abuse for women. Since we know that 'unaffected' siblings are actually not entirely unaffected on average, but do not meet the threshold for diagnosis, these siblings will be about halfway towards the mean assuming a sibling similarity of r ~ 0.50. As such, if a standard score of 2 is required to get the diagnosis, the siblings will have a standard score of about 1. I think what these findings show indirectly is that some level of P factor, general psychopathology, which is mostly the same as neuroticism, is adaptive (associated with higher fertility), especially in women. Another way to think about this is to think of a normal distribution and the fertility rate as function of neuroticism. I think it may look like this:
In this model, neuroticism is under negative selection in men (below avg. fertility for men with above average neuroticism), but positively in women (above average fertility in women with above average neuroticism). At the same time, fertility is decreased for those of both sexes who reach the diagnosis threshold. The problem for the model is that the brothers of people with diagnoses also show some elevated fertility for some cases, whereas these should be negative. I am not sure how that happens, but my ad hoc hypothesis is that this is due to siblings compensating for their sibling. A simpler interpretation of the results would be that neuroticism is under positive selection in both sexes, but more so in women. Whatever the case for men, it is clear that neuroticism is under different selection pressure in men and women. Consequently, over evolutionary time, the sex difference will become larger. We can also reasonably infer that this pattern existed in the past, which is how we evolved this difference to begin with.
Speculating from the above, given the findings about relationship dissolution, it seems that men can opt for more neurotic partners at the cost of increased instability. They can counteract this by being particularly agreeable or low in neuroticism themselves, as well as reproducing early in relationships before neuroticism ends them.
Another ad-hoc hypothesis (which I invented on the spot right now): Look to the hunter-gatherer past. A certain amount of neuroticism is good if your most significant reproductive challenge is 'continuing to get along with the other people in your group'. Since women compete with other women through social ostracism, and expelling people they do not like from the tribe, being worried about what your female neighbours think about you is important for women. Men have historically been willing to put up with other men who are substantially less concerned with social harmony and deferring to what the neighbours think - as long as these men bring some other skill or ability to the teams they are in. If you are a really skilled hunter, it doesn't matter that much if you are oblivious to or ignore social disapproval. Your behaviour is going to have to be really egregious before the other people in the tribe decide to ostracise or kill you.
Fast forward to the future. A good many men find themselves similarly in positions where the skills they demonstrate means they do not have to be worried about social ostracism, i.e. "They couldn't cancel Joe Rogan". But many more others are predominantly competing in games of social one-upmanship, and where it is not what you know but who you know that gets you ahead. A little more neuroticism may help these people become more successful.
And thus, when you come by looking at the data about the value of neuroticism for men, you get results which boil down to 'it all depends ....' :)
For more on this angle I highly recommend reading Bateson, Brilot and Nettle's 2011 paper, Anxiety: An Evolutionary Approach. They also argue that negative emotions (namely anxiety disorders) are adaptive and they give a very good explanation for A) Why that is, and B) Why it relates to higher psychopathology, which is generally (personally) maladaptive. More broadly, Nettle's individual differences theory is a good way of explaining the adaptive benefits of neuroticism.