Given a conjectured Jewish population of several million during the Roman Empire, this is one heck of a population "bottleneck".
James Stone, an insurance executive, follows with an interesting, speculative paper on how the population diminished so much compared to its size in the Roman Era. He concludes as follows:
It is, in the end, the triple combination of carnage at the hands of crusaders and like-minded messianic populaces, attrition during forced exiles, and murder compounding the effects of the Black Plague that seems the most likely explanation for Carmi’s otherwise inexplicable bottleneck.
When all is said and done, religiously inspired violence over a protracted period cannot be discounted as the most potent contributor to the Ashkenazi bottleneck.
And what accounts for the times and places of "religiously inspired violence"? This paper suggests bad weather played a key role.