Manhattan had more tall buildings in 2010 than it did in 1910, but as this very cool map from Brandon Fuller shows it also had much less residential density:
Some of this reflects the expansion of the midtown office district, which crowded out residential housing. But most of it reflects the fact that modern-day people take up much more space than our predecessors of a century ago. The working class used to crowd into tenement dwellings, siblings sharing a bedroom was more common among the non-poor, and the households of the wealthy were packed with domestic servants living in tiny maid's rooms.
The Manhattan case happens to be particularly dramatic because it was ridiculously dense in 2010. But the general principle holds — in a desirable urban area, the square footage of residential real estate needs to increase merely to prevent population density from falling. Flat supply leads not just to a decline in affordability, but a decline in density and a shift away from local retail and walking.