At least that's been my experience.
When I work together with someone in the same place physically - we regularly interact on a variety of levels, developing a kind of implicit trust.
We also have frequent chances to chat on both the casual and work-related topics, which allows us to build mental models for other people.
So a large part of the problem here is an insufficient capacity for initiating casual social interaction.
Also if you do so - it is mostly deliberate, which puts you under pressure to return to work/prevents you from initiating the interaction in a first place.