Majority of time is spent at work and with work colleagues. Even if myself and spouse are both WFH, we are interacting with our colleagues, not each other for most of the day. So while your deep relationships are with family and friends outside of work, on a sheer hours basis you spend more of it in the physical or virtual company of your colleagues if working a standard full time week. And for me anyway, the mental challenges and stimulus right now are coming from my work (sedentary occupation, but I am sure similar applies for physical roles). Of course, remote working has changed this to a degree, I spend less time commuting and when we both do WFH we can spend more time together.
I think that is the perspective a lot of people start with. Then, when they have the time, ie in retirement, they can replace that with those deep relationships in quantity of time as well as quality, and perhaps the hobbies or skills they didn't have enough time to spend on when earning their living replace those that work provided.