Op, this is just me thinking out of the box and I could be completely wrong- but seen as you are paying the mortgage, could you move back in, apply for PP to build a self contained cabin type building in the garden (assuming you have a garden and space), move into that, then you can see the kids...