First thing you need to do, is forget about the loan. It's irrelevant in the context of CGT.
Secondly, you need to understand that while there is a total exemption from CGT on disposal of a PPR, it applies only to the "current use value" of the property i.e. its value for its current use as a residence (or in the case of part of the garden, its use as a garden).
Thirdly, you need to understand that for CGT purposes, transactions between connected persons (such as family members), are taxed based on the market value of the property involved, regardless of the actual amount of money changing hands.
Finally, the quick answer to your question is that yes, there's likely to be a CGT liability.