A nice problem to have
First thing you should consider is the level of risk in your investments. It may be a conservatively managed portfolio but even the most conservative of stocks carry a level of risk. 10 years ago bank shares and shares like Tesco's were considered low risk by many investors but look where they are now. 4-5% return suggests it is not gathering interest in a savings account.
Second thing to consider is the tax implications of whatever decision you make, for example, what is the CGT impact of cashing in your investment.
Thirdly you mention "we" in your post, I presume that means 2 of you. You need to think the implications of death, illness or separation became a factor in your lives. Could something happen that would result in your lump sum being used for other events?
Is a half and half approach a consideration, use half of your savings to reduce the mortgage but still retain a significant capital lump sum for a rainy day