As mentioned above, reducing the rate on the CU loan (by including it in the mortgage) will only make financial benefits if it isn't added to the entire term of the mortgage.
Depending on the structure of your mortgage (if fixed rate fees may apply for accelerated payments - if tracker/variable it should be ok) and your own ability to make increased payments to reduce the 20k over a shorter term (e.g. the 3 years remaining on the current CU loan) you should end up saving.
You can crunch the numbers on one of the many mortgage calcs out there (Jeacle calc one of the most popular on AAM).