By not doing things right, you have actually created a tax liability that probably wouldn't have existed otherwise.
For the years you weren't registered with the PRTB you can't deduct mortgage interest as an expense, so most of the rent you got will be taxable. The longer you leave it to sort this out, the greater the penalties and interest will become.
If you try buying your tenant's silence, bear in mind that he can take your cash, and then look for his credits from Revenue at any point in the next four years. He doesn't need your PPS, just your name & address.