Well first thing to do is calculate the loss to the exchequer, if any. If there were substantial setup costs for which there would have been input credit, then there may be little or no VAT lost. Revenue can't impose VAT registration for output (sales) VAT only - if they say the company should have registered from the get go, then it will be entitled to its input VAT also.
If the company deals mainly with VAT registered businesses who would be entitled to full deductibility themselves for any VAT charged by the company, then you're into a justifiable case for arguing there's been no loss to Revenue.