If the supplier is not genuinely supplying goods, then you have a pretty solid case. Call/write to your credit card supplier and explain what's happened. They've probably already got a case opened on the supplier in question. If you can, get copies of any adverts/websites (print out the page in question) that led you to the purchase.
You do not need to cancel your credit card - let the credit card company make that decision. They may advise you to shut down the card, and issue you with a new one, or they may not.
I've always found credit card companies very helpful when the supplier is obviously not behaving. The sooner you act the better.