You can gift 3000 in a year to anyone without any tax implications for them, so you should be fine on the tax front.
You could open a bank account yourself, with the child as beneficial owner, but it means you're the only person that can transact in the account until the child us about 12. Handy if you wanted to occasionally pop in extra money, and it's nice an simple.
Another idea might be to get them a solid gold coin for the year they are born. I've considered doing this for my nieces / nephews. Something that's enough to eventually be used for wedding bands, but it can readily be converted to cash if they need it.