Truth is, when you use only a gold card then your OSC card isnt getting the updates that they send out. When you give your card back they might say (So havent you been watching TV for the last x months then?). If you update your OSC I think there is a flag to say it has been written to outside of the planned updates, and they can find out again. The truth is though what can they do and what can they prove.. You might've had it but not turned it on for months and they cannot prove otherwise. While they can prove you have been paying for certain channels but not using your original card, they also cannot actually prove that a different card was used in it's place, the box doesnt log information like that.
Theres no need to panic but it might be worth leaving your original card in overnight on $ky 1 or channel 26 to get any updates before you hand it back.