each 100W headlamp will draw 10A max [assuming battery voltage of 10V in the worst case].
2 such bulbs will draw 20A max. a 30A relay should be more than sufficient. I think something else is wrong in your case. maybe a faulty relay.
did you upgrade to ceramic holders too?