I disagree, at least as far as my own system is concerned. I've had several reactions that were tied back to caramel coloring.
Same for me. I avoid all sugars/starches and all their derivates unless their origin is clearly stated on the label. "Better safe than sorry" is my motto when it comes to eating processed food. For the same reasons I avoid vinegars - I don't believe white vinegar is any safer than grain vodka or whisky (both make me sick).
But as for the original question, I wouldn't worry about caramel colour if it's labelled gluten-free.