You've got to keep in mind that not all of your reactions are going to be physical. Trust me you are reacting to this -- whether you feel it or not. Do some googling on "silent celiac disease".
This gluten goes from the gut to your brain. It may not be manifesting itself physically; but, trust me it is neurologically.