So, you take your own food, and say you don't want to seem rude or ungrateful, but you really enjoy their company, and so that you can stay a bit longer to chat, you brought your own so you're not distracted by pains in your gut.
When I go to my sister's house, I try not to touch anything. At all! I'm even scared sometimes to have a cup of tea!
Maybe eventually they'll get it.