You are hearing party chat through both TV speakers and headset because the Xbox is outputting audio to both devices instead of just the headset.
If issues persist, contact Xbox Support.