Xbox Live gaming (especially modern titles) consumes significant upstream/downstream bandwidth and is latency-sensitive. When it competes with other traffic, your whole network feels sluggish.
Run a bufferbloat test (waveform.com) while gaming. Grade A/A+ = good. C/D = enable SQM or wire it.