Volume control issues usually stem from HDMI-CEC (Consumer Electronics Control) misconfiguration or audio format mismatch.
Settings → General → Volume & Audio Output:
If CEC remains unreliable, use the TV remote for volume and set Xbox to "Fixed" volume output.