Accepting an Xbox Live invite is simple. Methods vary slightly by dashboard version (Xbox 360 / Xbox One / Series X|S / Game Bar on PC), but the principle is the same.
Quickest: Keep a controller nearby; press Guide → Parties → Invites → A. Two seconds.