You can't know for sure. Xbox Live sends no notification and shows no indicator when someone removes you.
If they removed you, it's their loss. Move on.