What’s your NAT sort?
Solely circumstance I can think about is that your router is obstructing connections initialized externally to your house community. Clearly your Xbox can initialize an exterior connection as a result of you’ve got add pace.
I might verify your router, ensure that UPNP is enabled, ensure that the proper port forwarding is enabled, see the hyperlink. https://assist.xbox.com/en-GB/assist/hardware-network/connect-network/network-ports-used-xbox-live
I might additionally strongly advocate utilizing CAT6e over WiFi. WiFi is nice in the event you’re shifting round on a regular basis, however wired is extra secure, dependable, and has greater bandwidth.