I don't use Cortana either but about a year ago there was a thread wanting to know why the location did not show accurately when Cortana was enabled. So before anything else, if you open Maps and check your location does it show within a couple of hundred feet of your actual location?
You probably know about the settings - Privacy - location page where you can turn on Location and set Cortana to use it. You may already have Maps set to use it. The only option I can see on that page which might be worth trying is to clear the Location history.