Physical toiling just means physical toiling, nothing more, nothing less. A character can go to a different planet and still, they will have not experienced anything if they’re exactly the same person.
Meanwhile, it’s possible for a person to “be in the same place” and yet find themselves in an entirely different place by being a different person.