The older we get, the more ability we lose. Therefore, everything has a specific time to realize for us. It is not impossible, but would be so difficult to turn back and start something from scratch.
For me, changing department even in the same company would be somewhat a challenge. New work, new co-workers etc...
Perhaps you would surprise yourself with the change, and once made, would welcome it :)
But yes, everything has a time to live, and an end to die.
One of my friends did it and regret :)