Targeting different sources of mental fatigue can help make your time away from work be more restorative.
You have to drive your thinking rather than allowing it to be buffeted by the information available in the environment. This ability to stay focused on tasks that you want to complete is called executive control. Maintaining this focus is difficult, and you can clearly get tired of doing it.
When you get away from work and feel this kind of exhaustion, you might want to recharge by engaging in activities in which you allow the world to drive your thinking rather than having to be internally driven in what you do.