Nobody claimed that time off with their children were more important than your time off for your fitness session. You were just the one who accepted the shift because you can move your fitness session. Your flexibility works in your favor, because you’re also the one getting paid for the hours that you decided to work. You can say no to extra work for no reason. You do not have to make up excuses for not working extra. It is not your boss’s job to evaluate which lie is more worthy of not working in their free time. If your boss ask why you can’t cover a shift outside of schedule, just call it what it is: None of his business.