Your neurologist sounds like a bit of an idiot. (IMO)
I think of energy as a sort of banking system. When you use it all up you run up an overdraft, which has to be paid back. So if you are already starting on very low funds (ie, you have MS or CIS fatigue), you simply have less energy to use up. And once it’s gone, the only way of replenishing it is rest.
So while exercise can increase the amount of energy you have, you need to balance it. So swim, but for a shorter time. And if your job is very physical, don’t attempt to go at it full tilt, then do some extra exercise on top. Start practising Fatigue Management, this means breaking physical (and mental) tasks up and doing things in small chunks, rather than all in one go.
And don’t push yourself further than is sensible. If you are getting sick and dizzy feelings (maybe you have a touch of vertigo?) then it seems that’s too much.
Happy Christmas to you too Geeman.