I'm sure it will be regarded by some as a controversial decision; however, it's your decision to make.
There is no such thing as "doing it wrong", if you've had access to all the information, and that's the decision you have come to. It's your body, and your choice. You are the one that has to be happy with it - not everybody else!
Others may have had quite different perceptions and priorities, that led them to a different choice. It's not "one size fits all", and never can be in medicine - or life generally.
As you will probably gather, I'm someone who's said no as well.
Then again, my situation was rather different to yours. I certainly hadn't had two relapses in eight months - which is rather a lot. I'd only just about had two in two years (one of which was fairly trivial) which meant I was only a borderline qualifier for DMDs.
I wouldn't have been denied them if I'd wanted - my neuro left it up to me. But when I announced my decision, he said it's what he'd have done too, and didn't try to talk me out of it - which I'd thought he would!
Like yours, he did say he'd be talking to me again if he thought we'd made a wrong decision, but so far, so good. That's all I can say.
It's been over a year now, and I'm as pleased as I can be with my decision, because nothing awful has happened. I accept that, statistically, I probably will have further relapses eventually (unless I've already turned Secondary Progressive, that is), but I've already made up my mind not to beat myself up if/when that happens.
There will be no point looking back with regret, because I'll never be able to prove they wouldn't still have happened if I'd taken DMDs.
It's not as simple or straightforward as thinking: "Yep, I definitely caused it!", because we can just never know.
So, I just take each day as it comes, hope for the best, hope I'll deal with the worst if it happens.
Of course, the more time passes that I still don't have a relapse, the happier I am that I made the right call on the DMDs.