Jenn, I'm not a doctor so I can't comment on rather or not you are 'crazy'.
I can only share my experience and what little I've learned about CLIPPERS. CLIPPERS is characterized by a remission and relapse cycle. The only treatment I've heard of is with steroids or a few other autoimmune suppressors. In my experience I have had two relapses. The first at the very beginning of treatment. I was 1 week late in my treatment cycle and I was really thrown for a loop. Mary, my wife, feels that many of my ongoing problems resulted from this relapse. The second occurred recently when after over two years on prednisone I was tapering from 13 mgs per day to 12 mgs per day. A return of old problems, stumbling, wonky vision, extreme tiredness, increased scanning which lead to a finding of new lesions in the pons via MRI.
I'd be careful about not taking you physician's advice. If you have CLIPPERS. Another puzzling thing about CLIPPERS is the wide variety of symptoms and outcomes. Think of the future, be careful and good luck.