Does being 'too clean' lower our body's defence systems?

There may be an element of truth in that question.  There was a time when I hardly fell sick. No colds, fevers or any other aches or pains.  In fact, I don't remember visits with a doctor in my school days except for maybe one or two. I could drink water straight out of the tap and eat anything from the busy roadside food stalls. 

I can do none of these things now.  I need to take shots to travel, flu shots every season as my doctor recommends them, constantly wash my hands, drink only bottled water in a foreign land and can never risk eating just anywhere.  As a matter of fact, I avoid using public restrooms if I can help it because I wonder about the type of bacterias that breed there. 

I know one reason is paranoia and the other reason is my weak immune system.  Don't know if one feeds into the other.  Whether it is paranoia that has weakened the system, or if it is the not-so-strong an immune system that scares me from doing anything normal.  On the other hand, by being too careful I could have possibly lowered by body's own capacity to fight off the usual bacterias.

Not sure what is right anymore!