Exhausted Immunity

Why is it when I work out too aggressively and don’t eat until many hours later I get sick? Some times it seems like allergies.

Does the workout weaken my immune system?