Yes it's possible to be allergic to weed. It's a plant after all and it's pollen or even the leaves/buds when touched or smoked could cause some kind of weird reaction. But it depends on the person probably. Thousands, perhaps hundreds of thousands, of people seem to interact with it and not have any problems.