The answer is yes, but there's more to it than you might think!
Soap in general works to keep your hands clean because one part of the soap molecule is water-binding and one is water-repellent.
Soap helps bind all the dirt, bacteria, and oil together in droplets of water - making them easier to wash off. Absorbing this oil is not always beneficial because it can leave your skin dry and prone to irritation.
And the water-repellent agent can make this situation even worse.
What's interesting about anti-bacterial soap is that, for the active ingredient (usually triclosan) to work, it needs to remain on your hands for around two minutes!
Who takes that long to wash their hands?
Secondly, yes - having soap on your hands for that long will increase the chances of skin dryness and irritation.
So, my advice would be to ditch the anti-bacterial soap all together and simply use hot water (which will kill bacteria anyway if it's hot enough).