He also wears a colored cloth called a stole over his shoulders. It's a historical symbol indicating that God has placed a yoke of humble service upon his shoulders, and the color of it depends on the season of the church year.
These symbols can be a useful attempt at communicating the mystery that sinfulness is covered by the righteousness of Christ. Most people, perhaps, don't realize the pastor's outfit is symbolic. They might think he wears a robe because that's just how it's always been.
Some think the use of these vestments helps draw attention away from the pastor as an individual and instead puts the focus on his unique role in a worship service. Others think the pastor should wear something less strange-looking so as not to intimidate outsiders who may come in to visit. Still others adamantly protest any idea of doing away with the vestments because they think tradition is extremely important.
My opinion is the meaning behind the garments is beautiful and worth remembering. At the same time, it's only symbols and clothing. Our hearts are more important than what we wear.
People can nitpick and fuss all they want over what the pastor wears, but I'm just gonna be over here chillin' out.