Why do people call others racist because they didn't vote for Obama?

I know that there are racist people in the world however why call 'racist' just because someone didn't vote for him. There are people who are know of his policies and ideals and whom are truly educated who decided not to vote for him, yet there are some who only saw skin color and voted only because of that. Not knowing anything about his policies or ideals. Voting only because of someones 'skin color'/race is idiocity in itself, however calling 'racist' is just as bad. Whats worse is that when you've known a african american friend for over 12 years of your life, but then they call you racist for not voting for Obama. Some people shouldn't be allowed to vote.