Is it an ethnic slur to say you call a spade a spade?

During a recent conversation off of ALIPAC, I said someone was a jerk. The jerk fired back. Anyway, my final response to him was that I "call a spade, a spade" and if he didn't like my opinions, he should stop asking for them.

Someone else who wasn't even a part of the convo said he was a black and was offended at my statement of calling a spade a spade.

I sat there scratching my head trying to figure out what could have offended him about that statement, but he swears it's racist ?????

Anyone have any insight? I certainly didn't mean it to be racist nor did I know it even HAD a racist connotation, if it really does.