It depends upon the person.
For a lot of people, its like buddy, kiddo, buckaroo, comrade, etc, can just be informal terms for friends...and not have derogatory meanings.
If the body language tone and context doesn't seem to be patronizing, etc...its probably just that sort of usage.
Otherwise, you need to use judgement in interpreting what can easily be a simple more affectionate than insulting referral.
Does he call you this in the presence of others?
A housemate of mine from "up North", a male my own age, used to call me kid. It was just politely affectionate. He said it to a lot of people.
If they're older than you, I understand. Otherwise it seems odd.
Typically, kiddo is said in the same context you experienced it...parent/child.
In general it implies a differential in power. It can be playful or it can be condescending. Ya have to pay close attention to discern the speaker's intent.
I'm 58 and you could be my daughter. I've called you kiddo on more than one occasion because I've watched too many Humphrey Bogart movies and he always seemed to put the opposite sex at ease by calling them kid. Right, "Here's looking at you kid"... I sound a little like him too with my brooklyn accent..LOL...
If it bothers you I'll stop calling you kiddo or kid... It will be hard, cause you may be an adult in your son's eyes but to my 58 year old butt you're a kid...