I do not condone violence and I do my best to avoid it whenever possible; however, I have been in many fights and I can think of only one that I believe was not justified.
I was raised with two younger sisters and my father has always told me that it was my job to protect them and to stop anyone who might threaten them. My parents said this of not just my sisters but of my family and as I grew and this became one of my characteristics I placed my protection on my friends as well. This was very common in my family among my relatives and my cousins and I always defended each other. This does not mean we are like a gang going around beating people up but rather the opposite. We feel safe around each other and that gives us a greater sense of freedom because we never fear violence.
When an older bully at my school would not stop harassing me and school authorities did little to stop the abuse my father taught me how to fight properly so that I can be more effective at stopping some one quickly and efficiently.
I have never gone into violence easily. In over 34 years I can still count with the fingers on my hands how many times I have been in a fight that was not for sport. I can almost always win a fight with my brains and I have learned to use it effectively in that way. But there are those who would resort to violence because they lack the intelligence to defend themselves without it. I refuse to be a punching bag for those people and I will fight back and hit them hard and effectively.
I am very similar to you in your approach to violence, and when it is justified.
Background: I'm short - 5'5". I've always been the shortest in HS, etc. I usually fended off fights in two ways: humor, and having very big friends. In HS I got into 3 fights - I won one (against a bully); I tied one - a guy that didn't like me, and I didn't like him - we fought behind the school afterschool, beat the hell out of each other until we were both exhausted, and eventually became good friends; and I lost one (I picked a fight, and shouldn't have - I got my ass handed to me, and rightfully so - I apologized later).
Later on I learned to fight properly - in the service, from martial arts friends, etc. Once I had a family my two sons, and two of my three daughters and I studied martial arts for years - and now both sons, a daughter, and I are all black belts (I'm also an instructor). One of the core precepts in virtually all martial arts is that fighting is a LAST option - and only to protect yourself or others. Additionally you should never enter into combat in anger; anger makes you impulsive, and that leads to mistakes (which are usually bad).
I have also taught my sons (and black belt daughter) to only use your skills in defense - of yourself or others - and only to the point of subdual, NOT unnecessary injury.
Both of my sons have had encounters where they protected someone else, not themselves.
My eldest son, Rocky Jr, protected a mentally handicapped boy in his school who was getting picked on and hit. Rocky stepped in, asked him to stop, the guy swung at him and Rocky took him to the ground with a hip throw and arm bar. A teacher came over, took over the situation, and eventually called me - not to condemn Rocky, but to say thanks for raising such an admirable young man. She had seen the entire situation, realized that Rocky didn't start it and was protecting the other boy, and did not injure the bully. She was amazed - and Rocky was in 5th grade when this happened.
My youngest son, Tommy, found himself in a similar situation recently. His best friend is a smaller boy. They were in our neighborhood, walking to the lake to fish, when an older teen stopped them and tried to get the friend to give him his fishing gear. He pushed the boy, and Tommy blocked it and got in front of him. The bigger kid threw one or two more punches, Tommy block both, and then unfortunately hit the kid with his hand in the wrong position - and broke it. He knocked the bully out - one punch. Once again the other boy, and his mom, came to Tommy's defense when the bully's mom tried to cry foul about the situation. Once it was explained to her, she backed off.
So, I do feel that violence is necessary, sometimes; but almost always only in self defense of yourself or others.
Oh, there's one more corollary to this stance - never start a fight, but always end it. ;)
Quick, humorous martial arts story... We were at an international conference in New Orleans. 5 instructors, including 2 masters (5-7th degree) and a grand master (8-9th degree) were a part of the 5 walking. They were approached by 3 individuals, wielding knives. The instructors politely and quietly said that they were not going to give up their wallets, and asked them to let them pass. They attacked, and were immediately disarmed and detained, while the police were called. The only ones engaged in the defense were the masters/grandmaster (the three assailants attacked them first because they were older - all in their 50s-late60s).
Should they have handed over their cash and valuables? Tough call - and I think the call would be dependent on the situation. How many assailants? What weaponry? What mindset (e.g. pissed, in control, sad, in a rage, etc.)? Can you avoid the situation without harm, definitely? All of this weighs on you when you are considering a response, and you must make these assessments very quickly. In the case of this particular attack, the assailants were outnumbered, scared and antsy, were not wielding a firearm (although sometimes even that can be diffused), and were young. So, it was a safe bet to stop them before they hurt someone else, or are hurt themselves.
In that situation, with that group, I would have made the same decision.