At what point do the Libyan rebels cease to be civilians and become the aggressors against loyal civilians?

It would appear that not all Libyans want Gaddafi out. The media say these Gaddafi loyal civilians are acting under duress but are they?