When did "extremist" (even without a further descriptor) come to have a negative connotation?

Doesn't it matter what one is "extreme" about? What about people who are "extremely honest" "extremely trustworthy" or "extremely benevolent"? Are they extremists? What about people who are "extremely moderate"? Are they extremists?