This is what some people just don't seem to get, that it's the fear that this will happen, day in and day out, that produces the stress - not usually one comment taken on it's own. It's also the fear of what else this guy or guys might do since there's no way to know until you are away from them (or they start doing something).