A character in Dostoyevsky's, Brothers Karamazov said "Imagine that you are creating a fabric of human destiny with the object of making men happy in the end, giving them peace and rest at last, but that it was essential and inevitable to torture to death only one tiny creature - that baby beating its breast with its fist, for instance - and to found that edifice on its unavenged tears, would you consent to be the architect on those conditions? Tell me, and tell the truth?"
Would you would torture a baby to bring peace and happiness to humanity?
Looking at the responses we can simplify the question a bit to get at the heart of the problem, a more general question is:
"Does every person have the right to refrain from performing Act A, if doing so results in others experiencing Act B?"
Does the answer depend:
1) Solely on Act A.
2) Solely on Act B.
3) The relative unpleasantness of Act A to Act B
4) A more complicated relationship between the two acts.
I think the result will be a very, very large grey area.