Imagine the satiation of some desire that is considered by one to be wrong (perhaps through tradition), and yet difficult for another to abstain from, perhaps precisely because not obiviously hurting others. Those values in relationship, produce strife both when the value is lived up to, doubly when the value is not lived up to, but not at all when the value is abandoned... if "no one gets hurt." Of course, if someone is sometimes hurt, or always hurt, or many people always hurt, the suffering might impress itself, in proportion of its visibility to the value-holder, upon the value and its maintenance.
So there is a pull and push, a dialectic, that amounts to a pressure toward some morality, for any given form of life, that is in equilibrium between the minimization of the physical suffering and the minimization of socio-psycho suffering.