One of the basic tasks of any philosophy, and especially of any ethical system, is to determine and assign praise and blame.
It follows that we have to ask what is worthy of praise or blame, and what to do when confronted by circumstance — especially blameworthy actions.
One answer that seems especially popular these days is, “Who am I to judge?” After all, no one’s perfect, and right is relative.
It’s certainly true that no one’s perfect. And sometimes right is relative: are bigger or smaller houses inherently better or worse?
But both of those ideas can be pushed too far. Bigger houses probably aren’t inherently worse than smaller ones, for example, but it is objectively wrong to harm communities or habitats. “That house is too big” is a judgment of taste. “That big house uses too many resources and filled in the wetland” is a moral judgment about harm and blame.
So — who are you to judge? Possibly just the right person. Not judging might be letting yourself or the other person off of an uncomfortable but necessary hook. The trick, though, is to assign praise and blame effectively: by separating aesthetic judgments from ethical ones, and by judging actions and effects that really matter.