In the workplace, diversity has become a highly personal and emotionally charged topic. Having a real, honest conversation about it requires us all to dig deep and think about not only how we see ourselves, but also how we view others. And in some instances, these uneasy discussions have inadvertently alienated certain demographics within the workplace — namely White men.

It’s a fear that diversity and inclusion professionals have long suspected privately — that too forceful a push towards diversity and inclusion would leave those in the majority feeling left out and overlooked. But a recent survey from Ernst and Young and ORC International confirms it — more than one-third of respondents felt that the increased focus on workforce diversity overlooks White men. This unexpected outcome breeds a quiet resentment as people begin to fear that they’re being passed over for promotions or other opportunities for advancement.

So how can you build a diverse workforce without alienating anyone? Here are four ways to prevent workplace diversity backlash.

1. Lead inclusively.
This may be a tough task for leaders who were first introduced to the workforce decades ago, but it can be done. Going beyond the business case for diversity, leading inclusively means being mindful of the tone that’s set from the highest levels of your organization. In every area of your company — from hiring to the weekly staff meeting — the goal should always be to make sure that everyone is treated with respect, and feels a natural sense of belonging.

2. Give everyone a voice.
In the diversity conversation, the trend has leaned towards giving a larger share of voice to under-represented employee groups. But because the goal is to allow everyone a seat at the table, all employees should have a say in how the organization is shaped, including those who are members of traditionally dominant demographics. Deloitte US took a radical step in this area in 2017 when it dissolved all of its employee affinity groups — internal organizations designed to support specific minorities.

3. Listen up.
As you launch new diversity initiatives, keep your ear to the ground. Seek out employee feedback regularly in order to better detect signs of backlash, and adjust accordingly.

4. Be courageous.
Creating a more diverse and inclusive work environment is a goal worth pursuing. It will be tough at times. But small, measured steps in the right direction will ensure that everyone benefits, and you’ll actually create and sustain the long-lasting change you’re after.