I've spent my career studying bad habits. Here's what I've learned about breaking them.

Let's face reality: Change is hard! Look at change rates for people with heart issues who are told by their doctors that they are seriously at risk of dying if they do not change their habits. These patients know exactly what they need to do: eat a more wholesome, lower-calorie diet, exercise more, stop smoking, and so on. Yet even when people's very lives are on the line, only one out of every seven patients is able to make the change.

Another statistic shows that two years after heart surgery, only 10 percent of the people had changed their lifestyles.

Take me as another example: One of my most important goals early in my career was to become a more confident public speaker. I read books, gleaning them for advice. I took a class. And I tried a host of other strategies, including practicing deep breathing, visualizing success, rehearsing my talk, and recording myself talking. While I did get better at looking more relaxed, I continued to feel tense and disconnected from what I was saying, and eager for my talk to be over. I wasn't accomplishing my goal.

Finally, I got tired of feeling so stressed. As a developmental psychologist on the Harvard faculty, I had co-developed and published (with Robert Kegan) a novel and specific model for creating lasting change called the "immunity to change." I decided it was time to take some of my own medicine.

I created a clear picture of what was going on inside me and how I was sabotaging my goal to be a better, more confident public speaker. It was eye-opening for me when I uncovered my unconscious goals: to protect myself from becoming full of myself, to not be seen as arrogant, to not be exposed for the fraud I must be. No wonder the strategies I had been using couldn't stick!

There was still hard work ahead, as I needed to test the validity of those self-narratives I was being run by: Was it true that I was uninteresting? To myself? To others? Did I really believe I was a fraud? While active testing took time and required me to be intentionally mindful, I was able to learn about the accuracy of my assumptions.

Taking the first step of uncovering my immunity to change was a turning point for me because it helped me understand and appreciate why I was so stuck, and it provided a path forward. Whereas I used to have sleepless nights before a talk and experience typical symptoms of anxiety, like dry mouth and short breath, before and during a talk, I now feel fully at ease — connected to myself, to the message I was to express, and to the people in the audience. I notice that I've stopped asking how many people will be at a talk I'm giving.

My personal experience, combined with the research I've done on change, taught me six lessons I wish more people knew about how to make important changes in their lives.

1) Achieving hard-to-accomplish change goals requires seeing our internal, personal landscape

One of my favorite cartoons, by Robert Mankoff, is of a fellow lying down on a psychiatrist's coach; the caption underneath the picture reads, "Look, call it denial if you like, but I think what goes on in my personal life is none of my own damn business."

Of course, looking at what goes on inside ourselves can be uncomfortable, or discouraging, if not downright distressing or painful. All too often, those negative feelings don't lead anywhere productive, and we can feel like we've wasted our time navel gazing. Yes, there can be downsides to looking into our inner landscape.

But there can also be huge upsides. Seeing our inner contradiction helps us clearly see the problem we need to solve, and we can't get to the right solution if we don't truly understand the problem. The problem is not that we can't change our behavior. The problem is not that we are weak-willed. The problem is that we have an unconscious goal to protect ourselves, which is producing the very behaviors guaranteed to keep us from accomplishing our goals.

2) Our standard change methods are flawed

Looking back at my earlier failed attempts at improving my public speaking skills, I see that I fell into using a typical approach to change. Most of us think change works like this: a clear goal + a careful action plan + monitoring of our behavior toward that goal + willpower = successful change. There's one small problem with this model — it doesn't tend to work.

More from First Person

I've spent years trying to get skinny. Oprah's Weight Watchers ads convinced me to stop.

On the face it, the model makes very good sense. It recognizes we need clear goals. And different from a "just do it!" orientation to achieving change, it acknowledges that our inner forces — motivation and willpower — are necessary ingredients to make those behavior changes.

So why doesn't it work? Because it relies on two inaccurate assumptions: 1) that we can succeed by directly changing our problematic behaviors into desired behaviors, and 2) that we can consistently apply our willpower.

Neither of these is correct, since we humans are complex creatures: we have competing commitments and goals. One of my favorite Walt Whitman passages is this: "Do I contradict myself? / Very well then I contradict myself / (I am large, I contain multitudes.)" While we are often unaware of these contradictory goals, that doesn't make them any less real. They exist, and they consistently inform our behavior.

Before we can act in new ways, we have to understand the unconscious goals that make our current behaviors brilliantly effective so that we can correctly formulate our change curriculum.

Here's an example of what I'm talking about: We can sincerely want to lose weight, and at the exact same time we may have a commitment to any of the following opposite goals: to not lose our independence or spontaneity by following a diet; to not want to appear to be vain and care about our looks; to not want to risk being rejected by or excluded from our friendships and family.

Every time we act consistently with our goal to lose weight, using whichever plan we chose to losing weight, we will move toward losing weight. At some point (which will differ for people), we will start to lose weight. As soon as that starts happening, we are now screwing up relative to our competing goal. The more weight I lose, for example, the more vain I feel. (If you are interested in learning more about immunity to change and health, you may be interested in our book Right Weight/Right Mind).

3) Maintaining the status quo takes a lot of energy

I want to use a metaphor of someone driving a car to clarify the power of competing commitments: Imagine a person has a foot on the gas pedal wanting to move toward his or her change goal, and at the same time the person has a foot on the brake. Those two feet are expressing two contradictory wishes: "Yes, move toward my goal; no, don't move toward that goal!" Notice how the foot that is on the brake is working to protect us from the disaster of using the accelerator.

I call this inner contradiction an "immune system," because the mind, like the body, has an immune system — an invisible, ceaseless dynamic that exists for one purpose: to protect us, even to save our lives. An immune system is not an illness, disease, weakness, or problem that needs to be fixed or cured. It is an intelligent, beautiful phenomenon that only wants to take care of us.

However, our immune systems — physical or mental — can still get us in trouble, even when they are working on our behalf. When the immune system is in error, when it sees a danger that is not there, it will go to work to "protect" us from the very awareness we may need in order to thrive. For example, will we really feel vain if we lose weight? And if so, will such outcomes matter more than our desire to take care of ourselves by losing weight?

Notice that no amount of willpower is going to help resolve this internal contraction. Fortunately, misguided immune systems can be overcome. First, though, you need to see the contradiction.

4) We need to recognize the ways we sabotage ourselves

(I don't want to suggest that this is the only method to use when you are stuck. I know the most about this approach since I have spent my career developing, testing, and tweaking it.) Through a series of specific questions, this approach reveals the way we unintentionally work against the behaviors we want to enact consistently, and shows how to take first steps toward creating sustainable change.

If you want to try the immunity-to-change approach, follow these prompts and write your responses down in a four-column grid (see example below for guidance):

Column one: Identify an improvement goal that means a lot to you and meets the guidelines below.

• It is important to you

• It implicates you (your goal isn't some clever way to get other people to shape up)

• It is stated affirmatively (it's something you want to move towards, not just what you want to avoid or eliminate)

Column 2: List all of your behaviors that work against that important goal.

After most people complete column 2, their default model of change kicks in and they start to think up ways to eliminate those behaviors. But the immunity-to-change approach takes a different turn, seeing that list as valuable information to be mined for competing, hidden commitments.

Column 3: Imagine yourself doing the opposite of each of the behaviors you wrote in column two.

What worry or fear comes up? That's your competing commitment. It should explain why you behave as you do in the prior step. You should now see your immune system. This is the "problem" you need to solve, not your behaviors.

Column 4: What assumptions are you making about yourself or about others and the world that explain why you hold your competing goal?

This last step is a deeper one. These unspoken and often unconscious assumptions are the ultimate source of the anxiety shaping any of our obstructive behaviors, and so can be the gateway to substantial, lasting change.

5) Then we need to identify and test the big assumptions that are holding us back

Only when we see our contradictory goals are we in a position to do anything to disrupt them. How do we do that? By putting words to the assumptions we hold that keep us intent to protect ourselves. As is, these assumptions are like a lens through which we see ourselves, others, and the world, and we automatically take them to be the truth. In reality, they are untested assumptions and typically false or overly restrictive.

The best way to create a test is to start by clarifying what data would disprove the accuracy of whichever big assumption we want to explore. Next, imagine the best-case scenario to generate that contradictory data. Then enact that best case and see what actually happens. Did the big assumption prove to be true? Not so true? Not at all true? Most of the time, we need three or four tests in order to genuinely know reality.

Here's my full example:

I chose two related big assumptions to test: "I need to be funny or tell stories in order to be interesting," and "I am not inherently interesting." I figured that if I could show myself I could develop a speaking style that fit me, then I would be able to learn that there were more ways to be interesting than telling stories or being funny.

Before I even began testing these, I intentionally became more attentive to when I felt relaxed and in the flow talking with small numbers of people or in informal settings. Over time, I recognized that I felt most comfortable when I could be in conversation. At first I was puzzled about how I could be conversational while speaking publicly. But with the help of one of my friends, I was able to craft questions that I could ask an audience and be more interactive with them.

I tried out a few of these questions in my first test and found that I was nervous but was able to warm up after several minutes. In my second test, I tried some of the same questions, but this time I framed them so they had "yes or no" responses that people could indicate with a show of hands. My comments took those responses into account, which led me to begin to feel like I was in more of a conversation with people. That felt great! Over time, I developed a set of questions that built on one another and got people talking to each other as well.

In the end, I disproved both of my initial limiting assumptions, and in the process I was able to meet my goal to become a better public speaker.

6) Failing to achieve a specific goal in the past does not predict our ability to achieve that goal in the future.

An important implication of my earlier point that our default model of change is flawed is that there is a fundamental mismatch between our approach and its capacity to enable change. In other words, we set ourselves up to fail before we even start by choosing the wrong tool. And then when it doesn't work, we attribute our lack of success to one of the three inputs: We weren't motivated enough, or we didn't have a good plan, or we didn't have enough willpower.

While some of this may well be accurate, it is unfair to us to take full responsibility for the predictable outcome. In fact, there is good reason to believe there is an internal contradiction that needs to be addressed.

Before we can internalize this promising reframe, however, we need to believe it. In my experience, many people's self-regard has been damaged by blaming themselves for their failure to change. They don't just think they had insufficient resolve or self-discipline, and they beat themselves up for it. In the worse cases, they feed their inner critic, which harps at them with self-defeating commentary such as, "You are bad and flawed."

One of the reasons I chose to write this article is because I want to give people reason to stop being so hard on themselves. We do not need to suffer at our own hands. I learned this lesson from all those people I've professionally worked with who were willing to give change one more try. But I learned it most deeply as I struggled with my own personal change agendas.

Lisa Lahey is co-director of Minds At Work, a coaching and consulting firm serving businesses and institutions around the world, and a faculty member of the Harvard University Graduate School of Education. She is the author of How the Way We Talk Can Change the Way We Work (2001), Immunity to Change (2009), and An Everyone Culture (2016).

First Person is Vox's home for compelling, provocative narrative essays. Do you have a story to share? Read our submission guidelines, and pitch us at firstperson@vox.com.