Warning: Spoilers for The Last Jedi follow.
The Last Jedi appears to have divided hardcore Star Wars fans on a number of dimensions, especially regarding its treatment of Luke Skywalker. In particular, the film gives us a curmudgeonly Luke who has largely rejected the Force and thinks the Jedi should go extinct when he dies. I certainly get why this portrayal has alienated some (even apparently Mark Hamill to some extent), but I’d like to explain a bit here why I think this story arc makes sense.
Luke, as Ari Kohen notes, was a Jedi idealist in the original trilogy. He could see the good in the worst people (including his murderous father) and was eager to bring back the Jedi in the wake of the Empire’s demise. But something has obviously changed in the intervening decades.
The key thing that has changed is Luke’s experiences as a mentor. (Dan Drezner wrote a great piece on the sad state of academic mentorship in the Star Wars universe back in October, and those lessons are all the more important in light of the new film.) One of Luke’s new students, the one with the most promise and raw talent, was his nephew, Ben Solo (who would later dub himself Kylo Ren). From Luke’s description in the new film, he was optimistic and eager to take on his nephew as a padawan.
But then things turned bad. Ben had too much raw power, too little reverence for tradition, and a soft spot for the dark side. Luke (briefly) came to the realization that the responsible thing to do was sneak into Ben’s room and behead him in his sleep. That scene is obviously of great import as we see it three different times during the film.
Yeah, that’s pretty horrible stuff. But my guess is that this is far more common in the history of Jedi mentorship than is generally known. Training a Jedi means first identifying and recruiting a promising child, and then teaching that child lethal powers. It’s possible for the mentor to simply misjudge a child, whose personality isn’t close to completion at the age of recruitment. (Anakin was already considered old for training at the age of nine in The Phantom Menace. To its credit, the Jedi Council decided that Anakin shouldn’t be trained, but Qui-Gon Jinn trained him anyway without facing any obvious professional penalty.)
Most forms of training face some version of this problem. A professor may take on a promising graduate student only to find that the student isn’t really very good and might have ethical issues as an educator or researcher. Such a student can be nudged out of academia or prevented from getting a long-term job in the field. There are things that can be done about bad police officer trainees, law students, apprentice chefs, and others to limit the amount of harm they can do and to drive them toward less destructive employment, even if the mentor fails to detect problematic tendencies prior to recruitment.
It’s different in the Jedi world. By the time problematic tendencies become apparent in a padawan, that student may already have a light saber and many other lethal tools. No Jedi wants to kill their student in their sleep, but on some level, isn’t that preferable to unleashing a limitless murderer into the galaxy?
My guess is that more than a few Jedi have had to kill their padawans in their sleep. It's ugly business, and I’m sure no one wants to talk about it — especially the Jedi. But at least in the Old Republic, it was likely to be tacitly and broadly understood among Jedi teachers.
Luke, lacking traditional training and access to other padawans, relied on the teachings of Obi-Wan and Yoda, who probably didn’t bring up the padawanicide part. Idealistic Luke had to figure this out on his own.
And then he takes on his nephew, someone with Leia’s (and possibly Vader’s) Force skills combined with Han's disregard for authority. That is a terrible combination.
Luke quickly realized what he had to do. This realization — that he’d have to murder a kid and then explain to Leia and Han why he did it — was surely enough to break him and disgust him with the whole enterprise. But then young Ben Solo caught him in the act of considering it and defended himself, which made the situation even worse. From Ben’s perspective, the light side’s greatest hero is the guy who tried to kill him, and his parents are the people who sent him to stay with Uncle Murder. Who would want to go back to that?
Luke’s disillusionment with the Jedi order is understandable from this perspective, as is his self-imposed exile. He has done things for which he is truly ashamed, and which he only did because of his role as a mentor. If the order is to be restarted, it will require some serious vetting of padawans as well as some thorough thought about the norms built into the training process. And there will need to be some cynics in the room.