Tesla Autopilot Innovation Comes From Team Of ~300 Jedi Engineers — Interview With Elon Musk

Sign up for daily news updates from CleanTechnica on email. Or follow us on Google News!

---

“It is extremely important to emphasize that Tesla Autopilot is the work of 300 super talented engineers,” Elon Musk tells CleanTechnica

How many Tesla engineers does it take to make your car drive itself? This is a question many a Tesla owner has surely wondered from time to time. Like Tesla vehicles and software overall, Autopilot’s advanced driver-assist and self-driving features are deceivingly smooth and simple on the user end. They seem to flow out of a Tesla like The Force flows out of a Jedi’s mind. We know, however, that an incomprehensible amount of work has gone into fine-tuning Tesla Autopilot hardware and software.

Kyle Field recently published an article about the kind of “pseudo-lidar approach” Tesla has taken to advance its computer vision and self-driving capabilities. As a followup to that, I had a short chat with Tesla CEO Elon Musk about the breadth and diversity of Tesla’s Autopilot team. Don’t get your hopes up too high and then crucify me for not delivering, though — I didn’t glean any industry secrets or even any real detail at all about what goes on behind the computer screens of Autopilot engineers. (Sorry — I tried.) However, I think that what I did find out is still a big deal, and Elon definitely thinks so as well. What I got was a little bit of perspective on how large and varied the Autopilot team is.

Let’s rewind for a brief moment first, though. A little more than a year ago, when Kyle, Chanan Bos, and I were visiting Tesla’s main automotive factory and seat factory in Fremont, California (oh, how the days of free-flowing factory tours tease us now), they and the Tesla PR rep we were touring the facilities with had to relieve themselves in the restroom after the seat factory tour. I was thirsty and headed to the water tank that we had been introduced to in the morning. As I was approaching it, Elon and a few other people came out of some doors right next to it. Since we were passing shoulder to shoulder, I quickly said hello. I didn’t expect anything more than a “hello” and perhaps a smile, but (after taking a moment to recognize who I was) he mentioned to me that he had to get into an Autopilot meeting that he was a little late for but quickly brought me to his assistant to line up a brief chat later in the day (which, unfortunately, couldn’t be arranged in the end). Looking into the meeting room, if my memory serves me, there were about 10 or a dozen people waiting or getting settled. On a quarterly Tesla conference call with the investment world later in the year, Elon mentioned that he meets with the heads of the Autopilot team once a week. The combination of those two moments struck me — this is a massive team with several directors in charge of different parts of the Autopilot system, and Elon is quite closely involved with every aspect of Autopilot development.

In two conversations with Elon that I had more recently, he indicated that there are actually around 300 engineers working on the Autopilot team — “just under 200 on the software side and a little over 100 on the chip design side.” So, there are several different directors in the Autopilot program that oversee hundreds of other Autopilot staff.

Even that is perhaps not the best way to explain the breadth and depth of the team, though. As we chatted a bit about the software team, it became clear that these ~200 software engineers could probably get a job anywhere they wanted. “No one is really their ‘boss,’” Elon said.

I think that conveys a portion of the Autopilot story better than anything I’ve heard or read before. The challenge is so large that you need nearly 200 of these high-level Jedi engineers and another 100+ top-notch hardware engineers each doing their part in order to get a supercomputer to move my Model 3’s internal organs and wheels in just the right way to produce a smooth, safe, elegant driving experience almost everywhere. Eventually, the goal is for it to move in a smooth, safe, elegant way everywhere that a car can drive. No small mission.

Getting a computer to drive a car is easy. Getting a computer to drive a car on roads shared with humans is quite difficult. Getting a computer to drive a car on all kinds of roads all over the world with a bazillion different rules, signs, and cultural norms is not quite the same as programming Tetris. As Elon put it, “Self-driving is not a fundamental AI problem, but it is hard to squeeze it into ~100 truly useful teraops.” Wiktionary tells me that teraops (TOPs) are “10¹² operations per second” in the computing world. That’s a lot of operations. In a single second.

How does a computer go about learning how to perform ~100 TOPs in a large, fast, pretty vehicle at a level of caution that beats a good human driver’s ability to protect that pretty car and its passengers from wallet-crushing and perhaps even life-crushing destruction? That would probably be a good course at MIT, but the extent to which I uncovered the answer was: it takes ~300 really freakin’ good engineers. Elon didn’t call them “Jedi engineers,” but I’m labeling them as such from here on out. (Lockdown got to me and I went through all of the Star Wars movies again.)

In our discussions, Elon emphasized at least a few times that credit for how amazing Autopilot is and will become goes to hundreds of extremely talented people. The followup request on the quote at the top is notable: “It is extremely important to emphasize that Tesla Autopilot is the work of 300 super talented engineers. Please quote me on that.” Toward the end of our chat, Elon emphasized even more strongly where he thought credit for Tesla’s Autopilot brilliance should go:

“Please put in bold letters my quote that what people experience in the cars is the result of a large number of extremely talented engineers working very hard. Please give me the least credit.”

That is what I was already planning to emphasize in this article, but honoring the request — as well as the engineers behind the curtain, of course — it hit me at the last minute to put that quote (in bold) at the top of this piece. Even this highlight, though, doesn’t capture the full scale of the effort. There’s more.

“We also have over 500 highly skilled labelers,” Elon added. “This is a hard job that really does require skill and training, especially with 4D (3D plus time series) labeling.”

Wow, 500 labelers on top of 300 Jedi engineers!

That’s still not the end of it.

“We are expanding to 1000 highly skilled labelers. Emphasis on highly skilled. Like I said, this sounds easy, but is actually hard and requires talent.“

This is a big team.

And, look, I understand that Elon wants to be both humble and honest and give credit to the highly talented people who make Autopilot a reality and constantly improve it — but there are a lot of people who don’t understand his role in the process either. He is not an actor pretending to be an engineer and asking for people to just create something awesome. He has numerous directors reporting directly to him on a regular basis, including several that are completely focused on Autopilot.

Hardware engineering leads who report directly to Elon include: “Drew Baglino for Battery & Powertrain, Lars Moravy for vehicle structural engineering, Pete Bannon for chip/computer & low voltage systems.”

Software engineering leads who report directly to Elon include: “David Lau for vehicle software, Nagesh Saldi for TeslaOS & Internal Apps, Ashok Elluswamy for Autopilot team overall, Andrej Karpathy for AI, CJ Moore for software integration & verification, and Milan Kovac for Autopilot foundational code (software layer between FSD hardware and everything else).”

One may ask, “Well, how much is Elon involved, really? Does he even know how to converse with these people on the deep tech of it all?” Everyone at that level who I’ve ever seen talk about this has gone far beyond a “yes” in answering such questions. Jim Keller gave humongous praise to Elon in an interview with Lex Fridman and I think elsewhere that I can’t remember now. George Hotz also gave him big props (which I don’t think Hotz does easily) in an interview with Fridman.

While this doesn’t relate to Tesla, I think this statement from Kevin Watson, who spent 24 years at NASA before working at SpaceX, is one of the most impressive testimonials I have seen on how thoroughly and usefully Elon gets involved in the engineering side of his companies:

You see this? pic.twitter.com/vRJID8p2nH

— Quantum Tarantino 🏐 (@OutAmongStars) August 2, 2020

Though, none of that takes away from what Elon thought was the most important takeaway from our conversations. Just because he is a highly experienced and brilliant engineer doesn’t mean there aren’t a few hundred other highly experienced and brilliant engineers in the company that make Tesla Autopilot what it is, and what it will become. To the contrary, most of the credit is due to them.

There’s one more thing to consider, though. You’re not going to make much progress if you’ve got 150 wild, wild west mavericks each trying to be Luke Electronwalker or Han Autohero. Not all of us have been software engineers, but we’ve all played team sports at some point. Just in a game of basketball, you’ve got to get a good mixture going between different players’ abilities in order to play a good game. That means some players need to swallow their pride and step back a bit and others need to step up in ways that don’t always make them comfortable. That’s just 5 people throwing a ball around and trying to get it in a hoop*. It’s surely a special challenge getting 100+ Jedi chip engineers, nearly 200 Jedi software engineers, and ~500 labelers to do their version of that in order to get a computer to drive a car. (*Okay, I’ve played a lot of basketball and I know that minimizing the skill of good basketball players and using the term “just” is not the perfect comparison. The day that 5 robots can play basketball as well as the Miami Heat will be a ridiculous day in the evolution of AI — and we’ll probably be lucky if we aren’t their slaves by then!)

A different metaphor we could use if we wanted to toy with the self-esteem and pride of these Tesla Autopilot engineers and Jedi labelers while trying to make this additional point about the magic that is happening at Tesla is we could consider them ~800 gears in a big machine. They each have to toil in their own way to fulfill their role, in order to design and build a software suite that tells my car, “Moonhopper,” how to act like the son of 800 Jedi.

As Elon summarized it at the end of our first conversation, “Lot of software grinding.”

Side note: There’s a complementary topic perhaps as fascinating as this one that we got into as well. We didn’t discuss it nearly as much, as it came at the end of the second chat, but stay tuned for a piece on that if you want to get a little more insight into what Elon Musk considers to be a rather big deal.

Related: Tesla Autostopping — 7 Notes On New Autopilot Feature

Want to buy a Tesla Model 3, Model Y, Model S, or Model X? Feel free to use my referral code to get some free Supercharging miles with your purchase: https://ts.la/zachary63404. You can also get a $100 discount on Tesla solar with that code. No pressure.