Software Governance and Automobiles - Session 5
Q&A and Panel Discussion on Autonomous Vehicles, with Leilani H. Gilpin, Mike Milinkovich, Eben Moglen
EBEN MOGLEN: Here it is. It’s your chance to ask about self-driving cars before we drive them all off the road. Go. Not a thing? Yes.
AUDIENCE MEMBER: : So you talked about Toolchain. First thing, do you know the Topcase Project that was doing something similar for avionics?
MIKE MILINKOVICH: I know lots about Topcase, yes. So Topcase is basically now called Eclipse Papyrus, and we have a group at Eclipse called PolarSys, which is building toolchains for very large, complex, embedded systems. So, yes, I know a fair bit about Topcase. So, question?
AUDIENCE MEMBER: : Because in Topcase, the toolchain prototype worked nicely, but as soon as it came to qualifications for toolchains, nobody had a clue how to do it. So the question is, you’re putting together functionality again. Do you have a strategy? How are you going to then qualify this as code emitting tools? That means you’re going to be at the highest integrity level which is T-3.
MILINKOVICH: So the short answer is no, not yet. Longer answer is a lot of the tools– not a lot– many of the tools that are targeted in the various points along the way, do have various levels of qualification. So, but as an overall system, like a system-level qualification, no, not yet, and that’s an obvious weakness, right? I mean, if you’re going to build a safety, functional safety systems, you need to be able to qualify the tools. That’s a prerequisite, but as a guy who’s been trying to qualify Linux for over a decade, I think you have an understanding of how hard this is.
AUDIENCE MEMBER: : My question is regarding data access, and you brought it up post the accident that it’s not available or it’s proprietary as far as Uber is concerned. Given that autonomous vehicles are slowly going to become ubiquitous as far as society is concerned, how do we access the data post an accident given the obvious liability implications for the company in question?
LEILANI H. GILPIN: Yeah, that’s a great question. So I should start by saying that I’m not a lawyer, but you know, I think in the case of an accident that data needs to be made available to at least the people who are involved in that accident and at least needs to be made available for people who work on explanatory artificial intelligence, who can basically tell stories, like we’ve been working on, of what happened to compare that in the case of some sort of proceeding. I’m not exactly sure about making that data completely public yet, because I can understand with good reason why they’re not making this accident data public, but not, you know, especially in the case of the Uber accident, not disclosing anything about the different sensor hits, if they were detected in enough time, I think not enough people are asking that question.
MILINKOVICH: Everything she said, but I’ll just add that, I mean, we already have some precedents both in the automotive industry and elsewhere where accident data is made available at a certain rate and pace depending on how the investigation goes. So the black box from the cockpit of an airplane, for example. Eventually, if they can find it, eventually that data is going to be made public, and that’s a known and accepted thing. You know automobiles today actually have sort of like a little black box that’s recording basically the last two minutes, five minutes, depends on the model, I guess. But yes, and oftentimes that ends up as evidence in accident trials. So that seems to be generally accepted.
I think what you’re bumping into with the Uber thing right now is that though cars that are on the road right now are owned by Uber or Waymo or these companies. So they’re asserting, you know, company confidential, company proprietary, to basically lessen their potential liability in a court case. So, I think this is eventually going to have to be something that gets fixed by law or regulation where there’s going to have to be limits on hiding that information. I don’t think companies like Uber or Google are ever going to give up that data just because. So we’re going to have to fight for it through our legislators.
MOGLEN: It’s not that difficult for the entities with subpoena power to get what companies have. The National Highway Traffic Safety Administration has power to pull data, which it mostly doesn’t need because it gets cooperation in part because it does so on a faultless basis that assumes that that data is not going to be used as evidence in liability proceedings.
But you might want to broaden your account of this to ask how much data we do not have about all kinds of operation? Leilani used to work at CAIDA, which is the Internet traffic assessment activity at the University of California San Diego run by my friend K.C. Claffy. If you want to understand how we get Internet traffic data to try and understand how the internet works, I have the same bad news for you with respect to the automobiles. There are two sets of people who have the information necessary to understand how the internet really works from a traffic management point of view.
One bunch of them are telecommunications companies who would never give that data to anybody, because it would go to their competitors, and the other one is the NSA that won’t give it to you because they’d have to kill you.
All of the data that we have about how the net works is scraps that fell off those two tables, that the telecommunications operators considered it in their temporary interest useful to give to researchers or that has come through unusual and unrepeatable pathways out of the intelligence services.
The same is going to be true for roadway data. You’re thinking primarily about accident data, which is the easiest data to get because something has gone wrong and people with subpoena power want to know. What about all the ordinary data? The ordinary data, the data which you would think of as the baseline for all the judgments about what is reasonable or not reasonable, and what really happens on roads is going to be subject to ownership rules and control rules that we haven’t talked about today because we were only thinking about software governance in cars.
But when we talk about all that connected transport and all the data being created by that connected transport and all the different public policy and economic market interests that are valuably now concerned with shaping that data and the rights about that data to their own needs, you can multiply by many orders of magnitude, the statements that were motivating you to ask about accident.
This is part of why, at least from my point of view, the question of the freedom of the software is so important. If we are really going to wind up with a world in which the companies Mike works with have a beautiful open source toolchain for making driving assistance and then autonomous driving software all for themselves, you can be sure that what they will do is build proprietary software that controls the data. They will build the software as proprietary software precisely in order to control the data, because the transport data which contains location information about identifiable human beings and is therefore the most valuable data in the market right now, will also contain an awful lot of information about energy usage and patterns of movement and personal habits.
Imagine just for a moment if you read The New York Times story yesterday in which the poor technology writer goes and asks Facebook for the information about himself, and his headline is “Yikes!”. Imagine what happens when in addition to all the social graph data Facebook has been keeping on him, including the contact information of all his exes, it also contains all the information coming out of his car.
Everywhere he goes, everything he does, every little corner that he cuts, every too fast this, every was at a bar and then drove home. Remember that all of that is going to fall into the possession of somebody. Understand why it is that if the user isn’t allowed to modify the information, to modify the software in the car, that information will go where somebody else wants it to go. Now we’re not just talking about accidents anymore. Now we’re talking about ordinary life.
MILINKOVICH: There are already insurance companies today that want to put a dongle into your OTB slot.
GILPIN: Yes, they already have.
MILINKOVICH: These things this already exist, so if you’re willing to share 100 percent of your driving behavior with an insurance company, they will lower your rates.
AUDIENCE MEMBER: I think most of the security breaches we’ve seen in cars have come from the dongles.
GILPIN: Exactly.
MILINKOVICH: Yes. So the thing about that, so what they’re doing, Leilani was mentioning that the CAN bus is inherently insecure. It was never designed for security because security has always assumed to be air gapped, and now we’re in the OBD thing. The ODB thing, you know, plugs directly into the CAN bus, and it’s a Wi-Fi or LTE-enabled device. What could possibly go wrong? And by the way, if you know what you’re doing on a CAN bus, you can accelerate, you can brake, you could turn the wheel. I mean, basically, you can control the car by sending instructions on the CAN bus. Again, I mean what could go wrong?
AUDIENCE MEMBER: There’s actually a race for autonomous vehicles that reverse-engineer the CAN bus so they can be entirely driven by wire.
MOGLEN: Sure. Nicholas.
AUDIENCE MEMBER: : It just means it’s time for creating a Raspberry Pi that he can plug the Dongle into the Raspberry Pi, and it can simulate anything. So that you provide fake data to your insurance, so that you can get a lower rate without giving away data.
MILINKOVICH: So I’ve been, I have been in a car in the parking lot at Bosch in Bangalore being driven by a Raspberry Pi. I’ve already had, personally, had that experience, so it’s not that hard.
MOGLEN: We can call that a Freedom Box.
AUDIENCE MEMBER: : Do you think that–
MOGLEN: If you wouldn’t mind turning on that microphone so the tape can hear you.
AUDIENCE MEMBER: : Do you think, as a rule in the future, that both Lidar and internal driver side cameras are going to be mandatory for auto driving in level 5 and such.
GILPIN: That I’m not sure about. And there are a couple reasons. I think what a lot of people don’t realize. So Lidar is this laser-based technology that basically comes out of self-driving cars. I showed a picture of it in my slide, and it basically pings back to your car when you basically run into something. It’s extremely expensive. So we’re talking about 180 degrees and multiple hits for each angle. It’s very hard to process that type of data, so when you think about the amount of computing power that’s just going into self-driving and then the sheer amount of computing that’s going on just to process the Lidar and then also to process the video. It’s a lot. That’s adding a lot of weight to the car. So, for that question, I’m not sure just based on pure computing power, but I think a lot of people are very interested in Lidar. Especially, there’s a group at MIT working on putting Lidar technology on the chips themselves.
MOGLEN: Let’s suppose just for a moment that we say that the political economy of this is that the only reason to spend all that money on hardware is to eliminate a job. The part of this that we need to think about is that from the economics of fleet activity, the only reason to spend all that money and take all that risk, technological risk, legal risk, all the rest of it, is in order to fire an awful lot of people. Which tells you that the gig economy isn’t quite as cheap for the non-employer as you might have supposed, because of course all the real money is in getting rid of those people.
This means that that form of automotive transport’s economics, like the economics of the Cloud itself, are really about firing a bunch of workers. The problem is that once you fired that worker once, you can’t fire her again. The savings are all up front, which is part of why we now see a price war in the public Cloud. Everybody fired all their IT staff. They moved everything into the public clouds, but they can’t recoup that savings every year. The people are gone. Now they begin to have to put pressure downwards on the price of a computing cycle.
I think, as apparently the enrolled skeptic here, about all this, I think the primary point here is that everybody now sees a killing in eliminating drivers. The trucking companies of the world see a killing in eliminating drivers. The gig transport companies see a killing in eliminating drivers, but it only works if it’s a tightly oligopolist system, which is why the shared mobility principles dot org view of the world is human beings should never be allowed to own their own self-driving cars.
This is a way of changing the labor economics of transport. This isn’t actually a really good way of changing transport. Everybody can see that there’s going to be a long lag in this technology before it becomes even nevermind safe, just theoretically possible.
And once again I want to point out that everybody is doing this in places where it never ever snows. I like the idea that maybe a hurricane can move a mailbox across the street and that you can teach a computer that. But if you could teach a computer to drive in a New England blizzard and to find black ice and not be confused about where it is then something really, really clever has occurred.
All of this is pure terrarium. It’s not the biosphere we actually live in. It’s some little tank built to make it appear that this can work, and money is flooding towards that because people have jobs, and we would like to take them away. And whether you’re in favor of taking away people’s jobs like somebody I could name who used to be speaker of the House of Representatives, or not, it’s certainly the case that the economics of this only makes sense in a very short run.
I know everybody in this room knows it, but you can’t take a self-driving car to a car wash, right? Nobody actually thinks you can maintain these vehicles by just sort of squirting soap and water on them from time to time or buffing with the wax and soft cloth. All of this stuff is incredibly fragile hardware, which says it works only under conditions of optimal maintenance in a world that– automobiles don’t do that in the 20th century. Automobiles drove around the world and went through the Sahara and went through dry riverbeds in Pakistan. And even my own car, if it could only have been cleaned with a toothbrush, man, I would have had to abandon it day one, right?
All of this is grossly unrealistic. It doesn’t make any immediate technological sense. You have to assume that there are a large number of people whose livelihoods can be eliminated before this begins to make any economic sense and tens of billions of dollars are flowing towards it for very little reason. But we know for sure that the data that is generated is going to be valuable to platforms. We know that that is really there. That’s the petroleum underneath all this, and how we govern the software determines who owns the data, which means that these issues we’ve been talking about have low-level software control are going to be where the real economic action in all of this is. Mark, you were meaning to say something.
MARK SHUTTLEWORTH: : So Leilani, the reasoned view of decision-making was very interesting. It strikes me that it’s essentially a programming language. You have some constructs as a sort of data or primitives. And then the outcome is completely deterministic based on the code. So like any software it will have bugs, in the refrigerator. There’s very clear evidence, hard statistical evidence, that the inference-based approach just produces a better result than humans. Humans are essentially doing a kind of computing that’s very similar and they’re doing it in a pretty average kind of way, and they’re easily distracted. So we will, in my mind, there is no doubt that inference-based driving will be much safer for everybody than humans doing the same thing.
So, therefore, it’s going to happen. Imagine, though, that people then also want to have your explanatory approach in there. How would those two systems work together? You’ve got a bunch of sort of silicon wetware effectively, that can’t explain itself but that is probably going to give you statistically a much better result than anything that’s codified, and you’ve got a codified system that might be able to rationalize or explain a behavior. How would you see those two working together?
GILPIN: That’s a great question. That’s a question I’m trying to answer. So in my group, in my research group, we’re a little bit turned off to the probabilistic-type approaches, but I think that they’re necessary, and I definitely think that they have to go hand in hand. So the way I would love to see it is just two systems working at the same time. One doing the decision making that it can do with its best sort of results, and then an explanatory system evaluating that that made sense with the current data and the current system state that you have. I don’t know how tangible that is right now, because it’s fairly slow.
SHUTTLEWORTH: I was struck with your mailbox crossing the road example. It is literally like a person saying, “I don’t believe my eyes.”
GILPIN: Well, exactly. Yes.
SHUTTLEWORTH: Which is not a good sign.
GILPIN: Our explanatory system is sort of two parts. One is to show the operator or whoever is in the car what’s going on and what’s reasonable or not. But the second case is for actually the system that you were describing is, can we take the evidence that we’ve created in this explanatory model and then feed that back into the system and say, “Don’t use this information as much,” or “Do use this information,” or “We think this doesn’t make sense.”
MILINKOVICH: I read an article just recently about chess, and one of the things is– so it’s been a, for years now, you can get a computer, by today’s standards not even a supercomputer, and they can, it can beat a chessmaster, but they did some interesting experiments where you take the best supercomputer and you team that up against a chess master who has just the cheapest possible AI for chess, like a handheld, the kind of chess simulator that you can get on your smartphone. And a supercomputer cannot beat that combination. So maybe to Eben’s point, we should stop thinking so much about fully autonomous systems and start thinking about assisted systems because there is some evidence, and chess is a very constrained environment, but there is some evidence that perhaps the combination of artificial intelligence and a human intelligence, that combination is actually much stronger than either by themselves.
AUDIENCE MEMBER: So what you’re saying is for deterministic systems or fully understood systems, they could be effective, but for highly non-deterministic systems, I really question that this will happen.
MOGLEN: Right, or it might happen for a different reason. I agree with you that the Grand Master assisted by the palmtop chess player example may not be very useful for this, but one of the things that happens when human beings drive automobiles is that they make reflex decisions which are wrong and they make them because they’re emotionally affected by the environment.
You can teach somebody to turn into a skid, but you may not be able to teach them right the first time, and it’s the first time that matters. There’s a reason that the airplane says to the pilot. Pull up. Pull up. And it isn’t because the pilot doesn’t know. It’s because under certain kinds of circumstances even a pilot who does know something is obvious as pull up, pull up, might actually have her behavior modified by being told that at the right moment by navigational aid.
So I do think that the argument that Mike is offering might be correct. If Leilani had a really good explainer that was working 15 seconds or even two seconds ahead of the driver, some kinds of interventions by assist might really make a difference to human behavior, even if you’re ultimately counting on the human being to do the right job.
MILINKOVICH: Not to mention extra sensory capabilities. Right? So Lidar is very expensive now. If you were producing a billion light hours a year, it probably would be a lot cheaper. Just to give one example. But even with cameras and so on, it is possible to easily imagine an assistant that could sense things that a human being might miss or would be distracted by. Driving late at night in the rain, there are definitely sensors that could easily augment our own eyesight.
AUDIENCE MEMBER: Don’t we have that problem in medical systems now, that you have assist systems that are resulting in doctors not being able to diagnose very basic things?
So it’s not a feedback-less system. The long-term consequence, we’re not looking at that. For short term, that’s the same thing that we had with ABS systems. When ABS was introduced it massively reduced the accident rate until everybody adjusted his driving style to ABS.
MILINKOVICH: That’s actually not true. When ABS was first introduced, the accident rates went up because people freaked out the first time their brake pedal started fluctuating. So there is actually, when ABS first went out, this is back to the first use thing where steering and steering into a skid, right? There was lots of evidence that– The first time my mom’s ABS went off, she had no idea what was going on, she went “Eeek!” and lifted her foot off the brake, which is not exactly the right solution.
But I think what you’re pointing out is a truism. It’s like humans and computers are augmenting ourselves throughout everything we do. They don’t, at least in Canada, they no longer teach handwriting, right? They don’t teach the multiplication tables, right? Why would a human need to know that stuff? So this is a fact of life in pretty much all aspects of our future cyborg experience.
MOGLEN: Now we are back again to one of the reasons we care so much about users’ ability to modify and improve the technology they use, because there is a feedback in the system, unquestionably. In fact, non-deterministic and hard to calculate, but one of the things we can do is to assist human beings to adapt to technology by making it possible for them to adapt the technology to themselves.
Everywhere that we make it possible for people to modify technology, the result ought to be over time, a convergence on forms of human/machine behavior which are more advantageous to human beings. This is what I have always thought the free software revolution was about, was allowing people and computers to adapt to one another in better and less power redistributive ways by allowing technology, by forcing technology to allow users the effect of their rights.
This does seem to me to be a part of the conversation we’re having today, my part I care a lot about, with respect to why we would want people to modify the software in cars. Whatever are those adaptations cycles that we are all now talking about, how we adjust, how the technology adjusts, how we adjust to how the technology adjusts. It is the recognition that users have rights and that they should be able to understand and affect what’s going on, which enables that cycle to achieve the kind of productivity we’ve seen in the digital world. I do not think that we are, in any way, likely to make this trip you all say we are going to make to fully autonomous driving at all quickly unless user innovation is a large part of how we get there.
And that ought to involve, one would think, evolutionary activity from less assistance to more assistance, back to less, forward to more, as we reshape driving. I don’t particularly want to lose handwriting, and I don’t particularly want to lose mental arithmetic. I also don’t want to lose the skills that human beings have built up over the past hundred years in driving cars. It may very well be that inferences are going to do better than people, Mark. I don’t know whether that’s true, but if it’s true, it means the machines had to learn from somebody. Who is there to learn from except us?
AUDIENCE MEMBER: We will know if they are better than us when they start lying to us for their own profit. [Laughing]
MOGLEN: This is why as Mark Zuckerberg got ready to go and testify, everybody started watching 2001 again. I think this is exactly, I think this is exactly right. I think that Stanley Kubrick saw it, and I think people are being reminded of it right this minute. That is exactly what they are afraid of. That’s why HAL 9000 continues to be such an important part of the history of computing, even though it never existed, because people really do believe that. I think that’s exactly right, Nicholas. That’s it in a nutshell.
AUDIENCE MEMBER: : On that note, I was wondering, when a human being is in a car accident, they’re presented with a lot of raw sensory data and they often confabulate, sometimes unintentionally, an explanation for what happened that puts them not at fault. I was wondering, what is the best safeguard to ensure that a machine does not also confabulate an explanation why it is not at fault? When some of the actors in the supply chain may want it to do that.
GILPIN: I wish my advisor was here. This is his bread and butter. So what he would say is basically you have to have this constant and consistent monitoring system. So you wrap all the parts in this sort of monitor that’s constantly explaining what it’s doing, and you keep building those sorts of layer. Everyone is checking on each other at a certain, at each layer, and each part of that system. So if you have those sorts of inconsistencies, maybe you say, oh, there are two scenarios, right? Let’s say that either the brakes lied or they were truthful, and you present two different stories and see what’s believed in that scenario.
AUDIENCE MEMBER: : All we have to remember is how the computer in the movie 2001 lied through its digital nose.
MOGLEN: Well, that’s one way of thinking about what happened. Another way to think about it was that it had no nose, but it had a profit motive or a self-interest.
I mean, autonomous agents are either going to explain how they work in the 21st century, or they will run things. I do believe that we are moving rapidly into a world in which there will be two kinds of autonomous agents in the 21st century: one of them will be built as Chinese technology under the control of the Chinese Communist Party, and one of them won’t.
I have a deep political concern for the autonomous agents of type 1, and I have a little bit of optimism about the possibility of autonomous agents of type 2. Skeptical as I am, I have no desire to abandon the field, because I recognize that human power has an awful lot to gain from the creation of unaccountable autonomy located in centralized platform systems that are subject to government power. That’s a recipe for an eternal form of despotism I do not want the human race to live in.
The mythology about that is entirely credible. It’s not that it’s off-the-shelf technology right now, but there is no doubt that we can be two decades from that. We can be two decades from a world in which powerful autonomous agents occupying themselves with vast data collected out of civil society are running civil society. Chinese Communist Party has already attached the citizen-rating system to the purchase of mobility. Train tickets and airplane tickets in China are now being sold on the basis of checking against your citizen rating. That is essentially a system of restricting mobility for unpopular persons and further control over those forms of human activity which are dependent upon mobility, which pretty much all human activities are, of any value, will come increasingly to fall under algorithmic control. That part I don’t feel skeptical about.
The question how good we can make the products for the people is a hard question, but 21st century political economy says that in the 20th century, we are products. In the 21st century, we are services. And as we watch products becoming services, and mobility is doing that. That’s the subject of our meeting. As that becomes a service, the people become a product. Everybody is feeling that now about social media and these cheap services we all know how to federate that are being offered in centralized form in return for total surveillance all the time. Now people are beginning to wake up to, Oh, so when Facebook is the service, the people are the product, and we are talking about the ways in which mobility, which is crucial to the 21st century economy as it was in the 20th, is going to become a service, and we are therefore implicitly discussing to what extent the people are gonna be the product.
We are worried about their safety, physically. We are worried about their insurance rates and the nature of whether they’re going to own a car or rent a mobility service, and behind that I want to say we are worried about what the effect is upon the autonomy of people, at the deepest level. As it becomes the case that we are the product of the mobility services we use, the product is knowledge about us, everywhere we go, how we get there, what we’re doing while we’re moving, what we do when we arrive, all of which at the end of the day is control somebody exercises over people. If we think about this in relation to the day we’ve just spent, and we cast our mind back to what it would have sounded like if we’d been talking about software for social networking 10-12 years ago, a thing I used to do back then when there was still some point in it, we would recognize the ways in which the concepts we are manipulating today are going to have an effect on the privacy and therefore the autonomy of human beings 10, 15, 20, I don’t care how many years from now, which will further affect what it means to be a human being in the 21st century. For this reason, at least for me, for this reason, it is absolutely crucial that we figure out what people’s rights in this technology are going to be and how to secure their ability to exercise them.
AUDIENCE MEMBER: : So, if it is inevitable that the useless class is going to kind of predominate the 21st century given the amount of automation, and discrimination becomes personalized based on the data that you and I as individuals are generating ourselves, how can we stop people becoming products, or is it inevitable given the interconnection of systems?
MOGLEN: That’s another conference on how to use Raspberry Pi and Freedom Box to liberate the human race. Come back in the fall, I’ll be about ready for that then. I’ll send you an invitation. More about cars?
AUDIENCE MEMBER: : Leilani, I am curious about, when watching your presentation, some of the ideas, it sounded interesting that a lot of the logic you describe sounds like exactly the logic that the car itself has to use to drive in the first place. Right? Many of the cars, as I understand it, the Teslas and many of the cars that are being developed with that kind of goal, actually, they are surprisingly based on heuristics instead of just network, neural networks, and so on. Which means it sounds like actually the kind of logic that the car itself uses to drive is precisely using that kind of semantics, right? It’s like, I can see a mailbox, and then they inform and decide, and there’s probably a log. So I would expect actually that the Teslas, for example, they have something like what you described. They have not just logs, but they have a decision-making procedure. How do you see that? Isn’t it a fact that it already exists? It’s already something that can explain that kind of logic?
GILPIN: Hi, sorry, can you repeat the last part of your question?
AUDIENCE MEMBER: : Yeah, so the question is really, how do you see that? It sounds like from, it may sound like looking at what happens, at least in that area, can see how your work is really useful in many of those systems. But at least in that area, it sounds like the car already has to make a decision, the car has to explain to itself that this sense while given those sensors, this is why I’m drifting. Is it the case?
GILPIN: So, I guess the short answer to that is I don’t know. I don’t know what any of the car manufacturers do as of now in any type of error detection. If I asked them, and I say we have our system, we’d love to compare it to yours, they come back and they say, “Oh, you know, that’s proprietary. We can’t tell you what we do.”
I can tell you what I think that means, but I’ll spare it. So I would hope that that’s the case. But I think something novel that we have that they don’t do is we keep track of all the dependencies of that decision. So a lot of car companies, I believe, don’t keep track of all those dependencies. Maybe they keep track of the one they thought was the most important in that decision, but we really keep track of that from step 1, and then we use our reasoning system to figure out who is at fault. So I think that’s, it might be the same process, they might be doing a similar thing. But I think that we try and keep track of the right sorts of things to make the explanations that need to happen.
MOGLEN: You know, Gustavo, this is going to be a place where performance is going to affect design fundamentally. Nobody has enough computing power to do everything in real time. They have to throw a bunch of stuff away. We are both aware of the ways in which technology design in our corner of the world has been affected by chasing performance at all costs. We’re going to have meltdown inspector in many different senses as Nicholas was pointing out with respect to the non-deterministic behavior of multicore machinery once you start doing branch prediction and all the rest of it.
What we can be pretty sure of is that whatever Mr. Musk’s programmers are doing, they’re throwing away most of the information she is keeping because it would take too long with any economically feasible set of machinery inside the car to consider everything all the time. So there’s probably an awful lot of triage in that design, right? There’s an awful lot of flying by the seat of pants because that’s the only way you can do it quick enough to do it in real time, don’t you think?
AUDIENCE MEMBER: : Not so sure. I actually can imagine that they actually do something very similar to what Leilani presented as an explanation to actually make the decisions. For example, even if we think about a case that is being so debated in the last few days about the accident, look at other videos and other people are sort of posting the same kind of behavior. It looks very clearly like the car is just centering it between the lines. The car is centering between the lines, and then there’s something in the middle that it cannot detect. The centering between the lines is very heuristic. There are two lines and then you make the machine actually center between the lines, which means this is easy to log. It’s easy to explain. It’s just that, I think, yes, I agree that it’s going to take a while, but it doesn’t sound like the problem of, for example, real time is so difficult in that kind of scenario.
MOGLEN: Ah, that depends upon how easy it is to see the lines. I like driving in California because it never snows there and the roads don’t break up in the winter time and the lines between lanes have nice little raised reflectors on them, so it’s much easier to drive. Over here or in Brazil, we don’t have quite such nicely marked lines, right? And so the algorithm for centering between the lines is a lot more complicated, and it has to deal with cognitive gaps when you don’t know what the lines are, and you decide what to do about that, which human beings have some heuristics for if we have been driving for several hundreds of thousands or millions of miles.
AUDIENCE MEMBER: : But, by the end of the day, you still have sensors that detect lines or detect people, or detect mail boxes. And those are inputs to a system that is probably not just the neural network. It is heuristics-based.
MOGLEN: Maybe, and it may have room for an ontology as deep as hers, but it probably doesn’t have time to consult it. So it’s not busy trying to distinguish between mailboxes and inflatable toys that look like mailboxes or skateboards with people chasing them or just a wheel from a skateboard that fell into the street and is lying by the side of the road.
I don’t know how that stuff is being done. I’m sure they’re keeping it very secret. I really want to read the code, like you want to read the code. Like you’re suggesting that if they could show her the code, she could finish her Ph.D. faster and better. We’re all in the same condition. Could we please see the code? To which the answer might be, yeah, we would show that to you in some sense, but don’t ever think of experimenting with it, in which case we haven’t really seen it, because we really don’t know how to deal with computer programs just by reading them. We all need to fool around with them. We always have.
I do think you may be right, that we may be going in there is really, really good, and it may be that if you and I spent an evening reading it, we would think that this code is terrible. Do they understand how dangerous this is?
AUDIENCE MEMBER: I’m not saying either way, just to be clear. I’m not saying either way, right?
MILINKOVICH: My take, I’ll bet, I would bet big bucks on two things. One is whatever that software is, it’s not rules based. I can almost freaking guarantee that’s not how it works. And the second thing is I’m pretty sure if you read the code, it wouldn’t be that interesting because most of the stuff that actually matters is the weightings and the data that they’ve that used to teach it. And so, you could have, the code would be very, very generic and boring. It’s all in the data. I can guarantee you they’re not giving you the data.
MOGLEN: Right? Or at any rate that’s what our next conference really ought to be about. Right? Because that is where the action is going to be. But in order to do that we first have to establish some other things. We have to figure out how we can prove to people that we can operate in ways that consult everybody’s interests and also have some room for our rights to inquire into and fool around with this technology.
Nicholas said a thing when we were at lunch together which bears repeating here. His argument was there should be no two identical cars on the road ever. That monoculture is a deep threat to safety and to good technological development here. And at 10 to the eighth units deployed, it’s going to be hard to make them all different. But assuming that there are, let us say, 10 to the fourth hackers who really want to do this, and 10 to the eighth cars out there for them to do it to, it would be really a good idea to allow an awful lot of diversity before we create technical monocultures of any kind, let alone lock them down or keep them secret or in other ways make it harder for us as a species to improve this stuff upon which our future does indeed depend. More?
AUDIENCE MEMBER: To the point about the curious separation of the explanation system and the thing that’s doing the things that need to be explained, I remember reading some neuroscientists say that in humans, when a human is challenged to explain something they’ve done, the mechanism in their brain that they’re using to generate that explanation is not the same one as the one that caused them to perform that action. There have been some split-brain experiments that showed this by showing a situation in which humans’ explanation could not have been the reason that it performed that action. So I suppose, to pose that as a question, is it possible that if we produce these explanations systems that they might in fact take a form like that. And if so, is it actually still going to be dependable as a way of having a neural network or whatever explain itself, if there is that separation?
MILINKOVICH: Left brain, right brain. Like two computers and one computer to drive the car and the other computer do nothing but explain why it did what it did later. Actually, you know what, I’ve heard worse ideas.
AUDIENCE MEMBER: Well, I mean, I say that because you know there’s precedent in the biological world that counterintuitive as it might seem that might end up being the solution. And if that turns out to be the solution, might we actually discover it’s of no use?
MILINKOVICH: I’m pretty sure you’d find out is that part of your brain that makes up the story that explains it is the same part of your brain that knows how to lie. I’m just guessing, but I’m pretty sure it’s the same spot.
AUDIENCE MEMBER: I think that’s right.
MOGLEN: Well it’s a thing we did before. We’ve done it throughout the history of computing. We’ve built machines basing them on the way we thought we thought, attempting to copy the nature of human intelligence, and we were wrong every single time. We built very interesting stuff, but never did we actually build what we thought we were building, which is us, and that’s either a really good outcome or a really frustrating outcome. I suppose again part of that disposition here is that I always thought that was really advantageous, as long as we could read the code and understand why what we had built wasn’t who we are.
You don’t talk about floating point problems and the APL interpreter on which I spent many years of my life trying to get it all to work right. It was very important that machines don’t do arithmetic the way people do arithmetic. That was really very helpful. It reminded you at all times that you couldn’t possibly trust anything you thought you knew about how you do arithmetic because ten times point one isn’t exactly one in there. And it makes a very big difference to know that the machines we built don’t understand math the way we understand it. They understand it differently. Fuzzy comparison could not exist without that insight, I believe.
All right. I think as it is past 5:00 o’clock that we should say thank you to our guests for an extraordinary effort.
Previous: 4b-milinkovich | Next: 6-closing | Contents