MOC: Can you tell us about your work and how it's evolved over time and how it has lead up to Kismet..
Breazeal: When I first arrived at MIT for Professor Brooks' group we were still in the middle stages of building insect-like robots. The thing that was interesting to me was the focus on autonomous creature- like robots, as opposed to manufacturing robots. I was always captivated by the droids in Star Wars as a young girl. Those were the kinds of robots I wanted to build. When I came here I saw these robots as being the precursors to what someday might be like R2D2 and C3PO. So that's always been kind of a motivating factor in the back of my mind. We built insect like robots for a number of years, until about1993, where I focused a lot on six-legged type insect robots. A lot of that work was really inspired by planetary exploration and the idea of building micro rovers. At the time NASA tended to send up very large rovers in order to do a mission, and our group really advocated the use of small robots. Of course that had a very important impact on the Pathfinder mission. So my early work focused on legged locomotion over rough terrain and fault tolerance of planetary micro-rovers.

In 1993 Rod went on sabbatical. I didn't know it at the time, but his intent when he came back was to do something completely different. A lot of people were then doing insect Robots and Rod always wants to do something different. He came back from sabbatical and announced that he wanted to do humanoids which was really quite a surprise. The dogma at the time was first you do insects, then you do reptiles, then dogs and finally you get up to humans, and Rod just wanted to jump straight to humans. We started building COG in 1993. I was a senior graduate student at the time so I was a chief architect on the project. COG was surprisingly similar to my six-legged robots- with more sophisticated sensors, actuators but actually a lot of similarity in philosophy of how you build robots. Of course it was a huge engineering endeavor just to build the physical robot. You couldn't buy a humanoid robot you had to build it all from scratch. We were building not only the mechanics of it, but also the electronics and the computers and everything.

A lot of the early work was focusing on sensory motor coordination. There was always a stress, not surprisingly on the ideas of embodiment, how important having a body was for intelligence, how important being situated in the real world was important for intelligence. The point was, if you really wanted to understand human intelligence, it was important to have a human- like body, to have human- like interactions with the world. So that was always a constant theme. The focus on embodiment was also tightly related to the question of learning, which is that in order to learn from experience, you have to have something to get experience through, and that of course is this body situated in this environment. We always talked of these questions of development- so not only learning but long-term learning, the trajectory of learning, heavily inspired by infant development. But again, a lot of the work was focusing more on these physical interactions with the world- Understanding how your own body moves, how as you move, your sensors change with respect to your movement- moving slinkies, turning cranks and so forth, very physical sorts of interactions.

I was always interested in the social side of it but that seemed to be a very high level sort of thing. There was a time around 1996 or so when we had finally come far enough along- the COG had a head, an active vision system, a body, it had an arm. We had developed the vision system enough so that the robot would be attracted to moving things. It would turn to look at them, orient its head. We had a separate program running where the robot would go through a reaching motion. But the vision and the arm control were not connected at all. So it wasn't as if the vision was telling the arm anything about where to reach. But there was this moment where, in a spontaneous interaction with the robot, I started playing this eraser game on a tabletop with COG. I would wiggle this eraser on a tabletop and its vision system would be attracted to that, and it would look down at the eraser, and then the arm would do its thing, which was this reaching and grasping motion. The thing that really struck me was that, to an observer, watching this interaction, it really looked like we were playing a game. We were doing this turn taking thing. We were essentially having this social interaction, with very little actually programmed into the robot. What I realized after that was that this idea of social interaction, and socially situated learning, was actually much lower hanging fruit than what I had thought. I figured it was ten years off before we could even start thinking about those ideas, then I realized that, no, I can start thinking about them now. And that's really when I started Kismet. In stead of these physical interactions, pointing at things and stuff, what I really wanted to focus on was this close face to face intimate, interpersonal interaction and teaching scenario, that happens for example between a mother and her infant. Focusing on the facial expressions, focusing on the eye to eye contact, focusing on this close intimate interaction.

At that point I left COG and went on to build Kismet. Kismet has been a natural progression over even the algorithms and ideas we've used to program our robots. With the insect type robots we were talking about coordination between legs to do different types of gaits. Even back then a lot of the work was inspired by biology- looking at models of locomotion and putting them into the robot. That theme has held consistent through Kismet but now you're looking at very different kinds of systems. You're talking about computational models of emotions for instance, which you never had to address in our earlier robots. You had to think about very sophisticated perceptual systems, visual perceptual systems, speech processing, speech synthesis, all kinds of expressive abilities, different kinds of behaviors, so the models were now being drawn from psychology and ethology and less from neuroscience. There's been a progression in our style of programming and how we model our systems that's had to match the sophistication of the robots. Its staggering progress between building Kismet and building an insect robot in the early 90's.

MOC: Can you talk about why social interaction is important in a robot
Breazeal: There are two different angles to that question. One is a learning angle, and this has always fundamentally been the motivation behind Kismet. The question was: Is building a social robot the route to building a smarter robot? Robots have to deal in a very complicated world. You have to design these algorithms which tell the robot how to deal with a particular experience. It has to be very explicit. The learning algorithm is so specific to that task that the robot can't learn anything else.

A lot of the assistance that people give their children to help them learn is extremely social in nature. So you're talking about having to build a very socially intelligent robot in order to learn from people in this way. That's what we're really pushing with Kismet- this idea of socially situated learning. Learning from people.

The other angle to why you want to build a social robot has to do with robots coming into society at large. We see Sony's Ivo, that's one of the first entertainment robots, which is sold to be a domestic robot. We see the beginnings of I Robot which is a robot that lives with you in your house. So finally, robots are actually starting to come into society at large, which begs the question, how should robots fit into society?

MOC: Can you talk about what happens when you're interacting with Kismet.
Breazeal: Kismet can display and perceive a wide variety of social cues. Because much of Kismet is motivated by this question of social development and is heavily inspired by infant social development it's not surprising that many of the robot's initial perceptual systems, behavioral systems, what are called proto-social responses, are taken from the developmental psychology literature. We look at what infants are capable of and use that to design what the initial capabilities of the robot should be so that it can launch into these social interactions.

When you have a proto-dialogue with Kismet, the robot doesn't know the linguistic content of what you're saying. Just like when you hear Kismet babbling you have no idea what Kismet's saying. But the mechanics of the interaction is there- the turn taking is there; the idea of this regulation is there. One way the robot does this is by giving you a number of social cues, through its face and body postures that lets me know when its ready for me to take a turn. People use these cues very naturally with each other. Much of Kismet's facial expression is not only used for emotional expression but for communicative function.

Another important social cue is gaze direction. Gaze is probably one of the most critical powerful social cues we have. In a teaching scenario you're trying to direct the robots attention to something you want it to learn. Knowing when you've achieved the robot's attention is critical and the cue we use for that is gaze direction. One reason the robot has such compelling blue eyes is to draw you're attention to its eyes so you can tell what the robot is interested in.

As far as cues that the robot is perceiving from people. The robot can sense the presence of skin tone versus bright colors, motion. Those cues can be combined to find out: Are people engaging me versus a toy? We know that human infants have distinct responses when engaging people versus things. Kismet has that ability to determine whether it is engaging things versus non-human things in a very simple way. It can also detect the presence of eyes which is also critical. It can also detect the melody of what you say, It doesn't know the linguistic content, but it knows how you're saying it. So if you use a praising tone it will understand it as praise. If you use a very scolding tone, is understands that as a scold. It understands attentional bids, it understands soothing.

The robot can also express, through its voice, emotional state. The robot itself can sound angry, happy, and sad. As you interact with it you can get cues from its voice quality. There's a wide repertoire that the robot has in order to communicate and read cues from people through either vision or auditory channels. Eventually we're looking into extending into tactile channels as well.

MOC: In what sense does Kismet have emotion?
Breazeal: There are two distinct systems within Kismet. One can be thought of as being homeostatic regulation processes. These are drives like hunger and thirst. It's trying to regulate certain kinds of stimuli within a bounded range. For Kismet, its food is essentially different kinds of social interaction. It has a social drive, a drive to interact with people. It has a stimulation drive, a drive to interact with toys. It has fatigue drive and goes through an actual sleeping behavior. There are also these computational models of emotions. A lot of that work is inspired by the theory of basic emotions. Why do we have emotional responses and what function do they serve in a creature? Broadly speaking, we have emotions because they promote our survival. They serve to bring us into contact with a desired stimulus and to avoid dangerous or undesired stimulus. That's essentially the same way they're modeled within Kismet.

Essentially what emotion is for Kismet is a whole biasing over all these other systems. So it biases the expression and the behavior of the robot. It biases attention. Eventually it's going to bias things like learning and memory. So its more the idea of emotions as this influence over these other existing systems that shape the robots behavior and learning and memory retrieval. That's taken from our understanding of what emotions serve people.

Of course that totally side steps the issue of conscious experience and awareness of the emotional state which we consider to be feelings. I don't know what it would mean for a robot to have feelings yet. We're not addressing that issue yet either. That's not what Kismet has.

MOC: Can you tell us what emotions you are modeling and how they're expressed?
Breazeal: In the theory of basic emotions there's a bit of controversy as to which ones constitute this initial repertoire. For Kismet, they correspond to anger, fear, disgust, happiness, sorrow and surprise. They're basically emotional responses modeled for those as well as some arousal states; things such as being tired is also modeled within Kismet. The trick is really to match Kismet's emotional responses to people's intuitive understanding of why you have these responses.

You can look at all the eliciting conditions that give rise to certain emotional responses. For Kismet, the disgust response is used to let you know that the stimulus you're giving it is undesired. Sorrow for Kismet is the result of the loss of desired stimulus or prolonged lack of stimulation. In human infants, crying is one of the most powerful communicating modality children have. The reason children cry is because it elicits attention giving responses from adults. With Kismet it serves the same purpose. When the robot is looking really sad and lonely people can't help but want to come up and play with it. It's the same kind of function although Kismet doesn't cry per se. Kismet is fearful if there's a threatening stimulus. It tries to avoid the stimulus if it's bad enough. Kismet exhibits happiness upon success. If its seeking a desired stimulus and it encounters that desired stimulus it's successful and that causes it to smile. That's my cue that what I gave it is what it in fact wants. There are many different responses and reasons for those responses in the robot. It's designed to be intuitive for the person interacting with the robot. So if they see an expression, they have an understanding of why the robot is expressing that way. It's important for them to empathize with the robot.

MOC: What is it like bringing these inanimate parts to life?
Breazeal: Kismet is explicitly designed to tug on your emotional heartstrings. What we're trying to achieve is this very natural infant caregiver interaction. Everything from the way the robot appears aesthetically, having these big blue eyes, being small, having a small distance between the lips and eyes; all these characteristic infant- like features, so that a na•ve subject will be likely to treat the robot like it's a young creature or like a puppy, in a very nurturing way.

The physical appearance of the robot has been designed to elicit care-giving responses. We have paid a lot of attention to making the quality of movement lifelike to make people want to nurture the robot. People like Kismet, they don't want to upset it, and they want to keep it happy. Those are the kinds of interactions we want to promote for the learning. When I interact with Kismet I feel this strong affinity for the robot. I'm very attached to it. I never get tired of it. I think that's because so much attention has been paid to making the behavior as lifelike and engaging as possible. It never is absolutely predictable. To me Kismet is very special because when you interact with Kismet, you feel like you're interacting with Kismet. You know, there's someone home so to speak [laughs]. There's someone behind those eyes that you're interacting with. That's very different from most other robots which you observe as being this cool mechanical thing. But not this personal connection.

MOC: How does Kismet work?
Breazeal: The robot has this compelling lifelike behavior because it draws from two different disciplines. One discipline is the scientific study of natural systems, such as psychology, ethology, evolution, studies of intelligent behavior. The other discipline is from art, things like classical animation. How do you portray this illusion of life? What is it about behavior that people find lifelike? Both of those disciplines have informed how and why we designed Kismet.

As far as how the robot is designed from the mechanics to the sensing ability to the software- obviously it's a very complicated system. There are fifteen computers that drive the robot. Nine computers alone are for vision. Vision is by far the most computationally extensive processing that we have to do. There's one computer for speech synthesis. There are four computers that run the homeostatic drives for emotional models, the expressive abilities, and behavioral systems. There's two computers for speech processing. So that's a lot of computation driving the robot. Part of that is because we have to do these things in parallel and they all have to be done at an interactive rate. We could run everything on one computer but you wouldn't be able to interact with it because its responses would be horribly delayed.

As far as the internal structure of the robot; we're leaving a time where you're building controllers for robots. For many years, you build a controller to get it to walk, to move its arm etc. We're starting to build essentially synthetic nervous systems. Because we're having to integrate all of these capabilities within one robot. I stopped using the word control and started using the word synthetic nervous system because that's really what it's starting to look like. People underestimate the importance of integration.

MOC: Can you tell us a bit about agent architecture. Is that the correct term to think about?
Breazeal: There are many interacting processes. Each little bit has its job to do and no one little bit knows everything about the system; there's no one bit that's responsible for organizing the behavior. It's the interaction of al these bits that gives rise to the behavior that you see in Kismet. That's influenced by what we understand about how the brain is organized, by Society of Mind, Marvin Minsky's ideas on all these agents, all these bits giving rise to more complicated behavior. .

MOC: So how would you define Intelligence. Is Kismet intelligent?
Breazeal: There are a lot of schools of thought as for what intelligent behavior is. The one that has dominated AI has focused on analytical processing of intelligence. Though physical, emotional, and social intelligence are also important.

I appreciate intelligent behavior in many different species: animals, robots, and people. Kismet is behaving in a way that maintains its state of well being. How do you survive in a complex hostile world? That's what has driven our evolution.

Kismet has the beginnings of intelligent behavior. Is an infant intelligent? You can test an infant compared to an adult human and its going to perform dismally in most instances. But can you think of a more amazing learning system. It's incredible. How do you measure intelligence? I don't know, it's a tricky issue. Obviously with Kismet I Ôm very interested in the whole learning question. I'm hoping that Kismet can become a more effective learner over time.

Kismet exhibits intelligent behavior. It's organized, it serves to bring itself into a state of well being, to avoid bad situations, it can communicate to people and in some sense it can understand what people are communicating to it in simple ways. Those are aspects of it that characterizes intelligent behavior.

MOC: Does Kismet learn in any sense?
Breazeal: We're working on the learning stuff right now. There are a lot of different kinds of learning that go on in any natural system. For robots, there's learning techniques that we've already used in designing many of our robots such as COG and Kismet to train up low level sensory motor functionalities; Trying to coordinate motor with sensory stimulus. In natural systems that's done at the neural level in the visual cortex at a very early age, within the first month of life of a human infant. The robot has certainly learned certain things. When I talk about Learning with Kismet, I'm talking about this very intuitive sense of learning which is a socially situated learning.

MOC: Will robots ever reach human level intelligence?
Breazeal: Like animals: dolphins, pigs and dogs, robots also will have their own kind of intelligence. Ideally you can design them in a way so that they can have enough of a rapport or analog with people so that they can communicate and understand each other's behavior. I'm trying to get humans and robots to interact so the obvious step is to model robot intelligence on human intelligence. Instead of trying to compare robots exactly with humans, it's more about what a robot is good at. How can they fit into society, how can they relate to us, how can we relate to them.

MOC: What is the robot's place in society?
Breazeal: There could be robots as essentially glorified appliances. These are fundamentally still tools and you're going to interact with them in a certain way, probably not a social way. The critical thing, because we are so early in the game, is to try to understand the phenomenon from both sides. How it impacts the robot as well as how it impacts the human. Too many people focus so much on the robot they neglect to understand that the human is a critical part of the loop. My work is focused on exploring what is a good way to design these types of systems. Its such an early state that we don't know all the answers yet. We have to study it. In the process of doing that we need to inform and get insights from society on how they want to interact with these things. And also be aware that society changes.