GD: Let me just say, before we start, I grew up as a documentary filmmaker. When I say "grow up," I mean real life and got myself a job, which was at the Guggenheim Museum; I grew up in New York City. I was assistant conservator... I got the ninth Porta-Pack in New York City in 1970. That was about 22 pounds of reel-to-reel side pack, and a black and white camera that you held out in front of your nose. And I moved from being an environmental sculptor into being a documentary filmmaker.

Then I went to Maine for a few years, because the New York art scene and video scene -- I needed to understand how small societies work, and I was interested in documenting what people did, and why they did, and it was very hard to look at New York City as a changing thing, and understand why it was changing. It was too huge. So I went up to Maine, and wound up spending about seven years there, built two houses, did every small job you could imagine in Maine, to survive, and made two great films. It was super-8 at the time. We didn't really have a way of attaching sound to it -- sync sound, and I was trying to study communities that were changing.

I went to summer school and, there, met Richard Leacock, who, at the time, was the head of the M.I.T. film section. He had established himself with D.A. Pennebaker in New York as kind of the cinema verite gurus of the early 1960s. I, of course, didn't know them then; it was already late in the 70s when I met them. I decided that this was somebody who I really, personally, wanted to work with. So I came to M.I.T. on a kind of ad hoc basis, and projected films for Ed Pinkus, who was a documentary filmmaker also, at that time. And, in return, I was allowed to go to the editing rooms from about six at night until nine in the morning, where I edited my super-8 flims. I eventually went on staff at M.I.T., and then was hired onto the faculty in 1988. When I was a lecturer, before I was faculty, I made several films that went on television.

Along the way, in 1980, I came across computers. So here was somebody who started in art and sculpture and painting, discovered video, went into film, came back to video (which was part of the film section), and then found computers. Computers, to me, were absolutely amazing because, as a documentary filmmaker, you go out into the world and do a lot of research, you find out about your story, and where does all that knowledge go? It stays in your head; it never gets into anything that can later be used. Maybe it gets onto a note pad. When you're in the editing room is when you call all of that up and start making decisions about what your movie is about. I thought, "Well, we can make really interesting, better editing systems."

While I was getting into the idea of making better editing systems, I discovered that computers were pretty interesting from the perspective of actually telling stories, and I began to sort of have a dream that we could represent content to the computer well enough so that the computer could go into the video database and pull up sequences of shots that made stories.

That dream was pretty well formulated in the min-80's when I went down to New Orleans and made a very large documentary, which actually went online at M.I.T. as a kind of learning program on architecture and urban planning. I had a relational database associated with it, and it was kind of a hypertext program. It was terrific, but it wasn't good enough. So for the next, however many years it's been since then, I'm still working on the same problem. The problem, to put it in very simple terms: There are 20 million camcorders in the U.S. right now, and there is no home editing system. We're starting to see desktop editing systems, but from my perspective, those lack something that is incredibly important -- they lack any real feedback from the user about the content of the material, and they lack any real engine that could help you shape a movie. If you're an amateur moviemaker, you can go out and spend a lot of time making a very particular creation. But when you're my age, and you have a job -- I'm an avid home movie maker, I have to say -- I don't even have any time to edit. They don't even come with a logging system that tells you what's on the tape. It's very frustrating to me.

So I kind of work in two directions. In one direction, I work on this dream of making better tools for all kinds of filmmakers: previsualization tools for theatrical productions, editing systems for novices and children. On the other side, I work on making programs, making content, for a world that I imagine will have a digital database which you can add material to, and have the engine smart enough to pull out a sequence of elements. All my students -- I have seven students right now -- they all work on some aspect of these problems. And I work very collaboratively with them on some of those projects, and I'll show you some in which I'm more involved than others

Now we're going to segue away from me, and to the Media Lab for five minutes, then we'll start looking at work...

We moved into the Media Lab in 1985. It has an unusual position at M.I.T.; M.I.T. has many laboratories, and a laboratory means that you take in outside contracted research. We're the only lab at M.I.T. that is allowed to take in graduate students and give them degrees, and hire outside faculty that are promoted through the lab itself. We're just now starting an undergraduate program, which will probably be a double degree -- you'll be able to major in any other thing at the institute, and if you want to take a second degree, you'll be able to do a Media Arts and Sciences degree. It's a $20 million laboratory, and all of our graduate students are paid to come work for us.

When the Media Lab was in the making -- 1979 through 1985, we thought that this was the vision of what was happening in the broadcast and motion picture industries, the computer industry coming together with the print and publishing industry. We didn't think that it would all happen in a big way until about 2000, and of course it's happened quite quickly. So a few years ago, we reorganized, looked at our faculty, and came up with this grouping.

Most of the initial Media Lab was in the information and entertainment section, or in the learning and common sense section. Marvin Minsky and Seymour Papert were part of the Media Lab. They came from the artificial intelligence section. Meanwhile, Nicholas Negroponte, Murial Cooper, myself, Richard Leacock, Andy Lippman came from the information and entertainment section, which includes musicians -- Tod Machover is one of the musicians.

But in the course of hiring new faculty, we found we had hired a lot of faculty in what we called "preceptual computing." And that was interesting from my perception. One of the thing that is interesting about editing systems is having a machine that is smart about content. The perceptual computing group, were looking into the image, to try to find out what they could tell about the image itself. Could they, for instance, find image that had simlar textures, or could they look at a database of faces, and find similar faces.

They've become quite good at that, and I use that in some of the editing system research that we do. What is interesting about it is that they want to find similar faces, and I want to find different faces; they want to find similar textures, and I'm probably more interested in, "Well, it's a texture that's kind of like that, but really it's more like this." If I want composition in an editing system, I might say, "I want a composition that's wavy on one side of the frame," rather than saying, "I want a composition that has a circle in it." It's kind of a different language, and it'll be interesting to see how we'll be able to cross over in a world where their computing is very heavy-duty; we make virtual worlds and things like that, but the idea that you can really analyze what is in an image, or what is in a sound, is something that systems really haven't used very much. And it offers an enourmous capability for feedback if we can get that into a system....

People have spent an enourmous amount of time and money on stereo sets, but we've had very little opportunity to spend a lot of time and money kind of tuning up our televisions and visual apparatus, and we're stuck in a very impoverished world where the computer sits in one room that has a desk in it or something, and the television sits in another room where you have an easy chair, and your stereo system may be in either of those rooms. We've made this wonderful transition in music, where music is a part of our human space. And maybe in our lifetime it will happen with video as well....

If you're not aware, I'm transitioning into themes now. One of the things that the Media Lab talks about in many different ways is the idea of personalization. It's the idea that if you have information, or a story, or a piece of technology, you want that technology to work for you. One of the ways that we think about it is that you might have a database of lots and lots of different sources of material, and you want some kind of agent to go out and pick out a news profile that you like. I'm less interested in all of the existing news profiles than I am in what we actually make, and what are the tools that we use as artists to make what we make. After all, if you look at the AP, the BBC or Reuters, there are people actually behind the making of the stories.

I have a project called the Video Streamer. It's kind of a neat way of capturing video where you run video into the system, and the frames are storied along [a timeline]... if we can pull scenes out of that video stream and send them to a computer to analyze, we can collect keyframes, then compare and contrast.

One of the things we've used this idea for is to browse news. So you could have a smart VCR that collects news over a period of days, then allows you to jump around and follow the Somalia story, for instance, three days in a row. Right now our VCRs are very impoverished -- there's no way that you can browse a feature film, or news, your home videos. So one of the things we think about is, how can the VCR change? One of the ways that I've worked at that is talking about how we can describe the video... once we describe that video, we have a very simple interface. We can pull out clips with the streamer and drop them into bins, and we can say that a certain piece of video is an interview and it's also a particular person. We can then build up filters which allow us to make stories.

Once we have that information set behind the video we can do things like create interfaces where we have a story about being a journalist during the Gul War, say, or a story printed under Pentagon rules, and we can say that we like a story that's more about being a journalist during the Gulf War, moreso than the Pentagon story, and we can splay out that news story as a news story, but with a particular take on it.

One of the issues you get into very quickly as you start to build these story models that go into databases... one way we can change video news is to collect a huge database of opinions, and for all news stories we could collect opinions over time. Today's news, we have somebody who says something today, they're totally forgotten by tomorrow; we've got somebody else tomorrow who's telling us something totally different. For very long news stories, it's more interesting to gain a mass of opinions over time for a particular story, then allow you to orchestrate that story based on a title that tells you something about what you think you are going to get back. That's what we're trying to do in this case.

The problem domain is the same -- lots of pieces and images. Can we make variable stories out of this? What are the variables that make sense? How do we get meaning out of this? We don't want to give you just random clips; we want the clips to come in a sequence to answer real questions about the story. And we don't want to have you go to a menu and say, "What I really want to know is whether journalists were restricted during the Gulf War or not?" That's not a very interesting take with today's news. You know, "A tax break for the middle class -- who is that going to help?" What you want is "Who is that going to help or hinder?" as part of a longer story. We haven't gotten to the problem of interacting with that story yet. Here, we're saying, here is a story model, here's a database, based on your headline you can go in, and we can orchestrate that story for you as a linear story. That's problematic for me. This is based on the idea that you can create a sort of knowledge base of what these things are about, and this is a tool that allows us to attach descriptions to video, and we do that in various different environments. I'll return to that in a little bit.

That's the theme about variable stories; here is the theme about display spaces. You could have a display space that is very large, like a whole wall of your house -- that has a very different aesthetic quality than a watch. But if I could send you a story on either of those display spaces, I would like that, as a filmmaker. I am not interested, as a filmmaker, to say, "I only want to create it for a wide screen." I'm interested in your getting the story in whatever environment you happen to be in at the time.

One of the things we're working on is, how does a filmmaker put his or her intentionality about the script into the box I showed you in a way that can also tell us about the display space. Another person we work with is Andy Lippman.

Q: Does it have to be a two-dimensional space?

No, and in fact I'm very interested in transformational spaces -- that I could have a wall in my house that suddenly came alive. We're very interested in objects -- how does movie and object work together? Right now I have a student who is making a video jacket. It has a screen on the back of it that is about 3 x 3". The whole idea is that you will be able to download a short digital movie onto your back. Guess who's using it first? Probably the Rolling Stones. That's a different story.

The idea of clothing that carries movies around with it is really interesting. The limitation is how we are going to get the movies to download, and why? Obviously, if you're walking down the street and a movie is playing on your back, you have it on you for other people to see, and if people are passing by, that is a pretty short time frame. It's fairly limited.

There are a whole lot of ideas we have now where media and toys intersect and interact. One of the problems with the theme I showed you before -- the theme of story -- how do you make that experience responsive to your interactions with it? That's where we are in th is storytelling idea.

This is some work by Murial Cooper and the Visible Language Workshop. News was a very fertile ground for the laboratory to work on, for many reasons. One is that people across various groups working on news could share things. In this case it is a student that was working with me, and with Murial. This is a text database of the AP wire, where the keywords that are listed on the news items are placed in a vector layout. News articles are filed with keywords; the first is heavily weighted. A first heavily-weighted keyword might be "music," the second might be "university," and the third might be a particular artist's name. When you go over to "university," that article would come up based on a keyword match. But what happens in this case is that the initial layout looks at the most heavily weighted keyword, but as you push into these words, you push into a headline, and you push beyond it into an article. This is running on the SGI platform. We're looking at this paradigm of driving through space in order to get at some content.

Again, the problem, for me, of the storytelling model -- where you just go into a database and putall these pieces together, and get a linear story -- you may as well have a linear story, because you've got to go back out of it, and back in again to get a different story. What I'd like to do is to use that story model, but modify it based on where you drove backward. I might start my story with "miscellaneous industry, but as I push into it, I might find that I'm interested in the relationships between miscellaneous industry and government. And if I can get the story model, in video, to modify itself based on that driving pattern, then I'd be really thrilled. The closest we've gotten to that so far are the navigable movies we saw earlier. They're marvellous the way the sound and image go together, but you've prepackaged something, and it never changes; once you've seen it, you've seen it. My dream is really to have something that you can go into many times. But I want the actual video elements, and their relation to eachother, to change. In the case of documentaries, that is the issue of complexity in the story.

One last thing, which again has to do with graphics. We have something we call a "salient still." This was work developed by Walter Bender and others in my group. It's out in the lab, and we're starting to use it. There's a chunk of video. It begins with a wide-angle shot of Yo-Yo Ma in concert at Tanglewood. Over time, the camera zooms in to Yo-Yo Ma's fingers on the cello. This is maybe 30 seconds of video. We call it a salient still because you have a picture in time, and the question is, can we get a still out of it that a good story that you have in the video. Can we get that out into a still, and use that still as a way to navigate the story? You have to think of that as a chunk of time, like in the streamer. Then you run a sort of filter on it, which allows you to take it, and as you get more information from zooming, you start to combine the information of the wide shot and the close-up -- the close-up has the most information, the most resolution. We can run various kinds of filters and emphasize different aspects of the picture.... To me it's an incredibly powerful technology where you're taking this temporal dimension, and making it a two-dimensional spatial dimension, which we can use as a navigational device in movies.

What is interesting about this salient still technology, is, there was a page-turner that comes into the frame and turns Yo-Yo Ma's page, and and we erased him. We showed to all of these publishers and said, "This is great stuff. Send your news photographers out with a video camera, and they'll be able to take video salient stills. It will be terrific for your newspaper, and give your photographs a whole new look." Well, do you think that they went for it? Why not?

Manipulation.

They are very upset, and we didn't think that would happen at all; we thought they would be thrilled. They said, "It's beautiful. We love it. But we can't use it -- it's not "truth." So we're in a very complicated, interesting place -- a whole other place, where we can do things digitally, and the question becomes, "What is ethical to do?" It's a

But isn't it, too, truthful?

It's a different kind of truth, one that has to do with the time. It was incredible to us that not even the sports guys would go for it. You have a wonderful basketball throw, and you never get the surround, you never get the context. Here is a technology that starts to give you that context issue in a still.

I just had a class of undergrads, and we sent everybody out to do a processed video, and the assignment was to make however many salient stills they wanted, to just see what they'd do. It was fabulous! These images came back, and they're like paintings -- they're really rich. They do lead your eye through a very complex moment in time, often. For instance, one of the students did a potter throwing pots. You'd think that's nothing, right? The pot is centered, it's turning; what's going to happen in a salient still? Actually, what you see is like a Doc Edgerton photo -- you see the hand in time, moving up the pot.

There was a Steve McQueen movie in the early 1970's, and it looked like this. One person would be doing one thing, and you could see what the other people were doing at the same time. It was an amazing film. The Thomas Crown Affair.... I'd also recommend the cheesy Arnold Schwartzenegger video, The Running Man, from 1988 or so. It deals with this moral-ethical issue you were just talking about.

Well, we're in it very deep right now. Of course, all these newspapers are going online now, and they're all worried about how their materials combine with other people's, and what you really want in a personalized newspaper.

Fo me, what's interesting is that the photograph has become so much a statement, that this IS the moment, and it is only one moment, and that we can take a time-lapse photograph -- and this is, in some sense, not anything more than a time-lapse photograph -- it's very odd. We'll see. If you start seeing these weird photographs in the newspaper, you'll know that they finally "bit."

As a maker, one of the things that interests me most, I guess, is how society changes. What is the nature of that change, and how do you represent it? Clearly, in a feature film, you can represent by simplifying everything, and having a fairly clear message about life changes. So we've introduced a situation, we confront that situation, we have some resolution of it.

In life, things are enormously webbed and threaded. We now have a medium that is now webbed and threaded. So for me, the issue really is, Can I create an interesting portrait of urban change using this technology, and using a particular display technology that allows the viewer to navigate, without telling them to go left or right, or without them being constrained to the outside of a photograph; where their navigation really operates in a conceptual space.

I told you I did a big project in New Orleans about urban change, but what I thought was unsatisfying about that experience was that we really couldn't make the storytelling engine rich enough. That came up in 1987; the display technology was horrific. We were using X-Windows. Basically, you had a window with the video and then you had your text, and you couldn't really use fonts in the text. So now we're trying to get this very rich vocabulary of fonts we can use, salient stills we can use for navigation....

The problem, now, for me is, How do we create this interface? What are the interesting stories? There is a potential for communication, but what is the really interesting communication? What is the compelling environment that draws people in? There are lots of stories we can make; there are feature-type stories. Let me just give you a brief view of some of the stories we are making.

We have something called the "thinkie." This is sort of a story where we try to put you through a process where you have to think through an idea. The most recent "thinkie" will be like an M.I.T. hack. You'll be helping the hackers think through their hack. And you'll be using email as a part of your process of understanding what they're doing. It'll run on the Internet. You'll be able to reconfigure the elements at the end and put it back out onto the Internet, so the person who comes to your story will have your story rather than our story. We're kind of playing with those ideas.

We have a virtual environment which has a very fixed story, and we're trying to look at how you edit a virtual environment, if you have a head-mounted display. How do we take you out of your one-dimensional, first person story space, into a third person story space, and bring you back to a first person story space, gracefully? That's the first piece in which we're using sound; we're really going after the sound problem in that. We have a composer who is writing music that can vary in time, so that as a cut comes up -- I don't know how it's going to work, but -- the idea is that it will be able to segue out of a musical theme, into another musical theme. And I think sound, for all of us, is an incredibly important issue that we have to wrap our heads around. I've done not as much as I would like in that....

I grew up in New York City, and my parents bought a double brownstone on E. 65th St. before the [elevated train] came down. I watched the El come down, and I watched that area change. So I have an absolute love for urban change. I think, particularly, some of our urban images today don't mesh with some of the notions of electronic networks and the sort of communication structures we're building. I think it's an interesting time to be looking at the urban center, and saying, "What can we do about our visual environment and our physical environment?" I'm interested in the problem....

I have six graduate students, and in the Fall semester, typically, I've always done a collaborative project for them, plus other undergraduate students in the building. Sometimes that has been a theatrical project. For instance, I did a collaboration with Larry Friedlander where we built a walk-through, what we called a transformational environment, which allowed you to walk through some big theatrical spaces. You either walked through or were guided on a workstation affecting them.

The Boston tunnel project is this year's and last year's collaborative project. The challenge that I put out was, "OK, this is an idea of a media project of the future. These are my constraints; you can constrain the problem in some ways -- we're dealing with the North End, with the artery project. Go out and find characters, figure out how to use photography inventively, figure out how to use sound -- there's somebody else who is working specifically on sound for this project, for instance. It's something that I have an affinity for and want to work with other people on....

These are incredibly collaborative media, and in fact, if you look at a real need for networked communication, movies is one of the places where, traditionally, there has been that need. I think particularly at M.I.T., there is a real need, with programming. The first thing you do is you have your own program -- you know, it works for you, and you've got to do it that way because you've got to be able to think, and you've got to be able to debug, and you've got to learn how your concept matches the lines of code.

In some ways, it's similar to movies. The first thing you've got to do is go in there and mess around with the image, and see what comes out. But at a more sophisticated level, beyond that, you have a group of people thinking about the same problem, and building something exciting to that group. It sometimes gives you a real leg up, in terms of, you are your own audience, and you're sort of feeding eachother. I was the only person who did collaborative projectsat the Media Lab until last year, where I was really aggressively looking for collaborators for a project. And I think that's a very important part of your educational experience

It seems to me there's an awful lot of emphasis on using computers as a tool for manipulating things like video and audio. But are you also working with new forms of media like computer animation and sound, and the virtual reality aspects?

In this, we have a lot of computer animations. I would say that this is a new medium, and the medium is what it is -- it has sound elements, it has picture elements, it's captured in various ways. This is every bit as much of a new medium -- and as hard -- as virtual reality is. Maybe harder, from an expression standpoint, because to really do this well, you not only have to have the computer knowledgable about the story, but you have to have some very strategic and intuitive means of interaction and feedback. And one of the things that I think this lacks right now is that it doesn't know anything about the viewer.

I have more of a comment than a question, but this is in response to your question about how to make a virtual environment compelling. Last semester, I took a class with Nick West, in which we developed spaces for the Yorb. And he began with a question to the class -- he asked everybody, "What was the most exciting thing that happened to you in the past year?" People said everything from being in school to getting jobs, getting married, taking a great trip, whatever. But the idea was that they were very real-life events, and how could you create an environment for those sorts of events to happen inside the [virtual] world. So it wasnt' like you were leaving real life and entering an unreal life, then coming back; it just sort of extended from reality, through the virtual environment and back out again. There was a continuity between the two worlds.

The issue that you bring up that is really critical is that we have this thing, whether we like it or not, called interactive media. Some people at the Media Lab would like to just push that off and say that it's personalized media, personalizable media. That's one of the ideas that we play with a lot.... orchestrate variable stories for all types of screens truth v. fiction