Background and the Award Winning Technology
JE: For the record, can you tell us who you are and something about your role in relationship to this award-winning use of information technology.
LS: I am Lewis Sadler. I am an associate professor of Biomedical Visualization and head of the department of Biomedical Visualization at The University of Illinois in Chicago. I am also an associate professor of Bioengineering and a research scientist at the National Center for Super computer Applications in Urbana. The award which we received in 1989 was for an application that was developed to aid in the search for children who have been missing for a number of years. That was actually NOT the original intent of our research. The original research and work that went into this began with a medical related project trying to come up with some kind of an algorithm that would allow us to predict how children's faces change over time. We were working with kids who were being operated on for severe facial defects, and with the information we gathered we could make some predictions about what the immediate post surgical outcome would be as well as the post surgical outcome several years down the road. Those surgeries are reoperation after reoperation and if we could better predict what the outcome would be after the growth process, hopefully we could make some changes in the surgical procedures. At any rate, data was being collected on how children's faces grow and change when my partner in this effort, Scott Burroughs, was approached by a television crew who was doing a program on missing people. One of the segments was on some kids who had been missing for seven and a half years.
Origin of the Missing Persons Technology
JE: About when was this that they approached Mr. Burroughs?
LS: That was in 1987. They were really just looking for an artist that would make some kind of a "guesstimate" as to what these kids would look like. Obviously the photos that they had would not be very useful in trying to locate the kids after seven years. Scott and I had been colleagues for a long time; he knew about the research that I had been doing in trying to predict how kids' faces change. So, he asked if there wasn't a way to apply some of the information that I had been collecting to do a best guess on what these kids should look like. I provided him with some data; he made some drawings based upon the data that I'd presented him, and the drawings were then broadcast on this national television program. Within about twenty minutes, they had three positive identifications. So that was the beginning- it sort of opened the flood gates. We knew that there were probably better ways of doing it besides an artist sitting there for many many hours trying to make hand-drawn illustrations. We then began the process of trying to computerize the algorithm and make an application that could be used for this purpose.
Research Prior to Facial Development Visualization
JE: Can you tell me something about the research that you had been doing prior to this- the visualization work- were you using computer graphics to do those predictions of how the children might look and change postoperatively?
LS: No, we hadn't been using computers to do that. Most of the work was, at that point, book work. It's an amazing thing to me, in hindsight, that nobody had collected the global changes that were taking place in children's face development. Dentists had looked at the teeth and people that were dealing with cleft lip and palate had looked at the teeth and nose combination, but nobody had really taken the entire head growth into account. A lot of the work really dealt with going through the individual literature and gleaning that particular part of the story, then trying to correlate and assimilate that information into tables of how different parts of kids' faces grow and change over time.
JE: So a few people had done specific limited studies on some of these changes and you pulled together the existing information. Were those studies also connected with various post-operative or post-trauma changes?
LS: There were studies that looked at traumatic changes. We were more interested in what normal growth was, trying to define that first. A lot of the studies had been done beginning in the 1900's. There were a lot of early studies in comparing, for example, farm children's' growth with the growth of kids in the city. There was a lot of interest in nutritional differences.
Research Conerning Development Among Races
JE: Are there significant differences?
LS: There were. There aren't so many anymore, but there were significant differences. A lot of that was post-Industrial Revolution. There was a great interest in kids that were living in cities- going to work at an early age, those kinds of issues- and what kinds of nutritional differences there might be in farm kids who had ready access to higher quality food supply. There was an amount of information available, but almost all of it was on Caucasian children. That caused us no end of grief because, for example, in the state of Illinois if you look at children who have gone missing, roughly fifty percent are black. With a few exceptions in the dental literature, there are very few studies on facial growth in black children. We found a number of studies of Chinese and Japanese children and a number of studies in Europe.
JE: When you say studies of Chinese and Japanese children, you mean studies that were done in this country?
LS: No, they'd been done in those countries. It's a curiosity that we've never worked on the case of a Chinese or Japanese child that's been missing, but we do have information on those growth patterns. We have done a number of cases of black children and used the Caucasian growth algorithms (which we know are wrong). For whatever reason, we've also never had one of those kids recovered. We would like to do some growth studies on those kids. We have some interesting new technologies to allow us to do that, but so far we haven't found funding to do it.
Testing the Process Scientifically
JE: I understand that you did correlation studies on children by working backwards: you took pictures, predicted what the children would look like, and then, since the children were there, you could actually check and see how your algorithms and your theories were working. Is that right?
LS: That's exactly right. We knew early on that we couldn't rely upon whether or not a child was found to see how close we were- to see whether or not the techniques were working. In the first case, there was an uncanny likeness. That could have also just as easily been a fluke that our technique happened to work. I actually collected from friends, relatives, and others a whole series of about fifty kids of whom we had at least school photos. They were primarily within the school years from 5 to 18. I have a few medical art students who know a good deal of anatomy (which helps in this whole process) and are very used to working with facial images. I gave them the growth algorithms and said, "This is a child who is four years old. Age the child to fourteen using this statistical data." They did and they were able to compare what they predicted with what the child actually looked like. We set up several blind studies that way to test the technique. It was about seventy-five percent accurate. Accurate is perhaps the wrong word because there's not a good statistic for what we were doing. So to give us an idea of how well the techniques worked, we looked at the actual measurements of certain anatomical features on the child at the more advanced age and compared them to the predicted measurements. That doesn't necessarily mean they look exactly like it.
JE: This was the growth of specific areas of the face?
LS: Yes, in the anthropological literature, there is a lot of information on standardized facial landmarks. Those have been picked up and used by orthodontists, dentists, all kinds of people. They tended to be a standard way of looking at the face. Knowing where particular anatomical locations moved over time allowed us to then change the facial proportions and reap the face.
Deciding to Use Information Technologies
JE: In a sense, the idea, the hope, came as a problem from the outside. The idea was to see if you could find ways of predicting what children would look like. When did you decide to try to move into information or computer technology to help you with this? You said the earlier cases were done with standard graphical techniques by artists. That, as I understand from what you said, proved to be very difficult. When did you decide that you might be able to make graphics "grow" with anatomical accuracy using the computer?
LS: We did the studies with these sample cases where we had "Before" and "After" pictures so that we could actually check and see what the problems were and where problems occurred. Obviously, this was a labor intensive proposition. Each case example was taking thirty to forty hours. I didn't have, at that point, a good feel for how much of the graphics could be done via the computer. We simply wanted to take the mathematical components out of the artists hands.
JE: So, in a sense, you started with the idea of doing calculational algorithms just to avoid errors.
LS: Exactly. About the same time, there were some major advances going on in...
Connecting With the Electronic Visualization Laboratory
JE: About when was this?
LS: This was about 1987 and 88. There were some new graphics boards that were coming on-line that allowed you to capture very nicely photographic quality images in the computer. At the same time, I fortuitously got connected with another lab at the University of Illinois, the Electronic Visualization Laboratory. All their work is graphics.
JE: Is that Tom Defonte?
LS: Yes- Tom Defonte's group. We got a number of his students involved in the project at that point. They introduced me to Bicubic Interpolation and a number of computer graphics tricks that would allow us to take these anatomical locations and break up the face into a series of polygonal meshes. Then we could take that information and move specific points to new points in space, dragging all of the graphical information with them. It took a bit of massaging before we got all the bugs worked out of the exact method, but that was the basic strategy we pursued. Within about 8 or 9 months we had a program that was working quite well. It has been refined over the years, but that's where we began.
University of Illinois, Chicago Circle
JE: That's what used to be called the Chicago Circle- University of Illinois, Chicago Circle. Are you still in that location?
LS: Boy we hate that name.(Chuckle)
JE: Obviously you do because you got rid of it. Do you call it UIC?
LS: University of Illinois at Chicago- UIC.
JE: Old habits die hard...but are you still at UIC?
LS: Yes.
JE: Because you mentioned a connection with Urbana earlier...
LS: There's actually a bit of a story to the split campus that you're referring to- the old Circle campus. Before I came to UIC, it had tried to join. It's still struggling to make that connection. The Medical campus, which is the one that I'm on, actually predates the University of Illinois. It's a very old campus dating back to the mid-1800's. There was not much of a collaborative effort between the Medical campus and Electrical Engineering (which is, physically, about a mile from us). Tom Defonte and I got hooked up very early on and that collaboration has grown over time.
JE: He's one of the very best in the business. It's hard to know who you would mention as being better than Defonte in that business.
LS: It was an unbelievably fortuitous happenstance. It was not planned- it just happened.
JE: Is it possible to pinpoint more exactly the time- late '87? early '88?- when you thought about getting Defonte and his people in on it?
LS: It was in 1987- in September. The official collaboration began as the result of a grant- that's how I know the specific starting time. We were supporting a number of his students. Money is a great driver of collaboration (chuckle). This put us together in a very real way.
JE: Your studies had shown that certain points of facial geography can be used to develop an algorithm that will tell, say, triangles or individual facets of a polygon where to grow, how much to grow and so on. Then you program that information into a dynamic facial model as a function of time. Is that essentially it?
LS: Yes, that's essentially what it is.
JE: If you start out with a photograph at a younger age, you measure a number of parameters, plug them into an algorithm and the algorithm as a function of time can drive the image?
LS: Right. There are a number of issues. We needed to develop a user interface that was easy to use. We needed to be able to locate, on these photographs, the anatomical positions that we're using to drive the algorithm. The process is that you scan the image in.
LS:One of the initial problems we had to face was that we don't know the size of the child's face that's in the photograph. The photograph can vary in size as well. So, we needed some way of standardizing the photographs. Initially that was a problem, but we did find out, through the anatomical tables that I was able to collect, that the inner corners of the eyes are fixed almost at birth and do not change all throughout your lifetime. They're called the endocantheons. Populations vary by three factors- age, race, and sex. Within populations, the endocantheons did not vary by the age of the person- it did vary in range by the race and sex of the person.
JE: This is the distance between these two points?
LS: That's right.
JE: At what age does that get fixed?
LS: That fixes at about six months. At about three weeks of age (they've pushed that back recently), you first focus and the distance starts to fix. By the time you're six months, your bones have fused and they virtually stay the same throughout your entire life.
JE: So that provides a zero point, an origin, for the face?
LS: That's exactly the way we used it. It does move in real world coordinates- it moves forward over time- but because we were only dealing with frontal view photographs, that actually was not a factor. We were able to use the endocantheon measurements as a fixed standard and normalize all the photographs based upon that standard. We had to have a separate table by race and by sex but NOT by age. That was a fortuitous fact of the way our faces grow. At any rate, we needed to be able to identify those anatomical locations, so we developed an interface that queried the user to go through and pick the points one at a time. Those points were then dumped into a table. They could then be compared, mathematically, to what the norm or the mean for that age group was. The user also has to tell the computer the age of the child that he or she is entering. The computer then uses the ratio- the norm to what that child is at that point. For example, if you have a slightly longer nose than normal at age four, you're going to have a slightly longer nose than normal at age eighteen. We would maintain that ratio. It's really as simple as that. Mathematically it's not a very complex task at all. You tell the computer, "I've taken these measurements"; It figures out the calculations based upon pixel counts. It looks up a table. You've told it the age of the child and the race of the child. You've also entered some demographic information which is held in an archive. You tell the computer what age the child should be now and it does the calculations. It takes about a minute to redraw the face on a 386 computer. From the standpoint of today's technology, it's actually a very simple process and it goes very quickly.
JE: How many times did you check the accuracy of the computer algorithm? Were those the primary studies that you were talking about earlier as far as checking the process- you were actually checking the computer algorithm?
LS: No, the first time when I discussed that we were dealing just simply with the initial algorithms which were being done by hand. We went back and redid all those studies a second time based upon what the computer was now predicting. Of course, now the accuracy went up because it wasn't making the same kinds of mathematical errors that my students were making. All of a sudden now (based simply, again, on the locations of where these parts ended up and where they should have ended up) it was about ninety-eight percent accurate in being able to predict where those anatomical locations should be in that new face.
Dealing with Racial Differences
JE: The problem is that you have to use studies based upon Caucasian children- although you do have Asian studies- and extrapolate to work with the Afro-American kids.
LS: Which we know is not right. We have enough data that we can point to showing that what we're doing is not correct.
JE: Ideally, I trust you'll get the opportunity- you'll get the grant. Then you would want to do for the Afro-American children what you did with the Caucasian kids. That is, use kids who are at an older age and frontal photographs of them at various other stages to give you enough data to get proper algorithms.
LS: Yes. It's actually even more complex than that because once you get into the literature deeply enough you can see, for example, within Caucasian children there is a difference between data collected on children in the United States and data collected on Canadian children or European children. All are Caucasian populations but with different stock that went into the admixture. What that leads us to believe is that we should probably repeat it with the American children- even the Caucasian children- because most of those studies, as I said, go back to the 20's, 30's, and 40's. Kids grow differently now than they did in those days. We also should do it not only for the black population but probably also for the Hispanic population as well.
JE: And you may need different Asian data- there's been a study recently of raw size/ height in Japan. The Japanese are catching up with us rapidly according to what I read.
LS: Changes in nutrition make major changes in the way we grow. There is some very interesting new technology that's available to help with that problem- laser ranging devices. We have one of those in the lab. Within about 17 seconds it takes approximately two hundred and fifty thousand three-dimensional coordinates on the entire head, not just the face. One of our hopes is that we will be able to put those into several schools in the Chicago area where there are students from these different populations. We could very quickly create a cross-sectional database. That is better than the databases that have been collected over the past 100 years or so.
JE: Because you get the data so much faster?
LS: Not only do we get the data so much faster but we get 3-D data. The other data that we have, while taken in three-space, is really straight line linear measurement; it lacks the topology. For other potential applications using the same concepts, it would be much better to have three dimensional data than the two dimensional data that we now have.
JE: At the 1988 Siggraph, I saw what seems to be a similar, or perhaps even the same, technology of using a laser to take 3-D points of a person's face. They went further and plugged the information into a program that actually created a controllable face. It was then shown in the evening sessions of the Atlanta SIG-graph meeting. They had an algorithm that was able to control the face according to various parameters and have it talk with some accuracy. It was amazingly lifelike. In one of the sessions they had not only the face talking and looking about but the fellow who had supplied the face down below it acting as master of ceremonies. Is some of that technology embedded in the work that you've been doing?
LS: Yes. In 1988, we acquired the same piece of technology that you're referring to- it's made by the Cyberware corporation in California. We acquired it to put into schools to do the data collection. It turned out to be incredibly difficult to get access to the school system, especially in Chicago.
Getting Data from the School System
JE: So your hope is to get data on children in the school system using these laser ranging devices?
LS: Yes. The process of using one of those takes about seventeen seconds so it is at least theoretically possible to set one of those up in a school and go through a class of kids in really just a matter of a few minutes. One of the largest previous studies ever done, was done in Canada based on fifty girls and fifty boys. It took basically two people their entire lifetimes to collect the data. My feeling is with this new technology we can probably do the same sort of thing in a day or so.
JE: So the problem is getting access to the school system in Chicago.
LS: Yes.
The Purpose of Additional Studies
JE: So with this new technology, you'll be moving forward in both the scientific front, that is, you'll be able to get better databases of information on the growth of children's faces. Will that affect this project of predicting growth in children that have gone missing? If you're getting ninety-eight percent accuracy already, do you think you can get higher than that with additional studies?
LS: We're not interested in doing the additional studies to gain greater accuracy with the Caucasian population. It's really to get a database that will allow us to do a credible job with minority populations. It certainly does fit into other facial imaging problems that we're working on with kids but as far as this particular application of trying to find kids that have gone missing, we really need to add to the databases on minorities.
JE: What were some of the major obstacles, some of the major frustrations, in developing these methodologies both in terms of the computer technology and the data reduction? Can you think of things that were particularly troublesome as you were doing this or was it a fairly even playing field with everything being as bad as everything else?
LS: It worked well and it had its problems all at the same time. One of the things that began at that time, has continued through until today and has, if anything, gotten worse is that we brought in a lot of domain specialists to work together. One of the things we immediately discovered is that engineers and medical people don't share a common language, common custom, common starting work time, common anything. There were people problems- people management problems. The technology worked itself out quite nicely once we could get people to cooperate and work together. Those were the biggest problems initially. We have continued to add different kinds of domain scientists to our research efforts and now we have artists, medical people, engineers, psychologists, statisticians, and mathematicians all on staff. Trying to get all those people to come at a problem from the same direction can be difficult, but at the same time it's a very stimulating environment. Things that one group wouldn't have even considered, the other group thinks is the right way to go. That becomes an interesting management problem- which is what I deal with, mostly.
Motivation to Use Information Technology
JE: Prior to the time that you were pointing this to a technological basis- prior to using the computer- you said that you had two graduate students that were trained in the hand drawing technique. Do I understand correctly from the submission that the combined experience they had and you had was a major motor for your decision to move towards a computer-based approach?
LS: Yes, when we first started this whole process, when we had the first success, the phone rang off the wall. Fortunately, it was my partner's, Scott Burrough's, name and face that was on television. There were literally hundreds of parents from all over the world that began calling wanting to gain access to the same kind of help that he had given to the people on the television program. We immediately began to train some others to do this but it was apparent that if that was the only way we were going to be able to do it, it was going to drive everybody crazy. Basically, it was people working in their spare time. Parents were frustrated that they couldn't get the assistance that they needed and deserved; law enforcement wanted things to happen right away. Frankly, it just wasn't happening right away. We were very hopeful, although certainly very naive at that point, that in some way information technology would solve all these problems. It turned out that it did, but it was a longer, harder road then we had at first envisioned.
Initial Success of the Technology
JE: You had immediate initial success with the manual system, so you knew immediately that you were on to something and that your ideas were valid and essentially successful. When did you feel that the second approach, the approach using information technology, was successful? When did you feel that you were able to duplicate with the faster technology what you had done successfully with manual technology?
LS: By around Christmas of 1988, we had an operating system that we had done some verification studies on and felt very good about. We began to actually use it- apply it to some cases. It was almost immediately successful again. They're now being done by people in Washington who operate the system because we did not want to get into the service business- we were a research lab. But over the period of time that we were doing the cases within our labs, we ended up doing about 150 cases. We had drawn up parameters that we would not do children who'd been missing for less than three years because their photos were probably still useful (except in the case of very young infants who were changing really rapidly). Of the 150 cases that qualified and that we ended up doing we recovered 33 kids. I guess the thing that I should point out to people is that you could have an absolutely perfect image of that child at that age and if somebody who knew where they were didn't see it, the kid wouldn't be found. We don't know how accurate we were, but it was incredibly successful.
JE: Clearly, information technology is critical to this effort because it allows much higher production. Is that a fair statement?
LS: Yes.
JE: In order to do the same thing with the older technology you would have to have a huge battery of artists working by hand, at probably a much lower rate of production due to the mistake patterns that you discovered in your earlier work. Besides your partner and Tom Defonte, who were some of the other key people involved with the project? You said that the actual operation of the program had moved here to Washington.
LS: Right. There is a National Center for Missing and Exploited Children here in the Washington area. They're involved in manufacturing these now for law enforcement.
JE: You're continuing to do research in this area.
LS: Yes.
JE: Has the program, the algorithms- have they changed very much since '89, or has it essentially been a gradual evolutionary kind of development?
LS: There has been some changes in the algorithm, not in the approach. There have been some additional new algorithms. As you are probably aware, within the last year or so, "morphing" became an interesting concept that everybody has used. It certainly has been a commercial success on television. Basically our algorithms are morphing algorithms and while we did them earlier, there are now better algorithms then there were back in those days so we've adopted some of those newer algorithms. Conceptually it's the same as it was before.
JE: Looking back at it, in general, what are you most proud of about the work? How do you view it? There are the obvious enormous satisfactions of actually building something that has brought kids back, which is fantastic.
LS: It really is a very personal kind of affair. Especially in the early days when we were first applying it. We were in contact with a lot of the parents and we actually got to know individuals, their stories, and what their problems were. It's an emotional roller coaster. As a parent of four kids I can't really put myself in the position these parents are in but at a certain level I can really relate to it. It changed a lot of my habits with my own kids, I can tell you. When there was a success, it was a very personal success and very often, Scott or I went to the reunion to be there when the kids met their parents for the first time after seven or eight years. I really can't put into words the emotions of those reunions. I still hear from a lot of those people.
JE: Were most of the cases that were successfully concluded due to parental kidnappings- the parents' split-ups?
LS: Yes. All of the successful cases were parental kidnappings, which has its own frustrations. Obviously, this kind of activity isn't an act of love on the part of the abducting parent. What also came out in recent years, and was certainly true of the cases we were involved in, is that most of those kids were being abducted because they were being abused by the abducting parent. They either didn't want it to come out in court or they wanted to continue the abuse. The assumption by police is that it's a family matter- a family squabble. That had its frustrations too- trying to get cooperation from law enforcement. Many times it was difficult and these kids needed a lot of help.
Winning the Computerworld Award
JE: On a different track, what did winning the Computerworld Smithsonian Award mean to you and your colleagues aside from the personal satisfaction that comes with a recognition in this area? Were there any other byproducts, so to speak?
LS: We labor- like most others in universities- under the whims of the administration at times. Certainly, winning a major award such as this brought a lot of accolades and stature. It verified a lot of what we were doing and got us a lot of cooperation from the administration. At a lot of different levels it has paid off and been very rewarding for us.
JE: Are there things that you might have wanted to do differently? Are there lingering frustrations or regrets with the program or anything that strikes you as a useful lesson that you learned of things to be avoided?
LS: No, I think by in large the entire experience has been a very positive one. I can't really think of any negative or down side to any of those activities.
JE: Is there anything you'd like to add- something that suddenly occurred to you or something I didn't ask that you'd like to say before I ask my own questions? Anything we've overlooked or haven't covered?
LS: I still get a lot of press related calls about this- especially from people in foreign countries. Through reprinting of something (I never know exactly what) I'll get a call from Australia or Germany or Switzerland, recently. They of course have all the same problems. This is not a problem confined to the United Stated; it is a societal problem everywhere. I have the frustration that we did not want to become a service. We don't really have any way of handling foreign requests because the National Center is national to the United States. So we do end up doing cases occasionally. The heart wrenching stories are hard to say "no" to, walk away, and sleep comfortably at night. The frustration is that there has not been a good way to take this technology and transfer it to foreign countries.
JE: When you were in the process of development (from '87 through '88, '89) was this primarily a matter of getting together and talking about the manual process and the research with Defonte and his students and then their translating that into their particular technology? They tried things and showed them to you and back and forth- is that sort of the way it tended to be? How did it work?
LS: There was that component of it. There was a learning curve on both sides. Defonte likes to say that if you deal with a carbon atom, it's not engineering. So, from the side of his students, they really knew nothing about where we were coming from or how to look at it from our side. We were at least as ignorant going in the other direction. We would try to explain it and send them off and let them work on it. At the same time, we had to not only explain it, but also explain conceptually what was behind it, why it was important that we do things in a certain way, and why landmarks were important. They thought we had these "crazy Latin names" and wanted to know "what do they mean?" There was a great deal of education going on from both sides. I wouldn't have known a bicubic spline if it hit me in the face, so they were doing an equal amount of teaching from their side. Fortunately everybody kept an open mind and we came together pretty rapidly.
Changes in Biomedical and Art Departments
JE: Traditionally departments of Biomedical Illustration have been based on manual artistry. Aside from the work that you do has the department as a whole gotten involved in computer aided drawings, paintings, and design? Has it translated manual skills into the editable art of the computer?
LS: Yes. The illustration component at the University of Illinois dates back to 1913. The school has been there since 1921- the second school in the country. When I became department head in 1987, we changed the name to Biomedical Visualization.
JE: Is your background in classical art?
LS: My background is in the fine arts with a very strong science background. As a matter of fact, I'm still going to school; I'm going to school in anthropology now, but that's a different story. One of the things that we did in converting the name was we made a very concentrated effort to go away from the traditional techniques. Everything almost that we do now has to do with information technology. There has been a convergence of the printing industry, the television industry, the motion picture industry, and the computer industry. That convergence, when you interface that with medicine, is really where we live. Hence the new name and the new thrust of the department.
Association with Super Computer Laboratories
JE: The other associations that you described with the super computer laboratories, how does that work?
LS: We're a remote laboratory for Larry Smires' operation in Urbana. They suffer at the Super computer Center for not having a medical school. There is a branch medical school in Urbana but the main medical school campus in is Chicago. Larry, for many years, wanted to have a medical emphasis on some of the application areas and had not been able to get it organized. Very recently our department and our lab became Larry Smires' lab for super computer applications related to medicine.
JE: I want to thank you for sitting and talking with me.
LS: Thank you.