Telerobotic Applications
       By
       
      Brook Votaw
        
      Telerobotics is a very important and quickly expanding field in the face of faster processors, new algorithms, and higher expectations. There are many important telerobotics applications in use today, ranging from space exploration, to biomedical applications, to hazardous area exploration.  The leader in this field is NASA.  As NASA looks to more distant stars, there is a need for more advanced technology.  In investigating this new technology, NASA is developing commercializable applications which rely upon space telerobotics technologies  In so doing NASA hopes to improve the national economic competitiveness of the United States and improve the technology transfer efforts.  Applications here on Earth serve the dual purpose of providing the means for test and demonstration in realistic operational test environments.  The well understood and easily accessible environments on Earth provide a myriad of possibilities of problems that might occur in space.  One of the most useful environments for several of the space applications is undersea environments.  As a result, many of the technologies have been put to use underwater.

      The NASA space telerobotics program encompasses three basic areas including, remote operations on planetary and lunar surfaces, robotics tending of scientific payloads, and satellite and space system servicing.  General requires for these applications include automation to reduce crew interaction, hazardous materials handling, collision avoidance capabilities, high levels of autonomy, and finally robotic vision and perceptions systems.   Some of the obstacles NASA faces are multi-arm coordinated cooperative control, uncertain knowledge of operating environment, computation or communications induced time delays, high data rate science payloads, low command rates, and there will always be an issue of human safety considerations.

      Exploring distant planets is perhaps the most well known of NASA's programs as a result of the Mars Pathfinder mission.  Teleoperators with the appropriate functionality demonstrate high levels of local autonomy, i.e. limited ground command intervention because of the vast distances which data will have to travel, with which they can perform local navigation, identify areas of potential scientific interest, regulate on-board resources, and schedule activities.  It was seen in the case of the Mars Sojourner that simple teleoperation commands took several minutes to arrive on the Sojourner.

      The robotic tending of scientific payloads will be applicable for use pressurized living spaces such as space stations and undersea environments.  This application will basically off-load requirements of intensive astronaut, or aquanaut, maintenance and permit operation of the payloads during periods when the astronauts, or aquanauts, may not be present.

      Finally NASA envisions an eventual application to satellite and space system servicing.  Two different types of telerobots would be useful for such an application; namely free -flying and platform attached telerobots.  In this case, something called virtual reality telepresence, which will be discussed in detail later in this paper, would be useful so that the environment could be more realistically reproduced and a "repairmen" could virtually work directly on the malfunctioning system in a simulated environment.


        What Is a Telerobot?

      In order to have a better understanding of today's field of robotics, one needs to understand the three groups that encompass robotics: terrestrial robots, teleoperators, and telerobots. They might all sound similar, but differentiating among them is key to the progress in the field. The most commonly known robot is the terrestrial robot. These are typically limited to industrial manufacturing. They are heavy, rigid devices, which use preprogrammed control in precisely structured environments to perform single highly repetitive tasks. These are generally not useful for the type of research this paper will entail. However, one positive aspect of these robots that should be noted is that they require very little man-hours to operate and perform tasks.

      The next type of robot is the teleoperator, which is operated by direct manual control of the manipulators by humans. This limits the teleoperator to real-time manual control and makes for a large investment in man-hours. But the definite advantage over terrestrial robots for the purpose of this paper is that they can perform non repetitive tasks. Teleoperators were originally developed for use in undersea and nuclear environments, and in general semi structured environments which are hostile to humans. Obviously, it is difficult to preprogram reactions to all of the contingencies that may occur during a task because the environment cannot be sufficiently controlled to permit robot operation, so an operator having direct control over the robot is an advantage over purely autonomous robots. The sort of tasks in which sufficient manual dexterity, sensing, and artificial intelligence is not yet available in robots, so tasks in which a human operator is warranted because of the cost of a possible failure of a robot is too high.

      Finally, there are telerobots which are a combination of teleoperation and automated control. Telerobots are the answer to many of our technological difficulties today. The telerobot is more capable than either a robot or a teleoperator as it performs a larger class of tasks than can be accomplished by either. Thus the advantages of both are magnified and the limitations of both are minimized, as the right cooperative mix of human and automated agents for any given set of mission goals can be found. Telerobots are useful in semi- structured to unstructured environments. They can perform non repetitive tasks with an incomplete model of the task environment. One of the main difficulties encountered with telerobots is the variable time delay between operator and manipulator. For instance, it takes longer for instructions to travel from Earth to Mars, than it would for instructions to travel between two rooms in the same building. In the time that it takes instructions to reach Mars, there could have been a change in environment.


      HAZBOT
       

      In the early 1990s, Jet Propulsion Laboratory (JPL) in Pasadena, California began a project in Telerobotics as part of its Emergency Response Robotics Program. The primary objective of the design of HAZBOT was to allow safe exploration of potentially dangerous sites and handling of hazardous materials in conjunction with the Hazardous Materials team HAZMAT. JPL started with a commercially available REMOTEC ANDROS V robot, and added many important features to this robot with input from the JPL fire department for operation in combustible environments. For instance the special chassis and manipulator design allow all areas containing electronics and motors to be positively pressurized in the case of HAZBOT entering a combustible site. In addition, a six-degree-of-freedom manipulator with a 30 pound lift capability allows the robot to perform a variety of tasks including the unlocking and opening of doors. The manipulator also incorporates a parallel jaw gripper with a 60 pound squeeze force and a gas detector to aid in material identification. The on-board computer system, which JPL developed, controls the manipulator, track drives, and camera positioning. In addition it processes information from the temperature pressure and chemical sensors which allows the robot to provide vital information on spill location, magnitude, material type and concentration so a well prepared response team can safely enter the site. Two video cameras, one located on the gripper and the other on a movable pan/tilt platform, provide feedback to the system operator who can then control the robot from an operator control station from a distance away.

      Richard Welch, the task manager of Emergency Response Robotics at JPL remarked on additional modifications, which would add to the effectiveness of HAZBOT. A simple force sensor in the gripper would prevent the robot from breaking what it picks up. Stereo vision would correct manipulations problems caused by lack of depth perception. And along the lines of my project, there are plans to further improve the performance by implementing a Radio Frequency link to replace the tether which currently limits HAZBOT’s maneuverability. This would allow HAZBOT to enter a site while his human counterparts were safely located some distance away. The creators of HAZBOT envision additional commercial uses for HAZBOT in law enforcement, mining, bomb squads, and fire fighting (Edmonds, 1993).
       

      DANTE II
       

      In 1994, a walking robot named Dante II explored the Alaskan volcano, Mt. Spurr, on its own for nearly a week. The principal objective of Dante II was to develop and demonstrate technologies which could lead to solutions for robotic exploration of rugged terrain on the Moon and planets. The secondary objective of Dante II was to obtain scientific data on Mt. Spurr, as it had never been explored before.

      Dante II was not self deploying, which would be a hindrance in potentially dangerous situations and would make space missions impossible.  Instead, Dante was tethered at a site on the edge of the volcano during its repelling decent.  This tether created maneuvering problems because during operation there was a constant issue of tangling Dante up.  In addition to this tether, there was a satellite band communications antenna at this site used to send video and data between the robot and remote base station.

      The operators were located about 120 km from the volcano because of the volcano's instability, which allowed for slightly more realistic (i.e. limited) space communications situations. Transmission of large packets of information (teleoperation commands, for instance) caused delays of 30 seconds or more. This made autonimity extremely important for this mission. Mt. Spurr was an ideal location to test close to real life conditions on the moon and other planets as the terrain was very rugged and the communications bandwidth problems.

      The expedition itself was a success. Dante repelled into the crater of the volcano taking data samples as it went. It used a 3-D elevation map, enlisting a laser, to help the robot and humans guide the robot and minimize the number of retreats the robot had to perform as a result of obstacles. Humans chose a particular path for the robot, and the robot found the safest and most efficient way of getting there.  The robot was able to operate five days without any human presence on the volcano and achieved all of the mission objectives.  On the way out of the volcano, toward the end of Dante’s mission, the robot fell on its side and had to be recovered by scientists.

      Dante II used a combination of supervised autonomous control and remote teleoperation which the creators called contextual operation.  There were five different contexts of operation ranging from direct actuator control, to enhanced teleoperation, to full autonomous control, with the human operator being able to instantly switch between contexts. Each context defined a certain mode of human robot interaction and enabled corresponding variables to be altered by the operator.  For example, direct actuator control involved the operator providing every command for every variable in order for the robot to continue.  Whereas, in behavioral mode, the robot used data from force sensors on the legs and inclinometers in the body to generate a motion sequence that safely adapted to the terrain.  And finally  full autonomy allowed Dante II to perceive a path, and avoid obstacles along the way.  The makers of Dante II found that programming layers of operator control capabilities in this fashion was ideal for the unexpected conditions that might arise in robot exploration.  John E Bares, from the Robotics Institute at Carnegie Mellon University envisions an ideal robot that will operate autonomously until it encounters an impassable situation, in which it will be able to ask humans for further assistance, thus reducing the need to continuously monitor the state of the robot.

      Telepresence
      When teleoperation took effect, the robot's progress slowed because of all the sensory information  from sensors, laser, and cameras that the humans had to deal with before assigning any particular motion to the robot.  Therefore, it was important to the creators of Dante II that there be a graphical representation of all of the sensory information, in order to get an accurate picture of the terrain Dante was covering, and the state of the robot itself.  By creating a graphical representation, the operators did not have to manually go through every bit of information Dante sent back from the crater and had a better chance of successfully maneuvering the robot. From this perspective, a robot like Dante II is ideal for exploration, since its sensory information is provided much more quickly and in depth than any human mapping could do.  In general, this technique of displaying all the sensory information is called telepresence and has been implemented in many such applications.
       
       
      Robotic Assisted Microsurgery

      Robotic Assisted Microsurgery (RAMS) is a highly useful application of teleoperation created by NASA's Jet Propulsion Laboratory, in Pasadena in 1994.   It allows surgeons to perform dexterous, detailed procedures at 2-to-3 times smaller resolutions than achievable by the unaided hand.  It achieves this by enhancing the manual positioning and tracking in the face of myoclonic (irregular involuntary contraction of a muscle) jerks and tremors that limit most surgeons' fine motion skills. So the surgeon's hand motions are down scaled.

      RAMS utilizes a six degree of freedom "slave" robot, which moves under either teleoperative or shared computer programmable controls of a surgeon's "master" hand positioning device.  Primarily teleoperation is enlisted which includes task frame reference manual force feedback and textural feedback.  The operator shares automated control of robot trajectories.  As advances in areas like telepresence displays, dexterous teleoperative controls, small manipulators, and graphics based planning and training tools, surgeon's capabilities will be increased and the results of these highly specific outcomes will be improved.  As a result it is envisioned that patient recovery times will be minimized and costs of health care will be reduced.  In addition, new procedures for the brain, eye, ear, nose, throat, face, and hand will be introduced as JPL continues to improve on this device in conjunction with leading micro surgeons.

      Mars Sojourner
      The Mars Pathfinder spacecraft was Launched from Kennedy Space Center on December 4, 1996.  It took the Pathfinder about seven months to get to Mars.  Once on Mars, the Pathfinder deployed a telerobotic rover named Sojourner.  Sojourner was be humanity's first attempt to operate a remote control vehicle on another planet, its primary function being to demonstrate that small rovers can actually operate on Mars. The constraint of the mission was that there was only a once-per-sol (a sol being a day on Mars) opportunity for command and telemetry transmissions between the lander and earth operators. Communications with the rover were not done in real-time because of the approximately 11 minute light-time delay in receiving the signals.  Therefore Sojourner needed to be able to carry out its mission with a form of supervised autonomous control where move commands were sent to the rover ahead of time and Sojourner then navigated and safely traversed the rugged Mars terrain to these locations on its own.

      In order to further justify such a mission, the Sojourner was responsible for conducting a series of experiments which validated various technologies for an autonomous mobile vehicle.  Some of these experiments included deploying an Alpha Proton X-ray Spectrometer (APXS) on rocks and soil to determine the elemental composition and constrain the mineralogy of rocks and other surface materials present at the landing site; imaging the lander, using IMP, as part of an engineering assessment after landing; Mars terrain reconstruction from imagery; basic soil mechanics; dead reckoning sensor performance and path reconstruction; and testing ultra high frequency (UHF) link effectiveness graphed as a function of location by logging data transfer errors.

      The Sojourner telecommunications system was a two-way wireless UHF radio link between the Lander and the Rover. This link was used to send commands from Earth to the Rover and receive images and data from the Rover.  The Rover communications were not done directly because the Micro rover radio had a signal range similar to a walkie- talkie.  The rover's wheels and suspension used a unique system with no springs. Rather, its joints rotated and conformed to the contour of the ground, providing the greatest degree of stability for traversing rocky, uneven surfaces.
       


      Remote Teleoperation of BRAK and SHRIMP
      This interest and research in Teleoperations has resulted in a project of my own, involving some equipment that we have lying around here in the School of Engineering at the University of the Pacific.   As a researcher in the Computer Vision and Robotics Group (CVRG) under the supervision of Dr. Stark and Dr. Hughes, I familiarized myself with BRAK (Basic Robotic Architecture Kit) and SHRIMP (Stereo Head Research Image Processor).  I envisioned a need for these to be remote controlled to increase the mobility and use of the robots, and to increase my understanding of teleoperation.  For this project BRAK and SHRIMP operate as a unit, with BRAK being the means of transportation and SHRIMP being the operator's means of identifying BRAK’s environment through a pair of CCD cameras behaving as "eyes".  Radio Frequency (RF) is the means of remote communication.  The scope of this project includes a thorough understanding of communications between serial ports, data transmission, and game-port communications to the motors that drive BRAK and SHRIMP. Complete teleoperation capabilities willm be completed by April of 2000.

      Specifications:
       

          General
        • Use of Radio Frequency for data transmission
        • Robot must be able to operate with no external links/cords
        • Robot must time out if link broken
        • Robot must have ability to recover and continue operating if the link comes back up
        • Motors must disengage if link broken
        • Check-Sum to validate data
        BRAK
        • Necessary game pad commands
          • Two speeds
          • Forward /Reverse –stopping before changing direction
          • Wheel turn commands to provide understanding of wheel position
          • Kill
           
        SHRIMP
        • Necessary game pad commands
          • Three degrees of freedom
          • Right camera/left camera / stereo vision
          • Capture image
        • Video Image sent back in real time
        Status Screen
        • Link status
        • Game pad position status
        • Right camera/left camera/stereo vision
        • Speed
        • Picture mode
        • Degree of freedom
        Project Options
        • Graphical Interface to affect control of BRAK from terminal
        • Make the RF Module compact, and bring 12V power from CPU to avoid the use of additional 9V battery
        • Additional modes for game pad control
        • Include sensor readings in teleoperational abilities
       
      Overview

      In order to achieve remote teleoperation of SHRIMP and BRAK using RF, we need to be able to send commands to BRAK and SHRIMP from a remote location. Figure 1, below, shows a block diagram of the various transmissions that need to be performed for teleoperation.  Implementation of real-time video trasmission from SHRIMP to the operator control station will be necessary for appropriate guidance of the unit from a remote location. The operator will need to receive status back from BRAK and SHRIMP in order to react more appropriately to various situations.  A check-sum will be included in this status to confirm that the data that BRAK received was the same data that the terminal sent. 

       

      Figure 1. Block Diagram of Teleoperating SHRIMP and BRAK. The operator sends commands to SHRIMP and BRAK, and receives video and status back from SHRIMP and BRAK.
       
      The off-the-shelf Radio/Transmitter Module that has been chosen for this project can be see in Figure 2.  It has a frequency range of 902 MHz to 928 MHz. To specify a particular frequency with which to send data a particular channel on the receiving and transmitting end for one direction is chosen.  A different frequency is chosen for the opposite direction. The specifications for the module indicate that the maximum range for the module is about a quarter of a mile, maximum. This will come into play when we try operating BRAK from a different room. Currently the module works within a room. In addition the modules transmit at 50kbps (kilobits per second). This is important to know for choosing a "packet" size to send between BRAK and the operator station. The "packets" we will be sending for this application will not exceed on the order of tens of bits, which translates to a fraction of a second. Thus, there should not be problems with real time control. The Modules use a surface acoustic wave filter, which attenuates RF energy out of range of the specific frequency we have chosen. This filter was chosen over other filters because it provides higher out-of-band attenuation than other filters.
        
      Figure2. RF Transmitter/Receiver Module.

       

      Implementation
       
      In general, the remote teleoperation system being developed will consist of three main procedures: get data from serial-port, translate the data and process, and send the data. The packets of data that will be racing across the RF lines will look similar to this:
      **s 0 194 85 b 1 98 98#

      Each character in this string has an important function. The "**" is what the system is looking for to indicate the beginning of a string of data. Without this important indicator, the system assumes it is reading garbage and will not process the data. The next character, an "s", will tell BRAK that the next variables, until "b", are for SHRIMP. BRAK will then send this information to SHRIMP. The "b" lets BRAK know that the next variables are commands for him. The "#" at the end of the string lets the system know that this is the end of the data. The system is expecting to see this "#" before it fills up its buffer with data. So, we can tell if the link has broken while we were receiving a packet of data, if we do not see "#" before the buffer fills up. Finally, it is important to understand what the numbers in the string represent. The first number, a single digit, will translate to a button being pressed: "0" for no buttons, "1" for button A, "2" for button B, and "3" for both buttons. The second number ranges from about 0 to about 195 and corresponds to the x position on the game pad. For example, 194 might represent turn right. The third number corresponds to the y position on the game pad, and ranges from about 0 to about 195; 85 would indicate that we the robot should not translate vertically.

      Now we begin our discussion on what reading from the serial port entails. First, of course, the serial port needs to be opened and characters need to be read, individually, from the port. The question arises: how do we know if the information is data or garbage? From the previous discussion on what a "packet" of data is, we know that the system will look for "**" as the beginning of the data packet and "#" as the end of the data packet. An additional indicator will be added to this packet to better ensure that the data in that packet is actually the data that was sent. This indicator would be a "check-sum", as mentioned in the introduction of the project.  A check-sum is the value represented by adding up all the data being sent.  If we include this value in the packet, the system at either end can compare the sum it was sent with the sum of the data it recieved to determine whether or not the data was corrupted.  If the sums match, the processing the data will ensue. If not, the corrupted packet will be discarded and a flag will be sent back to warn the system on the sending end that the data was corrupted.

      It is important that the system be aware of the link status at all times. A time out protocol has been developed to react to certain conditions being met during system transmission. These conditions include:

      • If the serial port does not open
      • If the system does not read a character at serial port
      • If the system reads characters, but finds no **. In this case, we can assume the information at the serial port must be garbage, since the system expects the appropriate data packet.
      • If the system sees too many **, there must be a problem with the link, since we only expect to see two for each packet.
      • If the buffer is full, and the system has not seen a #, then the link must have broken while receiving a packet, and garbage has been introduced to the data.
      When any of these conditions are satisfied, a time out process begins. However, if the link recovers before the system reaches the end of its time out procedure, the system is able to recover and begin transmission again.

      A system flowchart for the responsibilities of the operator control system can be seen in figure 3. The operator control station should be constantly checking to see if BRAK is "alive".  If he is "alive", the operator control station will be receiving a constant packet of information, indicating that the link is up. As long as the operator control station knows that BRAK alive, it will read game pad data from the game port. The kind of data it is looking for was discussed in the discussion of what entails a "packet." The various game pad parameters are button status (an integer from 0-3), and X and Y position status in the form of an integer from 0-190. In addition the operator station will be reading SHRIMP commands from an additional game pad, and possibly a graphical interface. It should translates the totality of that information to a string in the form that was seen in the "packet" discussion and sent across the RF line to BRAK. In addition, the operator station will display various parameters of the system. Ideally, this display will be an interface where the operator can switch modes by clicking on button. This way we can get around the limitations of having only a few buttons on each game pad, with several more parameters than buttons.
       

      Figure 3. Software flow chart for the operator control station

       
      The graphical interface will include information such as whether the link is up or down and which speed setting BRAK is set at. A constant display of game pad position status will be useful to ensure that we are constantly updating, and that BRAK and SHRIMP are processing the same data that the operator is sending. In addition, SHRIMP has the capability of either taking continuous video images, or capturing a single image to be processed. On the graphical interface it would be useful to change between picture and video mode instead of setting our limited buttons to perform such a task.  Using his "eyes", SHRIMP can take video or pictures from either camera, or from both in quick succession to achieve a stereo image. The graphical interface should have the capability to switch this parameter. Finally, SHRIMP has three degrees of freedom, as seen in figure 4. Using the graphical interface, we can excercise control over which degree of freedom we would like to control at any given time. The graphical interface has yet to be implemented.
       

      Figure 4.  The three degrees of freedom (DOF) of SHRIMP.  DOF 1 represents the turning of SHRIMP's "head" from side to side.  DOF 2 represents the movement of SHRIMP's "head" up and down.  DOF 3 represents eye convergence and divergence.

      BRAK’s processing duties are outlined in Figure 5. These duties are allocated to BRAK, not SHRIMP, because BRAK is responsible for processing all the data for the unit, except for the actual execution of SHRIMP’s motor commands. Regardless of what BRAK is performing, he needs to constantly be sending a signal to the operator station to let the operator know that the link is up and things are behaving properly. BRAK does this by repeatedly sending an empty string with "**#". Eventually, there will be information sent back in this string, such as a check-sum confirmation, which the operator station will then need to act on. BRAK reads in the string sent by the operator station from his serial port. He then translates the information into commands to be sent to SHRIMP and commands that he needs to process corresponding to speed and position of his motors. We saw in the definition of a packet and in the discussion of operator station chores that BRAK is expecting three integer values representing a button, an X position and a Y position. BRAK then takes these values and translates them from integer values into motor commands. If the link is broken the motors will stop, but can be re-engaged if the link is recovered. Finally, status needs to be sent to the operator to give the system more control and precision. The check-sum discussed earlier will be valuable in this case. If BRAK finds that the data that was sent does not match up to the data that was received, he will then send an interrupt signal to the operator station so that the system can find the source of the error.
       

        Figure 5. Software flow chart for the BRAK side of the project

      Initially, to test that the serial port was opening properly and characters were being read from it, a program called Minicom, which can be found in UNIX was used. Two systems were connected using a null modem. A null modem is basically a two-way physical link that sends and receives in a similar fashion to that which RF transmission is accomplished, with the exception that obstacles such as receiving garbage are avoided. Through this null modem we sent information from one system to another, and using Minicom, we could validate that information being received from one was what was being sent from the other.

      Another important step in confirming that the process was indeed responding as expected was to test the time out structure. That is, when the program was up and running on two systems, the link was broken in various ways.  These tests included quitting the program on one end, unplugging the link, and sending garbage. In all cases, it was important to note how long it took the link to go down, and if the link was able to recover if the problem was corrected (i.e. plugging the link back up, restarting the program on the side we quit out of, or sending valid information). When it was confirmed that the time out structure was operating as expected, the next step was to begin sending real data (i.e. game pad commands) to BRAK and processing it. A problem encountered was that if the link went down while BRAK was moving, he continued to move for an unacceptable period of time, until the program timed out completely. The only way to stop BRAK was physically disengage his motors using a kill switch included in his architecture. To remedy this problem, a drive_stop() command was added to the time out procedure, and the time to time out was reduced. After this fix, if the link went down, BRAK would come to a halt in a fraction of a second. If the link came back up, BRAK’s motors could re-engage and begin reacting to commands being sent.

      In conclusion, endeavors to create a mobile teleoperated robot out of BRAK have been successful thus far.  Next steps will include bringing SHRIMP online, creating a graphical interface to increase capabilities and the precision of the communication system.  It is important to have a means for the systems to know if the data being received is correct.  The capability of knowing and responding to link status is an integral part of creating any teleoperated device.
       

      References

       

      HAZMAT:
      "Applying Robotics to HAZMAT," by G. Edmonds and R. Welch, Proceedings of the Forth National Technology. Edmonds, Proceedings of the Forth National Technology Transfer Conference - NASA Technology 2003, Vol. 2, pp 279-287, Anaheim, CA, December 1993.

      DANTE II:
      http://img.arc.nasa.gov/Dante/impages/at-cmu.html
       
      Robot Assisted Microsurgery (RAMS):
      P. S. Schenker, H. Das, T. Ohm, "Development of a Master-Slave Manipulator for Dexterity-Enhanced Microsurgery" Telemanipulator & Telepresence Technologies, SPIE Proc. 2351, Boston, MA, October (1994).

      http://helios.jpl.nasa.gov/tasks/rams/icar97.pdf
      http://helios.jpl.nasa.gov/accomplishments/rams-robot/ramsTB.html