Robots take ultrasound to the fourth dimension

  • The continual search for more and more applications of machines and robots in medicine is intensifying as technology advances rapidly. Sophisticated robots are capable of movements similar to, or even exceeding, the suppleness and sensitivity of a human arm. Ultrasonic imaging techniques have also evolved greatly in the last decade. Professor Floris Ernst and his collaborators at the University of Lübeck research methods to combine the mechanical capabilities of robots with novel ultrasonic imaging and computing power to create automated medical systems.

    Professor Floris Ernst is a professor for Medical Robotics at the Institute for Robotics and Cognitive Systems at the University of Lübeck, Germany. The institute is a leader in German research on robotic medical navigation systems. Professor Ernst’s latest project is looking to take novel 3D ultrasound-imaging technology to the next level, developing a robot that can be used for automated ultrasound; for example to monitor a tumour which is being irradiated at the same time by another robot.

    Tissue in motion
    Our soft tissues, such as our bowels or lungs, constantly move around slightly due to their physiological function and position in our body. Some motions are predictable, such as breathing, while others are not, such as spontaneous bowel movements. This often poses a problem whenever a doctor wants to examine these areas using imaging methods.

    Our breathing motions can be both a hindrance and a help during medical exams. Our lungs, rib cage and any other tissue in their vicinity move due to respiration. In addition, our ribs block imaging rays, concealing what lies underneath. Here, directed breathing can be an advantage because it moves the ribs out of the way. This is useful for relatively simple examinations.

    Professor Ernst is developing an ultrasound robot that can monitor a tumour during irradiationQuote_brain

    In tumour treatment, tissue motion is accounted for by safety margins, i.e., healthy tissue surrounding the tumour that is included in any treatment. The more the tissue area moves, the larger the margin has to be. This creates an increased risk of overdosing more healthy tissue with radiation than necessary.

    Localisation with X-ray
    One common method to localise a target region is the use of artificial landmarks, such as gold markers, instead of visualising the target directly. By finding such markers with X-ray imaging and knowing their position in relation to the target region, the examiner can infer the location of the target in the body. However, the soft tissue in the abdomen has poor X-ray contrast, further complicating the analysis.

    Because X-ray examinations can only be performed infrequently, surrogate measurements, such as chest movement, are used to estimate the position of the target. Any inaccuracy may cause treatment errors.

    There are other imaging techniques, but generally current methods are too slow to track intracorporeal organ motion adequately.

    Ultrasound as an emerging alternative
    Two-dimensional ultrasound (US) is one of the most common imaging techniques used in routine diagnostics. It does not depend on harmful radiation, which means that a patient can be exposed to US repeatedly and for longer periods. However, using US requires a significant amount of training and the quality of results depends greatly on the skill of the physician. The examiner uses marker points to navigate through the body. Because organs are scanned in layers, it is often unclear which section of an organ is being displayed. Trying to establish a diagnosis retrospectively using saved images is nearly impossible without knowing the marker points, scanning angle and other details.

    Nevertheless, US devices are fairly compact and cheaper than alternative technologies, such as computed tomography or magnetic resonance imaging. Real-time images are created quickly but are generally less detailed than tomographic results and may be flawed due to technological artefacts.

    The recent emergence of three-dimensional US has solved some problems by eliminating the restriction to 2D layers. The target, along with surrounding tissue, can now be imaged at the same time. Where required, layer views can be derived from recorded data.

    Entering the fourth dimension
    Now that creating 3D US images in real-time (i.e., 4D US imaging) is possible, the next step is to eliminate manual positioning of the scanner because data from such moving images are impractical for retrospective analysis. This is where robotic arms enter the stage. Ideally, these detect the target region automatically and adjust to tissue movements by quickly processing 3D images whilst also maintaining a high image quality. Effectively, this creates motionless tissue representations on screen.

    The 4D ultrasound technology in action using a dummy for testing.

    There are only very few robotic real-time imaging devices available to date, none are used commercially. An ideal field of application would be radiation therapy. There, the two main approaches for image-guided tumour treatment are based on tracking or gating. Tumour tracking involves adjustment of the treatment beam through movement of the beam source or the treatment couch, or by adjusting the beam geometry and can be used for all moving tumours. Gating, however, is only used for tumours liable to move with respiration. In this approach, treatment is only applied during defined breathing episodes, i.e., end expiration. This can be improved by using a technology called deep inspiration breath hold, which prolongs the overall treatment time.

    Since a robot can work by itself, this will allow hospitals to make better use of specialist personnel’s time. Eventually, 4D US robots could even be used to assist in automated surgeryQuote_brain

    None of the emerging technologies for US-directed robotic arm positioning integrate radiation therapy yet. Professor Ernst’s team and collaborators are working on this specific application. Success would be highly beneficial because no US specialist can be near the patient during ongoing irradiation for safety reasons.

    A major technical challenge in Professor Ernst’s endeavour is the blocking of therapeutic beams by the US device. Unsurprisingly, the position for the US device that yields the best images is, quite often, also the best position for the treatment device. One idea to compensate for this flaw is to develop treatment plans which ensure that all robotic arm movements are used as effectively as possible. Another idea is to design devices that allow X-rays to pass through them unhindered.

    A glimpse into the future
    4D technology could be used in routine and follow-up diagnostics to calculate patient coordinates and allow image comparison over time. When mounted on a robot, it would be the first long-term imaging modality available for clinical use. It could help create virtual organs from 3D data for analysis. A long-term ambition for Professor Ernst and his team is the development of diagnostic algorithms from standardised data packages to improve the quantification of exam results. These applications may alleviate pressures associated with the lack of physicians in remote areas.

    Since a robot can work autonomously, the examiner would be able to leave the patient during lengthy examinations. This would allow hospitals to make better use of specialist personnel’s time. Eventually, 4D US robots could even be used to assist in automated surgery.

    The advances from 2D to 4D US imaging have been significant thanks to Professor Ernst and other researchers in this cross-disciplinary field. The next level is to now further improve and broaden medical applications. Such investments will not only advance patient care, but also be compensated through savings in staff and treatment time. Considering that funding for public health services is more valued than ever, such innovative developments should be embraced.

  • Could 3D ultrasound completely replace X-ray scans in the future?
    Even though 3D ultrasound is an exciting imaging methodology, it will never completely replace X-ray scans. There will always be situations where X-ray imaging is necessary. These will, for example, be imaging of structures behind bones or the bones themselves, such as cranial or dental imaging as well as imaging of fractures in adults. Additionally, boundaries, where the sonic properties (speed of sound, transmission, etc.) change strongly, will always be challenging for ultrasound. A typical case here is the tissue–air boundary: it’s hard to see inside or through the lungs using ultrasound, and this will not change much in the foreseeable future. Another technology which cannot work without X-ray imaging is computed tomography, and I am convinced that we will continue to use it in the future.

    Are there any known side-effects of US?
    Ultrasound has been in clinical use since the late 1950s, and there is no evidence that it is in any way harmful to the patient. Multiple studies have been performed to determine a possible influence of ultrasound on pregnant women and other patient cohorts, but no clinically relevant data was found. Nevertheless, it is still common practice to use the lowest possible intensity setting, following the ALARA-principle known from radiology (as low as reasonably achievable).

    Our progress is very promising, and we have come very close
    to building a fully functional clinical prototypeQuote_brain

    What other possible applications do you see for robotic ultrasound?
    While our main research is focused on ultrasound for radiation therapy, we are also pursuing other ideas. I imagine that 4D ultrasound could become the first medium- to long-term imaging modality available. It is conceivable that we could visualise drug uptake or metabolic changes due to medication. Using the robotic ultrasound device we’re developing, it will also be possible to do long-term US – like we can today do long-term ECG and 24h-monitoring of blood pressure. Another idea we’re pursuing is the use of ultrasound in the diagnosis of children’s arm fractures, which is currently done using X-ray imaging.

    If all goes well, how soon could a robot performing 4D US tracking in radiation therapy become commercially available?
    I believe that the technology is nearly there – our progress is very promising, and we have come very close indeed to building a fully functional clinical prototype. If our current research projects continue as planned, we could see a working device by 2020. This will be research only, however, and I estimate that it will take another four to six years to make it commercially available.

    Will this new technology require a substantial shift in the required skill set and training of physicians?
    Our robotic system can be used in nearly the same way as a regular hand-held ultrasonic probe. The robot is sufficiently supple and force-sensitive to allow unimpeded manual motion of the probe. The only thing to bear in mind is that the reach of the robot is limited, but experiments have shown that people adapt very quickly.

    Do you think patients will be put off or even scared by the lack of human interaction during robotic examination and treatment?
    Of course, an automated system may seem scary at first. One of our goals, however, is to make the device as safe as possible. Under no circumstances will the robot apply excessive forces to the patient – that’s precluded by the manufacturer’s safety settings. Furthermore, we have implemented a technology that will always allow manual intervention, i.e., the patient will always be able to push the robot away should something unexpected or scary happen.

  • Research Objectives
    Professor Floris Ernst’s research investigates the potential for using robots and novel 3D-imaging technology to carry out ultrasounds within the soft tissue of the human body.


    • University of Lübeck
    • German Research Foundation (DFG) ER 817/1-1
    • Business Development and Technology Transfer Corporation of Schleswig-Holstein (WTSH) 123 16 014
    • German Federal Ministry for Economics (BMWi) ZF4109402BZ6

    Germany: Prof. Alexander Schlaefer, Technical University of Hamburg-Harburg; Prof. Dirk Rades, University Hospital Schleswig-Holstein; Boll Automation GmbH; Saphir Radiochirurgie Zentrum Norddeutschland Switzerland: Varian Medical Systems Imaging Laboratory GmbH Denmark: Per Poulsen, Aarhus University Australia: Paul Keall, The University of Sydney

    Professor Ernst completed his PhD at the University of Lübeck, investigating motion prediction and correlation algorithms for use in robotic radiosurgery. Following this, he worked as a software engineer at an engineering consultancy before returning to the Institute of Robotics and Cognitive Systems at the University of Lübeck in 2013, where he was appointed Professor for Medical Robotics in 2017.

    Professor Dr Floris Ernst, PGDipSci
    University of Lübeck
    Institute for Robotics and Cognitive Systems
    Ratzeburger Allee 160
    23562 Lübeck

    T: +49 451 3101 5208

    LinkedIn: /florisernst

  • Robots take ultrasound to the fourth dimension

Leave A Reply