Jun 4, 2021

U of T Researchers Examine Use of AI in Radiation Oncology

Research, Faculty & Staff
CHRIS MCINTOSH

A team of researchers directly compared physician evaluations of radiation treatments generated by an AI machine learning (ML) algorithm to conventional radiation treatments generated by humans.

They found that in the majority of the 100 patients studied, treatments generated using ML were deemed to be clinically acceptable for patient treatments by physicians.

Overall, 89 per cent of ML-generated treatments were considered clinically acceptable for treatments, and 72 per cent were selected over human-generated treatments in head-to-head comparisons to conventional human-generated treatments.

Moreover, the ML radiation treatment process was faster than the conventional human-driven process by 60 per cent, reducing the overall time from 118 hours to 47 hours.

“We have shown that AI can be better than human judgement for curative-intent radiation therapy treatment. In fact, it is amazing that it works so well,” says Chris McIntosh, an assistant professor at University of Toronto’s Temerty Faculty of Medicine Department of Medical Biophysics.

In the long term this could represent a substantial cost savings through improved efficiency, while at the same time improving quality of clinical care, a rare win-win.

The study also has broader implications for AI in medicine.

While the ML treatments were overwhelmingly preferred when evaluated outside the clinical environment, as is done in most scientific works, physician preferences for the ML-generated treatments changed when the chosen treatment, ML or human-generated, would be used to treat the patient.

In that situation, the number of ML treatments selected for patient treatment was significantly reduced issuing a note of caution for teams considering deploying inadequately validated AI systems.

Results by the study team led by Chris McIntosh, Leigh Conroy, Ale Berlin, and Tom Purdie were published in Nature Medicine.

“A major finding is what happens when you actually deploy ML in a clinical setting in comparison to a simulated one,” says McIntosh, who is also a scientist at the Peter Munk Cardiac Centre’s and Techna Institute, as well as chair of Medical Imaging and AI at the Joint Department of Medical Imaging at the University Health Network,

“There has been a lot of excitement generated by AI in the lab, and the assumption is that those results will translate directly to a clinical setting. But we sound a cautionary alert in our research that they may not,” says Purdie, an associate professor at Temerty Faculty of Medicine’s Department of Radiation Oncology.

“Once you put ML-generated treatments in the hands of people who are relying upon it to make real clinical decisions about their patients, that preference towards ML may drop. There can be a disconnect between what’s happening in a lab-type of setting and a clinical one.”

In the study, treating radiation oncologists were asked to evaluate two different radiation treatments – either ML or human-generated ones - with the same standardized criteria in two groups of patients who were similar in demographics and disease characteristics.

The difference was that one group of patients had already received treatment so the comparison was a ‘simulated’ exercise. The second group of patients were about to begin radiation therapy treatment, so if AI-generated treatments were judged to be superior and preferable to their human counterparts, they would be used in the actual treatments.

Oncologists were not aware of which radiation treatment was designed by a human or a machine. Human-generated treatments were created individually for each patient as per normal protocol by the specialized Radiation Therapist.

In contrast, each ML treatment was developed by a computer algorithm trained on a high-quality, peer-reviewed data base of radiation therapy plans from 99 patients previously treated for prostate cancer at Princess Margaret.

For each new patient, the ML algorithm automatically identifies the most similar patients in the data base, using learned similarity metrics from thousands of features from patient images, and delineated target and healthy organs that are a standard part of the radiation therapy treatment process.

The complete treatment for a new patient is inferred from the most similar patients in the data base, according to the ML model.

Although ML-generated treatments were rated highly in both patient groups, the results in the pre-treatment group diverged from the post-treatment group.

In the group of patients that had already received treatment, the number of ML-generated treatments selected over human ones was 83 per cent. This dropped to 61 per cent for those selected specifically for treatment, prior to their treatment.

“In this study, we’re saying researchers need to pay attention to a clinical setting,” says Purdie. “If physicians feel that patient care is at stake, then that may influence their judgement, even though the ML treatments are thoroughly evaluated and validated.”