From The Evidence: What Else Can AI Find in the Data We Already Have?

Normally you order a chest CT to answer a question. Does this patient have pneumonia? Do they have a lung clot? Do they have cancer?

What if we could ask more than one question from the same CT?

A team of researchers recently published a paper about an AI that can identify oesophageal cancer from a standard non-contrast chest CT. This is interesting because CT is not a great modality for picking up early oesophageal cancer. The oesophagus is constantly moving and collapsing, which can make early cancers hard to distinguish.

They trained their AI, called EAGLE, on CTs from 6,813 patients and validated it across 12 hospitals in China, the Czech Republic and Australia, with over 80,000 patients in total. When externally validated, it was able to pick up around 90% of oesophageal cancers with a specificity of 98.5%. It could even identify some early cancers and precancerous changes.

Why is this useful? Well, CT is quick and readily available. It can image large areas of the body quickly and is cheaper and more accessible than an MRI. The downside of CT is radiation. So we probably shouldn't be scanning millions of healthy people with more CTs to try and find oesophageal cancers.

But what if we were already doing the scan?

If someone came to hospital with suspected pneumonia and got a CT of their chest, the radiologist would be looking for signs of that pneumonia, but the oesophagus is there too. Can we use that scan to look for oesophageal cancers? Maybe that person was having a low-dose CT for lung cancer screening. Could we use those images to look for other cancers, osteoporosis or muscle wasting? They had a CT coronary calcium score. Can we look at their lungs in that image?

They already received the radiation. They already had the scan. The information is there.

This is one of the interesting aspects of the EAGLE paper. The authors went as far as to say that we could use existing lung cancer screening programs to identify patients at risk of oesophageal cancer. Perhaps AI can help us do more tests. Or maybe it can help us get more out of the tests we already do.

There is one caveat though: If you look for more things, you will find more things.

In the prospective hospital study they looked at 17,446 patients and found 90 positive results. Thirty-eight of these were confirmed as oesophageal cancers or high-grade precancers, potentially allowing earlier treatment. But that also left 52 positive results that were not confirmed as cancer. Those patients may still have needed further imaging, biopsies or follow-up, along with the anxiety that can come with being told something might be wrong.

I've experienced this first hand. I recently had a coronary CT and was thrilled with my results. Calcium score of 0 and coronary age less than 30 (I'm 48). I was basking in this until I noticed something at the end of my report: degenerative changes in my spine.

It wasn't what they were looking for, so there was no context. Is this normal for my age? Should I be doing something about it? Does it even matter? I wasn't as happy about my results now.

There may be good reasons to mention it, but just because you can find more doesn't mean you should. As we use AI to pull more information out of our scans, we need to know what to report and how much context to give.

That's the caveat. But it doesn't detract from the big picture that EAGLE presents. Information collected for one purpose can answer other questions.

We already collect tons of information about our students. Throughout a degree they will write many essays, sit exams, complete short-answer questions, present and reflect. We mostly use each piece of information once. Write an essay. Get marked. Receive a 72%. Move on.

We are starting to look at using AI to identify cheating or misuse of AI in student work. If we can use AI to look for bad things in a student's work, why can't we use that information to look for good things?

Have they gotten better at writing an argument? Evaluating evidence? Making original connections? Writing more clearly? Being more creative? Maybe we can use AI to identify growth.

This reminds me of the Athlete Biological Passport. Instead of just looking at one test, they look at multiple biological markers in an athlete over time to identify potentially suspicious changes. You can see patterns that may not be apparent with one result.

A student passport could do this for students, but it could also look for positive changes. Instead of looking at one essay and asking if it looks like AI, we could look at their work over several years and ask how they have changed.

When you finish your degree, you could not only see that you got an average of 74%, but that you improved at using evidence, using argumentation and writing clearly, but didn't improve much at quantitative reasoning.

Some of this is already happening. UNSW has started a Skills Passport that allows students to identify skills they have developed throughout their degree. They also have an AI Skills Buddy that can help students identify these skills. Researchers at the University of Melbourne have been looking at undergraduate essays over multiple years to see how things such as argumentation, research and referencing have changed across cohorts. Others have looked at thousands of student reports to see if writing has changed since ChatGPT. By combining these concepts, we could look at the work students already produce and see how they grow.

This could also help us answer the question of whether AI is helping or hindering learning. If a student writes better essays after using AI, that doesn't tell us much. If they also do better in supervised work and can explain their thoughts verbally and write well without AI, then maybe they are learning. If their submitted work improves drastically but everything else stays the same, that may be a concern.

Not only could a passport look back, it could also help guide students in the future. If a student is good at being creative but poor at quantitative reasoning, we could point them towards units that will help them develop that skill. If they keep gravitating towards a certain topic, we could show them elective or research opportunities. The degree could start to tailor itself to the student.

This also changes the conversation about AI and university. Right now, much of the discussion is about what AI might take away from education: students using it to write essays, assessments becoming less reliable, and universities being less certain that graduates can actually do what their marks suggest they can. These are all valid concerns. But what if AI can help us show what our students learn?

If an employer can see a transcript, they can see what the student studied and what marks they got. What if they could also see that they have grown in their ability to analyse evidence, communicate thoughts, solve problems or be creative over the three years at university? That could be beneficial to the student and maybe the employer as well.

Similar to the CT scan example, we will run into false positives. If someone's writing suddenly improves, it may look suspicious, but they may have received tutoring, improved their English or they just started working harder. Students are meant to change when they go to university. So if someone changes too much, it doesn't necessarily mean they cheated The AI shouldn't make that decision for us. It can help us and our students identify trends.

We don't have to go as far as creating a student passport. An educator could start much more simply by giving students a short piece of supervised work at the beginning of a unit and choosing three things they would like to see develop — perhaps argumentation, use of evidence and originality. Later in the semester, students could complete another piece of work that allows those same areas to be compared.

If your university allows you to use AI, you can have it compare the two pieces of work using those metrics. Don't ask, did this student cheat? Ask, what have they improved at? What have they not improved at? Why do you think that? Show the student the information. Do they agree? What have they improved at? What did the AI get wrong? Now you are using AI to help your students learn instead of policing them. This could be a relatively simple way to test whether AI can help identify genuine growth over time, and it could also make an interesting area for further educational research.

Obviously there are ethical considerations when it comes to universities looking at students' work over long periods of time. Just like with CT scans, we don't want to overdo it and look for everything. But that's the takeaway from EAGLE. We gather so much information and only use it for one thing. We do a CT to answer one question. We write an essay to get a mark.

AI can help us ask more questions.