Finding Needles in a Hue-Stack

Reading time:
time
min
Grid of 44 fluorescence microscopy images of single blood cells in blue, green and red channels

Over the past few months we have partnered with the CTC Atlas team, led by Dr. Natalia Bednarz-Knoll at the Laboratory of Translational Oncology, Medical University of Gdańsk, to embark on a perilous journey: armed with deep learning methods, we set out to build (semi-)automated workflows for identifying the few Circulating Tumor Cells (CTCs) among millions of cell images from imaging flow cytometry of blood samples.

Why chase a handful of cells? As a tumor grows, some of its cells break away and slip into the bloodstream. These are circulating tumor cells, and some of them may go on to seed metastases elsewhere in the body, which cause most cancer deaths. That makes them a valuable source of information. A tissue biopsy is invasive and gives a single snapshot. CTCs can be collected from an ordinary blood draw, a “liquid biopsy”, and sampled again and again: before treatment, during therapy, and as the disease changes over time. Their numbers have been linked to prognosis and treatment response in many solid tumors. Because CTCs come in many shapes and molecular flavors, they can also show how varied a tumor is, which a single tissue sample might miss. They even travel with company, such as circulating cancer-associated fibroblasts (cCAFs), which may help prepare the ground for new metastases. The catch is that CTCs are extraordinarily rare: often just a few cells per milliliter of blood, hidden among millions of white blood cells and billions of red ones. Spotting them means sifting through a mountain of similar cells (each imaged as a stack of fluorescent channels), which is exactly where our needle-in-a-hue-stack story begins.

The Atlas team had already collected hundreds of samples from half a dozen cancer types. They had meticulously sifted through them using labor-intensive, semi-manual gating to narrow down the cells that needed expert review. After weighing protocol consistency, availability of clinical metadata and several other factors, we jointly settled on an ambitious goal. We would train a model to tell CTCs and cCAFs apart from normal cells, using only samples from a single cancer type (plus some healthy donors), collected and analyzed at a single clinical site. Then we would see how well it generalizes along several axes: to unseen donors with the same cancer type, to other cancer types, and to data from another clinic. As a final, deliberately far stretch, we also tested it on a different sample type altogether. Liquid biopsy usually relies on peripheral blood, drawn from a vein in the arm. We additionally used blood taken directly from the vein draining the tumor, which is only feasible for some cancer types and often only during surgery. In the end, we worked with around 100 blood samples from ~75 donors, covering five cancer types. Altogether, we used several thousand CTC/cCAFs among over 10 million cells.

After many rounds of improvements, we arrived at a fairly standard convolutional neural network (CNN) as our final model for the study. The performance of the model was strong on almost all studied cancer types and generalised to samples from the other studied clinical site. As expected, performance dropped on the tumor-draining blood samples, the furthest stretch of the lot. Less expected was a dip in one of the five cancer types, where more false positives appeared. The full analysis and results will be presented in a journal article we are finishing together. What we can share already is that combining the model into an expert-reviews-a-tiny-set-of-candidates workflow we were able to cut the manual effort of reviewing a single sample from around 30 minutes to slightly over 2. Below we share a few lessons learned along the way.

Talk, then talk some more

Firstly, the key to combining biology and machine learning in one project is communication. We have worked on several projects like this before, so we knew from the get-go how crucial it is for the two fields to exchange and challenge each other’s ideas: what is hard, what is easy, which assumptions are necessary and which can be relaxed. In particular, this took the form of several rounds of manual review experiments. We prepared different sets of cells and presented them to experts in a custom user interface for independent review. These rounds uncovered multiple insights. Some were directly useful for the modeling, and others will inform future sample collection and processing protocols.

Mind the size of the haystack

Secondly, one of the perils of this project was the sheer volume of data. In total we processed ~1.5 TB, which unpacks into ~100 million tiny images of individual cells. Not all of the tools we used (and, in some cases, had to use) were built for volumes like this, so we had to get creative. One trick was to pack each sample’s cell images into an LMDB database (a lightweight key-value store). That kept loading fast enough for model training and inference while cutting the number of files we had to manage down to a few hundred.

Bigger is not always better

Finally, while the machine learning landscape is dominated by huge transformer based neural networks these days, it is important to recognise that task-specific not-too-large models have an important role to play still. This was well illustrated in our case, where even tiny transformer based models memorised the training data, but were unable to generalise, while the best performing models were somewhat dated ConvNext-based architectures of moderate size.

Biology is complicated. We are a team of computer scientists, data scientists and machine learning experts with experience in biological and medical domains, but we don’t have a deep, detailed understanding of the frontier of oncology research. Our partners do. We were only able to find the needle because they let us work with their data and ideas, while we showed them their insights and issues through our lens. This way perhaps we moved the needle a tiny bit in the liquid biopsy space.

Keep an eye out for the paper.

Explore Possibilities

Share Your Data Goals with Us

From advanced analytics to platform development and pharma consulting, we craft solutions tailored to your needs.