Deep learning applications in visual data for benign and malignant hematologic conditions: a systematic review and visual glossary

Abstract

Deep learning (DL) is a subdomain of artificial intelligence algorithms capable of automatically evaluating subtle graphical features to make highly accurate predictions, which was recently popularized in multiple imaging-related tasks. Because of its capabilities to analyze medical imaging such as radiology scans and digitized pathology specimens, DL has significant clinical potential as a diagnostic or prognostic tool. Coupled with rapidly increasing quantities of digital medical data, numerous novel research questions and clinical applications of DL within medicine have already been explored. Similarly, DL research and applications within hematology are rapidly emerging, although these are still largely in their infancy. Given the exponential rise of DL research for hematologic conditions, it is essential for the practising hematologist to be familiar with the broad concepts and pitfalls related to these new computational techniques. This narrative review provides a visual glossary for key deep learning principles, as well as a systematic review of published investigations within malignant and non-malignant hematologic conditions, organized by the different phases of clinical care. In order to assist the unfamiliar reader, this review highlights key portions of current literature and summarizes important considerations for the critical understanding of deep learning development and implementations in clinical practice.

Introduction

Recent advances in large-scale data storage, availability, and computational power have led to significant interest in the development of new techniques for “big data” analysis. Rapidly evolving artificial intelligence (AI) algorithms aim to efficiently utilize vast amounts of information with minimal human interaction to address tasks that automate or improve upon human-level assessment. Artificial intelligence takes many forms and includes domains of deep learning (DL), convolutional neural networks (CNN), and other related techniques that are capable of processing imaging data quickly and automatically. Research divisions within commercially successful technology companies have popularized DL models for vision-related tasks, such as facial recognition, image segmentation, object detection, and many other examples that are currently being integrated into daily life.

Within the medical field, visual assessment of digitized clinical imaging and biospecimens by physicians is critical in numerous phases of clinical care for patients. As a result, early investigations that employ clinical DL using histology slides or radiological images within medicine have produced promising results, including diagnosis detection,¹ clinical subtyping,² cancer mutation prediction,^3,4 and survival.⁵ Recognizing the clinical importance of these algorithms, the US Food and Drug Administration has approved a number of novel AI and DL products.⁶

However, DL algorithms exploring malignant and non-malignant hematologic conditions are still scarce. With digitization tools generating larger biospecimen image databases^7,8 and researchers becoming increasingly familiar with DL techniques, examples of applications in hematology are growing exponentially.^9-14 As such, it is inevitable for hematologists to be familiar with the broad concepts, applications, and limitations of clinical DL.

In this structured narrative review, we aim to describe the general concepts, provide a visual glossary for key terms within image-based DL, and conduct a systematic review to provide an up-to-date assessment of the application of image-based DL in benign and malignant hematology across various phases of patient care.

Neural networks and deep learning

The concept of “deep learning” is poorly defined, imprecise, and often used interchangeably with terms such as “machine learning” and “artificial intelligence.” Traditionally, “artificial intelligence” is the use of automated systems to perform a particular task. “Machine learning” represents a subset of AI in which rules are not explicitly predetermined, but are acquired by training and optimizing parameters based on observed data. Machine learning workflows traditionally separate data into training, validation, and external testing cohorts for model assessment. Examples of machine learning that are probably the most familiar include linear regression, logistic regression, or Cox proportional hazards models. “Deep learning” is a recently-popularized subset of machine learning utilizing a specialized neural-network architecture undergoing millions of arithmetic operations (Figure 1).^15,16 DL architectures are loosely modeled after the complex neural connections of the human brain.¹⁷ Although the term “deep learning” is derived from “deep convolutional neural networks” and has gained interest particularly in clinical research, the strict definition has become increasingly ambiguous and may not completely represent modern state-of-the-art techniques. The field of DL and the list of essential glossary terms are rapidly changing, but in keeping with contemporary clinical manuscripts, this review will use the term “deep learning” to mean “deep convolutional neural networks and other contemporary techniques related to computer vision”. There are also non-image-based neural networks and image-based machine learning architectures without neural networks; however, both are beyond the scope of this current review.

Image preprocessing

A standard workflow^3,18,19 for DL research typically requires preprocessing input images, which can expedite DL training time or improve performance. Preprocessing steps are typically dependent on the modality of the image type. While radiological images may be input either whole or with particular Regions of Interest (ROI) segmented, histopathology slides are typically tessellated into smaller tiles representing tissue or segmented cells of interest prior to inclusion into the model. Normalization of pathology images may reduce artifacts specific to a clinical site or particular scanning device, but there is no current standard normalization process. Data augmentation may be performed with random image rotations, vertical or horizontal flips, and simulated compression artifacts to increase the size of the training set and broaden generalizability. In addition to using images alone, researchers can include other data modalities such as clinical information with multi-modal models to supplement image inputs in attempts to improve model performance.

Figure 1.Exemplifying differences in “Artificial Intelligence”, “Machine Learning”, and “Deep Learning” with regards to anemia. “A r t i ficial Intelligence” (AI) involves automation of tasks, and can be an explicitly programmed rule to categorize based on the level of a laboratory value. “Machine Learning” (ML) methods, such as linear or logistic regression, derive associations from given training data. More complex image-based “Deep Learning” (DL) models utilize complex architectures termed “Neural Networks” to associate subtle features associated with particular outcomes of interest by using input training data, similar to other machine learning frameworks. In this figure, the clinical condition of anemia based on hemoglobin (Hb) values is used as an example for the above computational frameworks. DL methods may extract features in non-traditional images, such as fundoscopic exams, to derive clinically meaningful categorizations.

General neural network structure

In a simplified viewpoint of neural network structures (Figure 2), the input image is transformed at various intermediate states, termed “nodes,” with each node representing a different graphical feature of the image. As the image is passed from node to node, the connection between each node involves mathematical transformations to represent more complex features in later nodes. Each node can be connected to multiple subsequent nodes simultaneously, and the group of nodes with similar numbers of sequential connections from the input image represent a layer of intermediate nodes. Shallow and Deep neural networks refer to the number of node layers within a particular architecture, but there is no strict definition to differentiate the two. In addition, nodes may not necessarily connect to the nodes in the immediately subsequent layer, but may connect by “skip connections” to nodes in later layers. The penultimate layer of nodes, each representing only a single numerical value, is termed the Logit Layer, the values of which are then normalized between the range of 0 and 1 to give the final probabilities for the outcomes of interest. Common outcomes of interest and examples include object detection, segmentation, classification, regression, survival analysis, and detail optimization (Figure 3).

Information propagation and parameter training

To develop a neural network model, the input image is represented numerically by each pixel. The numerical information is propagated though intermediate nodes and layers towards the direction of the output layer. The connections between nodes are mathematically represented by either non-linear operations or matrix multiplication and addition with potentially millions of trainable parameters, whose values are updated while optimizing the end outcome.

Upon initial model evaluation, the sequential movement of information from input image towards the outcome of interest is deemed “forward propagation” or “forward pass” (Figure 2A-F). To complement an initial prediction, the user defines a particular loss function to quantitatively describe the incorrectness of the model’s prediction from available ground truth. Using an additional user-defined optimizer algorithm, the trainable parameters are iteratively adjusted to decrease the loss value in subsequent forward passes. This framework of optimizing parameters in earlier layers using information from the predicted outcome is deemed “back propagation.” During training, forward and back propagation are repeated for a defined number of repetitions, or epochs, but training can also be stopped if other defined optimal conditions are met. Given the need for at least 109 calculations per forward pass, parallel computing often requires specific hardware such as Graphics Processing Units (GPU) to expedite necessary matrix operations to be finished within reasonable timeframes.

Convolutions

At the time of writing, the most popular type of DL architecture is the convolutional neural network (CNN). The prototypical CNN algorithm assesses a smaller grid-like portion of each input image prior to propagation towards the next layer (Figure 2C). CNN utilize the convolution operation between layers, which involves matrix multiplication across overlapping sub-sections of the input image to produce a lower-dimensional output representation.

Pre-trained networks and transfer learning

Initially, CNN trained to perform object detection required millions of manually-annotated images, training for days or weeks on industry-grade computational equipment.²⁰ After training is complete, CNN have traditionally been understood to learn “low-level” general features such as lines, edges, and shapes in earlier layers of the network, but more complex “high-level” features such as faces, patterns, and spatial distributions are learned in subsequent layers that are more closely associated with the evaluated outcome.²¹

In clinical research, it is rare for clinicians to have the resources to develop new CNN architectures with initially random parameters; such a feat requires large-scale databases with expert-level annotations and access to industry-grade supercomputers. Researchers have taken advantage of the learned features along progressive layers by using models previously trained on large databases for non-clinical tasks, but repurposing the final few layers to predict specific clinically-relevant outcomes. The concept of transfer learning involves utilizing a pre-trained network such as those already trained on the ImageNet database of over 1 million general images,²² initializing the model with the parameters that learned “low-level” features from images unrelated to the application of interest, and allowing the model to retrain and modify parameters in the last few layers to learn “higher-level” features on images for specific patient-related tasks. By utilizing transfer learning, the minimum required dataset and computational power is significantly less than fully training a network from completely random parameters.²³

Specific deep learning architectures in clinical research

While DL is a framework of neural networks for outcome prediction, each specific model architecture incorporates drastically different complexities with regards to number of layers, connections between layers, functions, and many other highly-engineered features. In fact, newer contemporary models lack any convolutional layers, and infer local and global image features by other methods.²⁴

Figure 2.Brief representation of the structure and training phases of deep convolutional neural networks. Collectively, A-H represent “forward propagation” and G-H represent “back propagation.” (A) Images are first passed into the network to predict an outcome of interest. In this example, 4x4 pixelated images of written numbers are used to train a network to predict the numeric value of the image. (B) Pixels are initially converted into numeric values based on pixel intensity. (C) Smaller subsections of the input images are transformed with the convolutional operation, which involves matrix multiplication and addition with trainable parameters. (D) As information is passed into subsequent layers, the image undergoes non-linear transformations, such as the Rectified Linear Unit function that allow the model to represent non-linear relationships within the data. (E) Within this figure, intermediate layers are restructured to a layer of single numerical values in the Logit layer. (F) After propagation through the pre-defined number of convolutional layers, the final activation function normalizes the Logit layer into a distribution of probabilities across the space of available outcomes. The value with the highest probability is deemed the model’s prediction. (A-F) The framework outlined as information is passed from image input to model prediction is termed “forward pass” or “forward propagation.” (G) After the first pass of the model’s predictions, a loss function specific to the outcome data type is calculated to quantitatively assess the level of error produced by the initial prediction. The loss function is chosen before training by the user. Common examples of loss functions are “cross entropy” for categorical outcomes and “mean square error” for regression outcomes. (H) Optimization algorithms iteratively alter the trainable parameters within each of the previous convolutional layers based on the defined loss function. The direction and magnitude of parameter adjustments is calculated by either maximizing or minimizing the loss function in future forward passes, as chosen by the user for the outcome of interest. (G and H) The framework outlined for automatically adjusting earlier parameters to optimize model performance is termed “back propagation.” The process of forward (A-F) and back (G-H) propagation is repeated until a pre-specified set of conditions is fulfilled, typically leading to more accurate predictions.

Figure 3.Examples of outcome tasks and explainability methods in Deep Learning. In this example, the initial input image is a promyelocyte with visible Auer Rods as seen on a peripheral blood smear, a possible pathognomonic finding for acute promyelocytic leukemia. Output tasks can include localization of white blood cells (Detection), creation of a region of interest around the nucleus (Segmentation), disease prediction (Classification/Regression), or increasing the visual quality of the input image (Detail Optimization). Explainability methods are necessary to ensure biological feasibility. In exploratory analyses, parameters within the intermediate layers can be directly visualized (Feature Maps), heatmaps can be generated to highlight specific areas associated with the outcome (Saliency or Attention Maps), synthetic images can be generated from noise to represent an outcome of interest (generative adversarial networks), or cluster analyses can be performed with dimensionality reduction techniques.

Thus far, the predominant architecture for hematology-specific questions tend to be from a class of CNN known as Residual Neural Networks (ResNets), which utilize skip connections. Most specific ResNet architectures, such as Inception, EfficientNets, MobileNets, and other various ResNet models are open-source and widely available.²⁵

Certain model architectures are engineered to provide an output that is an additional image; these model structures are needed for dimensionality reduction, bounding-box detection, segmentation, and noise reduction tasks. One specific architecture, Autoencoders, are networks that pass an input image through an intermediate lower-dimensional representation, followed by upsizing to a higher-dimensional space to recreate the input image.²⁶ Theoretically, the lower-dimensional intermediate representation still retains features of the original image which may be clinically or biologically relevant. Similar architectures such as U-Net require additional training data, such as object ROI or low/high-quality image pairs, to accomplish tasks such as image segmentation or digital optimization.

An additional relevant DL framework utilized is Multiple Instance Learning (MIL)^27,28 and its attention-based derivatives^,29,30 including the Clustering-constrained Attention Multiple Instance Learning (CLAM).³¹ The main distinction in MIL frameworks is the prediction for data subsets and not for single instances. Specifically, input images are separated into smaller subsets. The entirety of the subset is predicted “positive” if at least one image in the subset is predicted “positive”. As an example of MIL in histopathology, a biopsy whole-slide image would be predicted “cancerous” when one extracted tile is predicted as such.¹ This framework may be particularly helpful when single annotations are provided across an entire image, or “weak supervision”, and not necessarily labels for each specific segmented ROI. In addition, Attention, or a numeric weight, can be assigned to each image tile to produce weighted predictions, as well as provide explainable heatmaps. Using Attention, CLAM was developed to increase the speed of MIL and reduce the noise from irrelevant image tiles.

Vision Transformers (ViT) are a novel technique that do not utilize the convolution operator.²⁴ The entire image is separated into a grid of sub-images that are analyzed in parallel along with the relative location of each sub-image. With this method, global relationships across the entire image may be learned by the model as opposed to only local features that are seen by the previously-described CNN.

Currently, most architectures for hematology-specific questions utilize ResNet architectures, with just a few examples also incorporating MIL. However, the emergence of ViT and CLAM frameworks are part of a changing landscape of implemented DL architectures. In general, the choice of model architecture is somewhat informed by expected outcome task, but it is still largely empiric. However, there are broad advantages and disadvantages for each of the previously-mentioned frameworks. With weak supervision, MIL tends to require significantly larger amounts of training data than ResNets.¹ CNN and ViT perform equally well at the scale of currently available clinical datasets. However, ViT are superior to CNN for larger scale datasets and are more computationally efficient with significantly fewer parameters.²⁴ There are numerous methods to attempt to explain the inner mechanisms of standard CNN,³² but similar methods to “open the black box” of ViT are currently under development.³³

Figure 4.Literature search. (A) Search terms to extract relevant manuscripts related to deep learning in malignant and non-malignant hematology. Articles were queried in Pub-Med using one “Deep Learning” term in addition to one “Hematology” term. (B) PRISMA diagram of “Deep Learning in Hematology” survey. Initially 2,708 articles were found from a PubMed query. After initial review of abstracts and article titles, 237 reports were deemed eligible for further review of full manuscripts. Finally, 65 articles were included for the current narrative review. Justification for exclusion are provided.

Explainability

While “explainability” in DL research is loosely defined, in this review, “explainability” refers to the efforts in describing DL models and predictions in humanly-understandable concepts.³⁴

Although DL may empirically exhibit a high performance, DL is often criticized for its highly complex mechanisms and is often thought of as a “black box.” In multiple examples, seemingly high-performing models often utilize artifact or contextually irrelevant features for its predictions, as the artifactual features may be unintentionally over-represented in certain imaging subgroups.³⁵ Multiple methods are under development to explain and validate biologically reasonable predictions. As such, explainability is increasingly important in clinical AI development and in developing physicians’ trust of DL.³⁶

To give just a few examples, unsupervised data dimensionality reduction methods such as principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), and Uniform Manifold Approximation and Projection (UMAP) are statistical techniques used to group visually similar input images into clusters, which may overlap with relevant outcomes. These methods are also popularized in non-imaging data such as single cell molecular and cytometry time-of-flight analyses. Feature maps are direct visual representations of the intermediate trained parameters. Plotting Attention scores or using Saliency map methods such as Grad-CAM or Smooth-Grad can overlay heat-maps upon the input image to highlight relevant visual cues associated with the outcome of interest.³⁵ For example, the heatmap explainability methods of a peripheral blood smear image may highlight pathognomonic Auer Rods for the accurate diagnosis of acute promyelocytic leukemia (Figure 3). More complex methods such as Generative Adversarial Networks are architectures trained to generate synthetic images, which can create representations of a particular class or outcome.³⁷

Metrics

Common performance metrics for the evaluation of DL classification models include Area Under the Receiver Operator Curve (AUROC), sensitivity, specificity, and accuracy. The AUROC represent the tradeoff between true and false positive rates for a binary model along a range of possible threshold values. AUROC values nearing 1.0 represent a model with perfect discriminatory power, and values tending towards 0.5 perform no better than random chance.

For segmentation tasks, the Sørensen-Dice similarity coefficient (Dice) represents the overlap between the predicted area of interest with the ground truth, where a Dice coefficient of 1.0 represents ideal predictive overlap. Other segmentation metrics include the similarly defined Jaccard index, also known as Intersection over Union (IoU).

Literature review for clinical application of deep learning in hematologic conditions

A Boolean query was submitted to PubMed to extract articles created between January 1, 1990, and August 1, 2022. Search terms included both a “deep learning” and a “hematology” specific term (Figure 4A). The query resulted in 2,708 initial articles. Further refinement by manual review by one author excluded a large number of articles (Figure 4B), resulting in 65 manuscripts. General trends and findings of the resulting articles are described in the context of how DL has been utilized to enhance phases of clinical care within various hematologic conditions, including task automation, detail optimization, disease detection, differential diagnosis, disease classification, risk prediction, complication assessment, therapy response, and survival prediction (Figure 5). General considerations for critical appraisal of the following manuscripts include performance metrics, use of external or prospective validation cohorts, use of explainability methods, and comparison with human expert performance (Tables 1-4).

Figure 5.Deep Learning applications within 65 malignant and non-malignant hematology manuscripts. Applications are divided into separate phases of clinical care, including task automation, detail optimization, disease detection, differential diagnosis, disease classification, risk prediction, complication assessment, therapy response, and survival/relapse prediction. Image domains include radiological, pathological, and other atypical image types such as electrocardiograms or funduscopic exams. Specific image modalities are detailed in Tables 1-4. CBC: complete blood count; MDS: myelodysplastic syndromes; MPN: myeloproliferative neoplasms; RBC: red blood cell; VTE: venous thromboembolism; WBC: white blood cell.

Table 1.Summary and performance metrics of deep Learning applications in hematology for task automation and detail optimization.

Table 2.Summary and performance metrics of deep Learning applications in hematology for disease detection.

Table 3.Summary and performance metrics of deep Learning applications in hematology for differential diagnosis and disease classification.

Table 4.Summary and performance metrics of deep Learning applications in hematology for advanced phases of patient care, including risk prediction, complication assessment, therapy response, and survival/relapse prediction.

Task automation

Routine clinical workflows in pathology and radiology may involve repetitive actions. Automation models can be developed to increase efficiency and decrease physician burden for tasks such as counting cell types in peripheral blood smears or contouring the borders of suspicious lesions on imaging. For pathology workflows, DL models trained to contour white blood cell (WBC) borders in peripheral blood smears were highly effective with near perfect Dice co-efficients in multiple cohorts.³⁸ Automatic detection of cells can be put through downstream analyses and provide an automated cell count, for which DL-based methods achieve high accuracy.³⁹

In addition, chromosomal analyses are standard for diagnosis and prognostication for multiple hematologic malignancies. Manual segmentation and rotation of digital karyograms is time-consuming, but automated models can significantly expedite throughput.⁴⁰ In radiology workflows, contouring suspicious lesions or organs can help characterize downstream parameters such as volume, width, and avidity. Hypermetabolic lesions on PET/CT have been localized with DL algorithms for multiple adult and pediatric lymphomas or multiple myeloma lesions.^41,42 Segmentation metrics were reportedly high, with Dice coefficient 0.86-0.98 among various lymphomatous conditions.^43-45 For other conditions, the automated volume calculation of particular regions of interest have been explored in myeloproliferative neoplasms (MPN) for spleen volume,⁴⁶ as well as clot burden quantification for new pulmonary emboli.⁴⁷

Detail optimization

For expert diagnosticians, image quality is critical for the identification of disease. Using U-Net architectures, DL-enhanced images may improve user readability and potentially reduce the amount of toxic contrast material given to patients. Enhancement of peripheral blood images to assess red blood cell (RBC) aberrations have yielded promising results. For sickle cell disease, mobile-device photos of peripheral blood have been digitally upscaled to match laboratory microscope quality; upon further validation, the upscaled images retained relevant visual cues with near-perfect classification.⁴⁸ However, similar attempts to detect malaria RBC inclusion were less successful, noting that CNN-based enhancement of peripheral blood images was insufficient to resolve parasites that were not already easily distinguishable at low resolution.⁴⁹ Multiple optimization efforts in radiology have investigated whether DL can improve image quality from lower-contrast images, which may help spare patients from nephrotoxic or radioactive risks. For both positron emission tomography/magnetic resonance imaging (PET-MRI) in lymphomatous conditions and computed tomography (CT) scans in multiple myeloma, authors have concluded that reduced contrast volumes may be feasible while still maintaining diagnostic quality.⁵⁰, ⁵¹

Disease detection

In clinical practice, a common initial diagnostic step for hematologic disorders is the analysis of peripheral blood to observe morphologic abnormalities of RBC, WBC, and platelets. The detection of structural RBC aberrations can identify certain infectious diseases and hemoglobinopathies. In endemic areas of malaria, the Plasmodium parasites are often identified by light microscopy as RBC inclusions. Multiple DL initiatives report high accuracy and good model performance for the diagnosis of malaria from peripheral blood in both cross-validated and external cohorts.^52-56 Other RBC aberrations, such as hemoglobin H inclusions in α-thalassemia, can be detected by DL with appropriate peripheral blood staining protocols.⁵⁷ With regards to transfusion medicine needs, the quality and degradation of RBC products prior to transfusion can also be determined with DL methods. Using explainability techniques, Doan et al. explored their proposed autoencoder network trained on RBC images to identify novel features associated with poor storage quality RBC products.⁵⁸ For certain disorders, the detection of aberrant WBC morphologies from peripheral blood is paramount. DL algorithms consistently detect dysplastic neutrophils pathognomonic for myelodysplastic syndrome (MDS),⁵⁹ as well as other white blood precursors to aid in the diagnosis of MPN,⁶⁰ acute promyelocytic leukemia (APL),¹² or acute lymphoblastic leukemia (ALL).^61,62 Many DL models for WBC detection have performed with high accuracy and AUROC upon internal validation strategies. If translated into clinical practice, DL models for peripheral blood assessment may expedite critical diagnoses which necessitate emergent therapy, such as APL.

Particularly for myeloid malignancies, bone marrow assessment is usually needed to establish a diagnosis. DL can detect particular cellular morphologies of neutrophils, megakaryocytes, promyelocytes, and plasma cells associated with MDS,⁶³ MPN,⁶⁴ APL,¹³ and multiple myeloma,⁶⁵ respectively. Similarly for lymphoid malignancies, assessing lymph node architectures can aid the diagnosis of various lymphomas, such as diffuse large B-cell lymphoma (DLBCL)⁶⁶ or follicular lymphoma (FL).⁶⁷ DL models developed by Li et al. maintained high accuracy for the diagnosis of DLBCL from lymph biopsies across four separate institutional cohorts.⁶⁶ Furthermore, Syrykh et al. utilized the clinical challenge of differentiating follicular lymphoma from follicular hyperplasia to develop a novel DL method quantifying prediction uncertainty, which is not often reported in DL studies. With their uncertainty method, the authors report higher classification capabilities when only considering the newly categorized low-uncertainty images.⁶⁷

In addition to pathologic analysis, clinical guidelines commonly suggest radiologic assessment for the initial workup of suspected malignancy or thrombosis. Using PET/CT images, DL models exhibit high classification of the hypermetabolic lesions for DLBCL diagnosis.⁶⁸ However, similar attempts using PET/CT images of mantle cell lymphoma (MCL) patients are challenged with tradeoffs between sensitivity and false positive rates for diagnosis in external cohorts.⁶⁹ For select non-malignant conditions, multiple studies explored DL for the expedited and more affordable diagnosis of pulmonary emboli (PE) and deep vein thromboses (DVT), for which a diagnosis may require immediate intervention.^70-72 Huang et al. integrated clinical data in conjunction with CT scans to improve their DL model for PE detection. The authors report that multi-modal models exhibit higher classification performance than image-only DL models.⁷¹ In addition, automated detection of common thrombotic conditions may reduce the financial burden, with cost analyses revealing positive financial benefit to health care systems.⁷²

Finally, a particularly novel use of DL is the prediction of disease from imaging modalities beyond standard pathologic or radiologic domains. Multiple studies have shown that anemia can be detected with high accuracy utilizing DL on atypical modalities such as electrocardiograms (ECG)⁷³ or funduscopic examinations.⁷⁴ Both authors have implemented explainability methods to reveal features associated with anemia, such as QRS complexes in ECG or optic disk aberrations in funduscopic images. Thus, screening for anemia may offer a low-cost benefit for patients already undergoing these common examinations.

Differential diagnosis

Various hematologic conditions share similar features and presentations, posing challenges in providing a definitive diagnosis in clinical scenarios where radiologic findings may be non-specific and pathological morphologies may be subtle. Differentiating among possible diagnoses is a common clinical task, and various approaches of DL have been explored as a potential means to increase objectivity towards a true diagnosis. For example, Li et al. used transfer learning to pre-train their model with images of common household objects, such as bananas, rings, and pears to learn the analogous morphologies of similarly-shaped RBC inclusions of Toxoplasma, Plasmodium, and Babesia.⁷⁵

Interestingly, DL models have been shown to better extract subtle features for disease differentiation than can be assessed by humans. Cytopenias can be a common presentation for either MDS or aplastic anemia (AA) patients. Though either diagnosis typically requires bone marrow biopsy assessment, Kimura et al. trained a DL model on peripheral blood images to accurately differentiate between the two conditions.⁷⁶ Newly diagnosed leukemia patients commonly present with blasts in peripheral blood. The categorization of blasts into either myeloid or lymphoid lineages requires identifying cell-surface markers by flow cytometry; thus, visualization of blasts is not usually sufficient for classification. Similarly, lymphoma histology share visual commonalities and require immunohistochemical staining of cell-surface markers on biopsy specimens. To address these classification challenges, DL algorithms reportedly differentiate between acute myeloid leukemia (AML) and acute lymphoblastic leukemia (ALL) utilizing only peripheral blood or bone marrow images,^9,77,78 and similarly among various non-Hodgkin lymphomas (NHL) utilizing standard hematoxylin and eosin (H&E) lymph node biopsy images.^79-82

For patients with malignant brain lesions found on MRI imaging, clinicians may be tasked to differentiate between primary central nervous system lymphoma (PCNSL) and glioblastoma multiforme (GBM).⁸³ DL models for this revealed seemingly high initial performance but with a significant reduction to an AUROC of nearly 0.5 in external cohorts.⁸³ The problem of generalizing results highlights the continued need for critical appraisal of any newly-developed DL model across patient populations.

Disease classification

The classification of blood cells in standard peripheral blood smear review is a ubiquitous task useful in a broad array of diseases. The differential of WBC is necessary to stratify the likelihood of the malignant and non-malignant causes of WBC abnormalities. Numerous studies developed DL models as a single cell WBC classifier. Across the studies, performance remained robust, with the majority of studies achieving accuracies above 90% and explainability techniques highlighting sensitive cellular features.^11,84-86 However, validation upon external cohorts, which commonly reveal a lower performance,¹¹ is still needed prior to deployment in clinical practice. In addition to WBC classification, the categorization of RBC morphologies is useful within various anemias,⁸⁷ including sickle cell disease.⁸⁸ To explore platelet abnormalities, Zhou et al. developed a highly accurate DL model predicting the identity of agonists causing platelet aggregation using imaging flow cytometry.⁸⁹

Specific sub-classification of diagnoses is often necessary to guide prognostication, counseling, and therapeutic considerations. In numerous non-hematologic applications, previous DL models can accurately further categorize various cancers into genetic and clinical subtypes,⁴ which has led to similar explorations within leukemic and lymphomatous conditions. For leukemic classifications, ALL bone marrow images can be separated into the historically relevant French-American-British (FAB) classifications.⁹⁰ Furthermore, genomic subtypes may be accurately identified by DL models; Eckardt et al. identified NPM1 mutations among newly diagnosed AML patients and characterized novel cellular morphological features that had not previously been reported.¹⁴ Broader DL efforts to identify each clinically relevant molecular or cytogenetic abnormality have been attempted for MDS sub-classifications.⁹¹ For lymphoma, Swiderska-Chadaj et al. developed a DL model predicting MYC gene rearrangements in DLBCL patients using lymph node biopsy images. Though MYC rearrangement is typically assessed with ancillary fluorescent in situ hybridization, the DL model using only H&E images maintained high accuracy upon external cohorts.⁹²

Advanced stages of patient care

There are currently few examples of DL for the assistance of later stages of patient care, including risk prediction, complication assessment, therapy response, and survival prediction. For such tasks, the disease processes and image modalities are heterogenous. Risk has been assessed with CT images or digitalized bone marrow biopsies (BMB) for DLBCL outcomes. DL models predict the transformation of low-grade lymphomas to high-grade DLBCL using BMB images,⁹³ and, furthermore, known clinical risk factors such as sarcopenia can be extracted and quantified in CT images of DLBCL patients.⁹⁴ Risk in thrombotic conditions can be characterized automatically using DL classification of right ventricular strain in chest imaging for PE workup.⁹⁵ Cai et al. assessed complications of sickle cell disease by detecting sea fan neovascularization in funduscopic images, which is a vision-threatening complication warranting prophylactic management.⁹⁶ Doan et al. evaluated therapy response in ALL patients by using DL methods to detect residual lymphoblasts after receiving induction chemotherapy.⁹⁷ Finally, DL models for relapse prediction using baseline imaging have been developed for extranodal natural killer/T-cell lymphoma⁹⁸ and mantle cell lymphoma.⁹⁹ However, further evaluations upon external cohorts are needed for these advanced stage tasks.

Conclusions

The use of deep learning in hematologic conditions has attracted significant interest in recent years. As noted above, researchers have utilized multiple data structures including radiologic images, pathology specimens, clinical data, and atypical imaging such as funduscopic examinations to perform a variety of clinically relevant tasks. Most Authors reported high model performance for disease diagnosis, segmentation, and subtyping. Other studies explored tasks beyond human capabilities such as genomic inference and prognostication from imaging analysis alone. Few studies have used hematologic conditions as a means to implement state-of-the-art architectures to improve the field of DL in general. Compared to other clinical domains, DL in hematology is still in its infancy, so it is not widely used in clinical practice. As such, the intention of this review is to introduce broad concepts to hematology clinicians to assist in the evaluation and understanding of future DL implementations, as well as to provide an overview of the clinical uses currently being explored throughout patient care.

The fact that it is still early days for DL in hematology may be due to a lack of appropriate algorithm design, data availability, computational resources, and insufficient disease-specific expertise involved in DL development.¹⁰⁰ To the best of our knowledge, there are still no large clinically-annotated multi-modal public datasets for many hematologic conditions. In addition, critical morphological information in hematopathology may only be available at higher magnification levels, surpassing the limits of standard pathology scanners. Although these structural barriers continue to compromise the development of DL in hematology, rapid technological advances continue, and interest for DL within the academic community is growing.¹⁰¹

Though promising, the methods and conclusions from the numerous studies are heterogenous and challenging to compare. As yet, there is no standardized approach in DL research, reporting, or implementation. In the present overview, the majority of publications were evaluated by internal validation strategies, with the minority evaluated on external institution cohorts. Explaining model predictions were not ubiquitous, and few DL models were compared directly against human evaluation. Major government initiatives currently aim to standardize DL protocol design,¹⁰² and, despite the variance in outcome reporting in DL analyses, the SPIRIT-AI, STARD-AI, and CONSORT-AI initiatives aim to standardize future clinical trial design and reporting of artificial intelligence interventions.^103-105

The research and results of DL analyses must be interpreted cautiously, as a number of practical and ethical issues have arisen in other domains of machine learning. CNN are prone to “memorize” the training set; thus, the initial high performance may fail to be carried forward on new previously unseen data. For this reason, it is imperative to evaluate DL models on external cohorts from separate institutions. If training data are acquired from multiple institutions, care must be given to correct for known “batch effects,” as DL models may infer site-specific artifact signatures not related to the underlying disease biology.¹⁰⁶ Similarly, researchers should investigate explainability and error analysis to ensure that the models rely on scientifically reasonable features and ignore irrelevant factors. In addition, uncertainty in model predictions are rarely reported but are arguably necessary for clinical implementation of DL algorithms.

In this review, the majority of DL applications are aimed towards earlier phases of clinical care, such as automation and disease detection. DL in lymphoma resulted in the plurality of exploratory analyses, likely due to the importance of both radiologic and pathologic findings in the care of lymphoma patients. Though explored in a myriad of malignant and non-malignant conditions, notably lacking are DL applications in stem cell transplantation and many other non-malignant processes where morphological assessment is paramount, such as thrombotic microangiopathies.

Future work is needed to address large scale applications of DL in hematology. As a hematopathologist typically assesses histology specimens at different magnification levels, customized architectures to implement multi-scale image analysis should be explored. DL in solid oncology is widely used, in part due to the publicly available digital biopsy specimens provided by The Cancer Genome Atlas,¹⁰⁷ of which there is no analogous database for hematologic conditions. In addition, the combination of multi-modal data structures that incorporate images in concert with flow cytometry, molecular analyses, cytogenetics, or other clinical factors may provide additional relevant features to improve DL models. While numerous considerations remain before large-scale implementation of DL is feasible, the development of new models and applications in hematology is rapidly increasing, and it is imperative for clinicians to be aware of the opportunities that DL may provide.

Footnotes

• Received June 3, 2022
• Accepted January 18, 2023

Correspondence

AT. Pearson apearson5@medicine.bsd.uchicago.edu

Disclosures

ATP reports support via grants from NIH/NCI U01-CA243075, grants from NIH/NIDCR R56-DE030958, grants from Horizon 2021-SC1-BHC, grants from DoD Breakthrough Cancer Research program BC211095, and grants from SU2C (Stand Up to Cancer) Fanconi Anemia Research Fund – Farrah Fawcett Foundation Head and Neck Cancer Research Team Grant during the conduct of the study. ATP reports grants from Abbvie via the UChicago – Abbvie Joint Steering Committee Grant, and grants from Kura Oncology. ATP reports personal fees from Prelude Therapeutics Advisory Board, from Elevar Advisory Board, from Ayala Advisory Board, personal fees from Abbvie Advisory Board, and from Privo Advisory Board outside of submitted work. MES is an employee of Sonic Healthcare. AS has no conflicts of interest to disclose.

Contributions

All authors conceived the manuscript. AS wrote the manuscript. All authors approved the final version.

Data-sharing statement

No applicable.

Funding

ATP reports this study has been funded in whole or in part with Federal funding by the NCI-DOE Collaboration established by the U.S. Department of Energy (DOE) and the National Cancer Institute (NCI) of the National Institutes of Health, Cancer Moonshot Task Order N. 75N91019F00134 and under Frederick National Laboratory for Cancer Research Contract 75N91019D00024. This work was performed under the auspices of the U.S. Department of Energy by Argonne National Laboratory under Contract DE-AC02-06-CH11357.