(19)
(11)EP 3 889 893 B1

(12)EUROPEAN PATENT SPECIFICATION

(45)Mention of the grant of the patent:
30.03.2022 Bulletin 2022/13

(21)Application number: 21169411.2

(22)Date of filing:  18.05.2017
(51)International Patent Classification (IPC): 
G06T 7/00(2017.01)
(52)Cooperative Patent Classification (CPC):
G06T 7/0012; G06T 2207/30096; G06T 2207/10016; G06T 2207/20076

(54)

TUMOR PROXIMITY MEASURE

TUMORPROXIMITÄTSMESSUNG

MESURE DE PROXIMITÉ DE TUMEUR


(84)Designated Contracting States:
AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

(30)Priority: 18.05.2016 EP 16170224
29.11.2016 EP 16201272

(43)Date of publication of application:
06.10.2021 Bulletin 2021/40

(62)Application number of the earlier application in accordance with Art. 76 EPC:
17725929.8 / 3459046

(73)Proprietor: F. Hoffmann-La Roche AG
4070 Basel (CH)

(72)Inventors:
  • GAIRE, Fabien
    68440 Schlierbach (FR)
  • KLAIMAN, Eldad
    82319 Starnberg (DE)
  • KORSKI, Konstanty
    4058 Basel (CH)

(74)Representative: Richardt Patentanwälte PartG mbB 
Wilhelmstraße 7
65185 Wiesbaden
65185 Wiesbaden (DE)


(56)References cited: : 
WO-A1-2015/189264
US-A1- 2016 123 964
US-A1- 2015 133 321
  
  • JULIANE M KRÜGER ET AL: "Combat or surveillance? Evaluation of the heterogeneous inflammatory breast cancer microenvironment", THE JOURNAL OF PATHOLOGY, vol. 229, no. 4, 15 February 2013 (2013-02-15), pages 569-578, XP055181126, ISSN: 0022-3417, DOI: 10.1002/path.4150
  
Note: Within nine months from the publication of the mention of the grant of the European patent, any person may give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention).


Description

Field of the invention



[0001] The invention relates to the field of image analysis, and more particularly to the field of image-based tumor classification.

Background and related art



[0002] Microsatellite instability (MSI) is the condition of genetic hypermutability that results from impaired DNA mismatch repair (MMR). The presence of MSI represents phenotypic evidence that MMR is not functioning normally and thus is an indicator of the genetic state of a cell considered as a potential tumor cell.

[0003] Microsatellite instability is known to be associated with colon cancer, gastric cancer, endometrium cancer, ovarian cancer, hepatobiliary tract cancer, urinary tract cancer, brain cancer, and skin cancers. MSI is most prevalent as the cause of colon cancers. Each year, there are over 500,000 colon cancer cases worldwide. Based on findings from over 7,000 patients stratified for MSI-High (MSI-H), MSI-Low (MSI-L), or Microsatellite Stable (MSS) colon cancers, those with MSI-H had a more positive prognosis by 15% compared to MSI-L or MSS tumors.

[0004] MSI is a good marker for determining a prognosis for cancer treatments. Additional approaches which are based on genetic information of a tumor exist. For example, extensive research is ongoing to decipher the underlying genetic patterns of tumor tissue cells with the hope to improve early cancer diagnosis and treatment. The recent progress in next generation sequencing technologies has revolutionized the field of cancer genomics. However, one caveat of these studies remains the large amount of genetic variations identified and their interpretation.

[0005] Patent application WO 2015/189264 A1 describes systems and computer-implemented methods for assessing a risk of cancer recurrence in a patient based on a holistic integration of large amounts of prognostic information for said patient into a single comparative prognostic dataset. A risk classification system may be trained using the large amounts of information from a cohort of training slides from several patients, along with survival data for said patients. A machine-learning-based binary classifier in the risk classification system may be trained using a set of granular image features computed from a plurality of slides corresponding to several cancer patients whose survival information is known and input into the system.

[0006] Patent application US 2016/123964 A1 describes a method of analyzing a biological sample from a subject that has a tumor or cancer. The method comprises: determining, for target cells having a phenotype of interest spatial resolution of the target cells, density of the spatially resolved target cells in the sample; and proximity between spatially resolved target cells of interest in the sample; and determining an overall score based at least in part on the preceding parameters. A method for identifying a patient as a responder to single agent antiPD-1 or anti -PD-L1 therapy is provided. Similar methods are provided for detecting adaptive immune resistance, the presence of cancer in a patient sample, determining efficacy of cancer therapy, and determining response to and monitoring the efficacy of cancer therapy.

[0007] Patent application US 2015/133321 A1 describes the characterization of biological samples. Acquired image data of the sample may be used to segment the images into epithelial and stromal regions to characterize individual cells in the sample based on the binding of the probes. Further, the biological sample may be characterized by a distribution, location, and type of a plurality of the characterized cells.

[0008] Juliane M. Kruger et al. in "Combat or surveillance? Evaluation of the heterogeneous inflammatory breast cancer microenvironment", published in the Journal of Pathology, vol. 229, no. 4,15 February 2013, pages 569-578 (XP055181126) describes a supervised learning algorithm that clustered image areas according to lymphocytes abundance, grouping behavior and distance to tumor cells. Recurrent infiltration patterns reflecting different grades of direct interaction between tumor and immune effector cells were identified. The approach provided comprehensive visual and statistical assessment of the inflammatory tumor microenvironment and allowed quantitative estimation of heterogeneous immune cell distribution.

Summary



[0009] It is an objective of the present invention to provide for an improved method and image analysis system for classifying tumor cells as specified in the independent claims. Embodiments of the invention are given in the dependent claims. Embodiments of the present invention can be freely combined with each other if they are not mutually exclusive.
  • In one aspect, the invention relates to an image analysis method for tumor classification as specified in claim 1.


[0010] Said approach for classifying the identified tumor cells into cells of an inflammatory vs. non-inflammatory tumor may be advantageous for multiple reasons:
The distance of immune cells and tumor cells appears to more accurately reflect cellular processes related to the immune response of an organism against a tumor than, for example, the MSS/MSI status. Thus, the proximity measure appear to more accurately reflect the inflammatory status of a tumor (an inflammation is- very generally speaking - the body's immune system's response to a stimulus) than a genetic approach. It has been observed that at least in some cases inflammatory versus non-inflammatory tumor were correctly distinguished based on the proximity measure which were not correctly distinguished based on their MSS/MSI status.

[0011] The high mutational load in MSI tumors creates many tumor-specific neoantigens, typically 10-50 times that of MSS tumors. Some of these neoantigens will be processed, presented on MHC, and recognized as foreign by T cells. The high neoantigen burden might be one explanation for the high level of tumor-infiltrating lymphocytes ("inflammatory tumor state") in MSI tumors. MSI tumors have a better prognosis than MSS tumors and respond better to several classes of drugs which boost or modulate the immune response. By determining a proximity measure based on the individual distances of tumor cells and immune cells, a more accurate classification of inflammatory vs. non-inflammatory tumors can be made and/or a more accurate prognosis and treatment recommendation (whether or not to prescribe a drug that boosts or modulates the immune response) can be given.

[0012] According to a further beneficial aspect, the proximity measure can quickly be computed from a digital image of a tumor tissue sample having been stained with an immunofluorescence assay. Thus, an additional time consuming and expensive (and-as it appears-less accurate) MSS/MSI status analysis of tumor cells may be avoided.

[0013] In a further beneficial aspect (compared to e.g. to various immune score computations), and increased accuracy of tumor classification and prognosis may be achieved, because distances between individual immune cells and tumor cells are determined. Instead of roughly determining the outline of a tumor region and then determining whether an immune cell is contained in said tumor region or not, embodiments of the invention use individual immune cell-tumor cell distances as a basis for computing a proximity measure from all said determined distances. This proximity measure is a more accurate indication of whether immune cells and tumor cells are located sufficiently close to each other to allow for an effective immune response against the tumor cells. In other words, the proximity measure for example indicates whether a sufficiently large number of immune-promoting immune cells such as cytotoxic T cells, B cells, memory cells, T-helper cells and/or macrophages are within an "immunologically effective distance" from the tumor cells.

[0014] An "immunologically effective distance" as used herein is a distance between an immune cell and its nearest tumor cell which is sufficiently small to allow for the killing of said tumor cell by said immune cell. The killing of the tumor cell may be performed directly or indirectly and may be performed by different pathways which may depend on the type of immune cell. For example, the killing of the tumor cell may be achieved via direct cell-cell interaction between the immune cell and the tumor cell, via a direct or signal-molecule based interaction of said immune cell with a further immune cell in the proximity of the tumor cell whereby the further immune cells kills or triggers the apoptosis of the tumor cell, or by triggering the apoptosis of the tumor cell (e.g. via direct cell-cell interaction or via signal molecules secreted by the immune cell).

[0015] In a further beneficial aspect, the distance information may not only be used as an input for accurately classifying tumor as an inflammatory versus a non-inflammatory tumor, the distance information may also be used for generating a graphical output that may allow a pathologist to easily and accurately identify the location and distribution of inflammatory regions within a tumor tissue.

[0016] In a further beneficial aspect, once the individual immune cells and tumor cells have been identified, the distance computation for individual pairs of neighboring tumor cells and in immune cells may be performed very quickly and efficiently. As the computed distances on numerical values, also the proximity measure computation as a function of a plurality of numerical values may be performed quickly and with comparatively low computational effort even in case in the depicted tissue sample comprises many thousand cells.

[0017] In a further aspect, the invention relates to an image analysis method for tumor classification as specified in claim 2.

[0018] Thus, according to said alternative approach, at first the tumor cells and corresponding tumor regions are identified in the digital image. Then, immune cells in said tumor regions or within close spatial proximity to said tumor regions are identified and the immune cell - tumor cell distances are identified. In this scenario, the distances may be computed starting from the individual identified immune cells ("for each identified immune cell") or may be computed starting from the individual identified tumor cells (for each identified tumor cell"). For example, gray scale and color segmentation techniques, edge detection, voting and radial symmetry based image analysis techniques can be used for identifying and/or classifying cells and for identifying regions consisting of or mainly comprising similarly colored or shaped cells having been identified as tumor cells. Moreover, various machine learning techniques with supervised learning methods such as SVM (Support vector Machines), DNN (Deep Neural Networks), Random Forest, etc. can be used to classify cells into types and/or to identify tumor regions, non-tumor tissue regions and glass regions on a slide. Limiting the distance determination on the tumor and its boundary may ensure that in case a whole tissue slide comprises only a small tumor region and the majority of immune cells is located far away from the tumor, the effect of said distant immune cells (which may be present in said far-away regions irrespective of whether the tumor is inflammatory or not) does not modify the proximity measure.

[0019] According to embodiments, the image analysis method further comprises analyzing the received digital image for determining a tumor cell density of the identified tumor cells. The classification of the identified tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor comprises inputting the proximity measure and the tumor cell density into a classifier configured to perform the classification.

[0020] According to embodiments, the image analysis method further comprises analyzing the received digital image for determining an immune cell density of the identified immune cells or of a particular type of immune cells (e.g. T-cells, B-cells, macrophages, etc.). The classification of the identified tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor comprises inputting the proximity measure and the immune cell density into a classifier configured to perform the classification.

[0021] According to embodiments, the classification of the identified tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor comprises inputting the proximity measure, the tumor cell density and the immune cell density into a classifier configured to perform the classification.

[0022] According some embodiments, the determination of the tumor cell density comprises determining the tumor cell density for each tumor cell individually, e.g. by determining the tumor cell density in a predefined area centered around said tumor cell. In this case, each of the determined tumor cell density values can be fed as input into the classifier. Alternatively, an average value is computed from the individual density values and is input as a single value into the classifier. Likewise, the determination of the immune cell density comprises determining the immune cell density for each immune cell individually, e.g. by determining the immune cell density in a predefined area centered around said immune cell. In this case, each of the determined immune cell density values can be fed as input into the classifier. Alternatively, an average value is computed from the individual density values and is input as a single value into the classifier.

[0023] Using cell density information as an additional input parameter of the classifier may be advantageous as the accuracy of the classification may significantly be increased.

[0024] According to embodiments, the displaying of the classification result comprises representing all identified tumor cells as first pixel blobs having a first color and representing all identified immune cells whose distance to its nearest tumor cells is below a threshold as second pixel blobs having a second color. Then, the image analysis system displays the second pixel blobs as an overlay of the first pixel blobs.

[0025] For example, the tissue sample may be a biopsy sample of colorectal cancer, the identified immune cells may be cytotoxic T cells and the distance threshold may be 35 µm. All cells having been identified as tumor cells may be colorized with blue color and may correspond to a first image layer. All cells having been identified as immune cells whose distance to its nearest tumor cell is below 35 µm are colorized with orange color and may correspond to a second image layer. The second image layer is superimposed on the first image layer with the consequence that the orange "tumor affine" immune cells cover and hide at least some of the tumor cells. In this case, orange regions in the tumor tissue indicate inflammatory regions. If the tissue section represented in the generated overlay-image comprises only few orange regions, the immune response is predicted to be only weak or is predicted not to significantly affect tumor development. The patient in this case is predicted not profit from the application of drugs boosting or modulating the immune response because the immune cells are too far away from the tumor cells as to be able to kill them. Thus, a user of pathologist can immediately and easily derive from such an overlay image if a particular tumor tissue is inflammatory or not and/or whether the treatment of the patient with substances known to boost or modulate an immune reaction will be effective in treating this tumor or not.

[0026] According to embodiments, the tissue sample is a whole slide tissue sample and the digital image is a whole slide image. Typically, such whole slide tissue samples comprise several thousand or even 10,000s of cells. Thus, the proximity measure may be a function of the distances of many thousand neighboring immune cell-tumor cell pairs.

[0027] According to embodiments, the identification of the tumor cells comprises identifying proliferating non-lymphoid cells and using said identified cells as the tumor cells.

[0028] For example, proliferating cells may be identified by using generic proliferation markers such as fluorescent stained antibodies binding to the KI67 protein or to the PCNA protein. Non-lymphoid cells are identified, for example, as cells which do not express typical immune cell markers such as CD3. Thus, any cell having the expression profile KI67+/CD3- is identified as a proliferating non-lymphoid cells and thus as a tumor cell. This approach may beneficial as it provides a very generic way of identifying tumor cells that is applicable for many different types of tumors and for many different types of tissues.

[0029] In addition, or alternatively, the identification of the tumor cells comprises identifying cells expressing a (specific) set of one or more tumor-specific biomarkers and using said identified cells as the tumor cells.

[0030] For example, a cytokeratin profile (data indicating whether a particular set of cytokeratins is expressed in a cell or not) of the cells in the tissue sample is determined, e.g. by using fluorescence stains selectively staining the respective cytokeratins. The obtained cytokeratin profile is then compared by the image analysis system against known cytokeratin profiles of various tissues such as liver, lung, colon and so on. In case the cytokeratin profile of a cell in the tissue sample depicted in the image differs from the typical cytokeratin profile of the tissue from which the tissue sample was taken, the cells having this "deviant/untypical" cytokeratin profile are automatically or semi-automatically identified as tumor cells.

[0031] Cytokeratins are proteins of keratin-containing intermediate filaments found in the intracytoplasmic cytoskeleton of epithelial tissue. For example, basic cytokeratins CK1, CK2, CK3, CK4, CK5, CK6, CK7, CK8 and acidic cytokeratins CK9, CK10, CK12, CK13, CK14, CK16, CK17, CK18, CK19, and CK20 are known. Expression of these cytokeratins is frequently organ or tissue specific. Therefore, cytokeratin expression profiles can be used by anatomic pathologists to detect the presence and/or cell of origin of various tumors.

[0032] Depending on the type of tissue of the tissue sample and/or the type of the tumor suspected to be present in a patient, the presence (sufficiently high expression level) of other tumor-specific biomarkers may be determined, e.g. the presence of the Her2 protein fur breast cancer.

[0033] According to embodiments, the identification of the immune cells comprises or consists of identifying cytotoxic T-cells and using the identified cytotoxic T-cells as the identified immune cells. For example, the CD8 protein that is specifically expressed in cytotoxic T-cells is used as a biomarker. A stain that selectively binds to the CD8 protein in the tissue sample is used and an image whose pixel intensity values are indicative of the presence of the CD8 protein is taken from the stained tissue sample. An image analysis operation is performed on the CD8 biomarker related staining and intensity signal, e.g. a threshold-based blob detection operation, and the corresponding pixel blob is identified as CD8+ cell and thus as a cytotoxic T-cell.

[0034] According to other embodiments, CD8+ and CD3 + cells (i.e., cells expressing both the CD8 protein and the CD3 protein) are identified as cytotoxic T-cells. The CD3 protein is a protein expressed on many different immune cells and is used as a further biomarker that helps to ensure that a particular blob is really an immune cell and not, for example, an artifact created by an unspecific staining reaction of the antibody used for staining the CD8 protein.

[0035] In addition, or alternatively, the identification of the immune cells comprises or consists of identifying T-helper-cells and using the identified T-helper-cells as the identified immune cells. For example, CD4 expressing cells ("CD4+ cells") are identified as T-helper cells. According to other examples, cells expressing both the CD4 and the CD3 protein ("CD4+/CD3+ cells") are identified as T-helper-cells. Again, CD3 protein expression was used as a further indicator to make sure a blob is really an immune cell, not a blob of an unspecific stain. Instead of or in addition to the CD4 marker, the T-bet marker may likewise be used for identifying T-helper cells.

[0036] In addition, or alternatively, the identification of the immune cells comprises or consists of identifying memory cells and using the identified memory cells as the identified immune cells. For example, CD45RO expressing cells ("CD45RO+ cells") are identified as memory cells.

[0037] In addition, or alternatively, the identification of the immune cells comprises or consists of identifying B- cells and using the identified B cells as the identified immune cells. For example, CD20 expressing cells ("CD20 + cells") are identified as B-cells.

[0038] In addition, or alternatively, the identification of the immune cells comprises or consists of identifying activated T- cells and using the identified activated T- cells as the identified immune cells. For example, CD137 expressing cells ("CD137+ cells") are identified as activated T-cells.

[0039] In addition, or alternatively, the identification of the immune cells comprises identifying PD1+ immune cells and using the identified PD1+ immune cells as the identified immune cells. In addition, the identification of the tumor cells comprises identifying PDL1+ tumor cells and using the identified PDL1+ tumor cells as the identified tumor cells. A PDL1+ tumor cell is a tumor cell that expresses the PDL1 protein. For example, as mentioned before, a tumor cell can be identified e.g. as a cell expressing the KI67 protein and/or expressing the PCNA protein and not expressing a CD3 protein. Thus, a PDL1+ tumor cell would be a cell having the expression profile PDL1+/KI67+/CD3-. Alternatively, any other biomarker indicating that a cell is a tumor cell (e.g. panCK+ or a particular cytokine profile that deviates from the profile of the surrounding tissue) can be used for identifying tumor cells.

[0040] A cell that "expresses" a biomarker BM ("BM+ cell") as used herein refers to a cell expressing said biomarker at least in a minimum amount. For example, the question if said minimum amount is generated can be determined by comparing an intensity value on a respective digital image with an intensity threshold value. Said threshold value may be specific for the biomarker or stain used. Depending on the type of biomarker examined, a cell that "expresses" a biomarker can also be a cell that expresses a greater amount of said biomarker than a "typical" cell of said cell type or than another reference cell type. In this case, the label "BM+ cell" refers to a cell that over-expresses the biomarker BM (relative to a reference expression level).

[0041] Programmed death-ligand 1 (PDL1) is a 40kDa type 1 transmembrane protein that plays a major role in suppressing the immune system during particular events such as pregnancy, tissue allografts, autoimmune disease and other disease states such as hepatitis. Normally the immune system reacts to foreign antigens by triggering proliferation of antigen-specific CD8+ T cells.

[0042] Programmed cell death protein 1 (PD1) is a cell surface receptor that belongs to the immunoglobulin superfamily and is expressed on T cells and pro-B cells. PD1 binds two ligands, PDL1 and PD-L2. PD1 is an immune checkpoint and plays an important role in down regulating the immune system by preventing the activation of T-cells. PD1 is expressed on the surface of activated T cells, B cells, and macrophages.

[0043] The formation of PD1 / PDL1 ligand complex transmits an inhibitory signal on the immune response. It appears that some cancers upregulate PDL1 expression, thereby managing to evade the host immune system. High expression of PDL1 by tumor cells has been observed to be associated with increased tumor aggressiveness and an increased risk of death.

[0044] By computing the proximity measure in by using the above mentioned PDL1/PD1 biomarkers, it is possible to determine if the immune cells lying within an immunologically effective distance from the tumor cells express PD1 receptor protein or not and to determine if the tumor cells (over)express the PDL1 protein.

[0045] In addition, or alternatively, the identification of the immune cells comprises or consists of identifying macrophages and using the identified macrophages as the identified immune cells. For example, cells expressing both the CD163 and the CD68 protein ("CD163+/CD68+ cells") are identified as macrophages. Instead of or in addition to the CD163 marker, the CSF1R marker may likewise be used for identifying macrophages.

[0046] It has been observed that automated full slide analysis of a cohort of slides stained with an immunofluorescence assay targeting the biomarkers CD3, CD4, CD8 and KI67 reveals that metrics of CD3+/CD8+ and CD3+/CD4+ immune cell distribution and proximity to KI67+ proliferating non-lymphoid cells (CD3-/KI67+) in the tumor area can serve as a strong indicator for immune response in the tumor area. By computing a proximity score that is based on individual tumor cell - immune cell distances, a particularly accurate tumor classification method is provided that is more accurate than state of the art MSS/MSI state based or immunoscore based classification schemes. Moreover, the proposed computation of a cell distance based proximity score and the use of the score for tumor classification has been observed in colorectal cancers to have a higher prognostic value than the AJCC/UICC TNM-classification.

[0047] Cytotoxic T-cells, memory cells, T-helper cells and macrophages are immune cells which are responsible or are able to initiate and/or boost an immune response. To the contrary, regulatory T-cells are often involved in the downregulation of the immune response. Thus, by selectively identifying those immune cells having a positive or stimulatory effect on the immune response, and by selectively identifying the distances of said positively stimulating immune cells, an even higher accuracy of the proximity score derived from said distances can be achieved: the presence of regulatory T-cells potentially having a negative impact on the immune response are ignored, thereby preventing the computation of a proximity score that indicates that many immune cells are in close proximity to tumor cells in a situation when predominantly regulatory T-cells but not macrophages, cytotoxic T-cells and/or T-helper cells are in close proximity to individual tumor cells.

[0048] According to embodiments, the identification of the immune cells comprises or consists of identifying immune cell of one or more immune cell types that boost an immune reaction (e.g. macrophages, cytotoxic T-cells, memory cells, B cells, and T-helper cells) and ignoring ("filtering out") immune cells of an immune cell type that suppresses or downregulates an immune reaction (e.g. regulatory T-cells). Thus, the totality of identified immune cells used for computing the proximity measure may consist of immune cells of the same or of multiple different immune cell types. In this case, for computing the proximity measure as a function (e.g. a histogram slope) of the distances, immune cell type specific functions (e.g. with immune cell type specific distance thresholds corresponding to immune cell type specific immunologically effective distances of said immune cell type) are used.

[0049] According to an example not covered by this invention, the immune cell ratios may be computed for each immune cell type individually and immune cell type specific ratios may be computed for immune cell type specific distance thresholds being identical to the immunologically effective distance of said immune cell type. Then, an average, e.g. the arithmetic mean of the immune cell type specific ratios may be computed as a final ratio and used as the proximity measure, for example. In case the immunologically effective distance of the different immune response boosting immune cell types is identical or very similar, the same function, e.g. the same distance threshold for ratio computation may be used for multiple or all immune cell types considered.

[0050] Computing a single proximity measure from multiple different immune cell types (using the same or immune cell type specific function of the tumor cell - immune cell distances) may be advantageous as a better and more complete data basis of the spatial relationship between relevant immune response promoting immune cells and their potential targets, the tumor cells, is provided and a more accurate classification into inflammatory and non-inflammatory tumor cells may be provided.

[0051] In addition, or alternatively, multiple individual proximity measures may be computed for multiple respective immune cell types and/or tumor cell types, e.g. for determining if a tumor of a tissue sample is a PDL1+ tumor and if a sufficient portion of the tumor cells is within an immunologically effective distance of a PD1+ immune cell. Said additional proximity measures may allow for a further sub-classification of the tumor and for a better prediction of the treatment schema.

[0052] According to embodiments, the identification of the immune cells comprises identifying regulatory T-cells and identifying immune cells of at least one type of immune cells that boosts the immune response and selectively using the identified immune cells that boost the immune response but not the identified regulatory T-cells as the identified immune cells. For example, cells expressing the FoxP3 protein ("FoxP3+ cells") are identified as regulatory T-cells and are ignored when the tumor cell - immune cell distances are determined. According to other embodiments, FoxP3+/CD3 + cells are identified as regulatory T-cells, whereby the expression of the CD3 biomarker is used as a further indicator to make sure a blob is really an immune cell, not a blob of an unspecific stain.

[0053] According to embodiments, the proximity measure is determined multiple times as described herein for embodiments of the invention, thereby respectively analyzing different digital images whose pixel intensities represent different biomarkers, the different biomarkers being indicative of immune cells of different types (e.g.: CD8 for cytotoxic T-cells, FoxP3 for regulatory T-cells, CD163 or CSF1R for macrophages, CD20 for B-cells, CD137 for activated T-cells, PD1 for some tub-types of T- and pro-B cells) and/or being indicative of tumor cells of different types (e.g. PDL1+/PDL1- tumor cells. Each of said multiple proximity measures may be used as a predictor of tumor cell class and/or prognosis and/or of a suitable treatment scheme.

[0054] Computing the proximity measure multiple times for different types of immune cell types and/or cancer cell types may allow to more accurately predict which ones of different classes of immune response modulating drugs will be effective in treating the tumor and which ones will not. By combining information contained in the spatial proximity of different immune cell and tumor cell types, the accuracy of classifying the tumor into inflammatory or non-inflammatory tumors may be increased and a classification into further sub-classes of tumors (e.g. PDL1+ or PDL1- tumors) may be enabled. In addition, the accuracy of a prediction if a particular drug class will be effective in treating the tumor may increase.

[0055] When computing multiple proximity measures for different immune cell sub-types and/or different tumor cell sub-types, also immune cells having an inhibitory effect on the immune system are taken into account. Although their presence within an immunologically effective distance from tumor cells may not indicate that the tumor is an inflammatory tumor, information on their spatial proximity to tumor cells may allow predicting whether applying a drug suppressing said inhibitory immune cell types may boost the immune response against the tumor.

[0056] According to embodiment, the image analysis system automatically outputs a prediction whether or not a particular class of drugs, will be effective in treating the cancer or not.

[0057] For example, one of said classes of drugs can be PD1 inhibitors. PD1 inhibitors activate the immune system to attack tumors and are used with varying success to treat some types of cancer by blocking or otherwise inhibiting the activity of the PD1 receptor. According to embodiment, the image analysis system is configured to predict that the PDL1 inhibitors will be effective only in case the proximity measure indicates that a sufficiently large fraction of PDL1+ tumor cells comprises at least one PD1+ immune cell within the predefined distance (immunologically effective distance). If the immune cells are not within an immunologically effective distance from the tumor cells or if the immune cells are located within said distance next a PDL1+ tumor cell but are not expressing PD1, the image analysis system is configured to predict that an anti-PDL1 drug will not to be effective in treating the tumor.

[0058] Likewise, other drug classes exist which boost or modulate an immune response and which specifically modulate the activity of individual immune cell types such as macrophages, T helper cells, B-cells, cytotoxic T-cells (T-cell engaging agents) and others. It may also be possible that said immune response modulating or boosting drugs are only effective in respect to specific tumor types. By automatically computing the proximity measure for multiple different immune cell types and/or tumor cell types, in a single step quantitative location information (cell-cell distance smaller than immunologically effective distance or not) as well as qualitative information on the cell classes is used as a basis for predicting whether or not a particular drug class will be effective in treating the tumor or not.

[0059] According to embodiments, the predefined distance is the "immunologically effective distance" of the immune cell. The immunologically effective distance is a maximum distance within which the identified immune cell is able to directly or indirectly trigger the killing of or the performing of apoptosis by the tumor cell.

[0060] According to embodiments, the predefined distance is larger than 15 µm.

[0061] According to embodiments, the predefined distance is in a range of 20 µm to 50µm, e.g. 35 µm.

[0062] For example, the tumor type may be colorectal cancer, the distance may be 35µm.

[0063] For other tumor types, the distance may slightly vary, but typically lies in the range of 20µm to 50µm.

[0064] According to embodiments related to the determination of the distances for each of the tumor cells, the computation of the proximity measure as a function of the determined distances comprises:
  • generating a histogram of the distances of the tumor cells to their respective nearest one of the identified immune cells; the histogram comprises at least two distance bins; the histogram covers a distance range of 0 µm to at least 50 µm (typically up to 80 µm), each of the bins corresponding to a bar of the histogram, each of the bars indicating a count of the identified tumor cells having a distance to their nearest immune cell that falls into said bin;
  • connecting the upper end of the first one of the bars with the upper end of the last one of the bars with a line; the line may be straight or curved; for example, the line may be a balance line; the first bar corresponds to the one of the bins covering the smallest distances; the last bar corresponds to the one of the bins covering the largest distances of the distance range;
  • determining the slope of the line; for example, the first bar may be located at the left boundary of the histogram, the last bar may be located at the right boundary of the histogram, and the slope may be determined of the line connecting the left-side, first bar with the right-side last bar;
using the slope as the proximity measure, wherein in case the slope indicates that the tumor cell count of the first bar is higher than the tumor cell count of the
last bar (i.e., is negative if the first bar is plotted to on the left side of the histogram), the classification result is that the identified tumor cells belong to an inflammatory tumor.

[0065] According to embodiments related to the determination of the distances for each of the immune cells, the computation of the proximity measure as a function of the determined distances comprises:
  • generating a histogram of the distances of the immune cells to their respective nearest one of the identified tumor cells, the histogram comprising at least two distance bins, the histogram covering a distance range of 0 µm to at least 50 µm, each of the bins corresponding to a bar of the histogram, each of the bars indicating a count of the identified immune cells having a distance to their nearest tumor cell that falls into said bin;
  • connecting the upper end of the first one of the bars with the upper end of the last one of the bars with a line, the first bar corresponding to the one of the bins covering the smallest distances, the last bar corresponding to the one of the bins covering the largest distances of the distance range;
  • determining the slope of the line;
  • using the slope as the proximity measure, wherein in case the slope indicates that the immune cell count of the first bar is higher than the immune cell count of the last bar, the classification result is that the identified tumor cells belong to an inflammatory tumor.


[0066] Computing a histogram slope as the proximity measure may have the advantage that the histogram that needs to be computed for determining the slope can be used as graphical output that visually represents the relative immune cell - tumor cell distribution in a tissue sample. The histogram may be output via a display device, thereby providing a user, e.g. a pathologist, with valuable information on the distance distribution of immune and tumor cells.

[0067] According to embodiments, the image analysis system outputs a signal selectively in case the classification result indicates that the identified tumor cells belong to an inflammatory tumor, wherein the signal is indicative of a treatment recommendation to use a substance that boosts or modulates an immune response as a drug for treating the tumor.

[0068] According to embodiments, the method comprises using the proximity measure in addition to or in replacement of an MSS-MSI status indicator for computing a prognosis of the responsiveness of the tumor to a substance that boosts or modulates an immune response.

[0069] According to embodiments, the identified tumor cells are colorectal cancer cells. However, the method and system according to embodiments of the invention are also applicable to other tumor types, e.g. breast cancer, pancreatic cancer, lung cancer, and the like.

[0070] According to embodiments, the identification of the immune cells comprises identifying immune cells of multiple different immune cell types by analyzing pixel intensity values in the at least one digital image, whereby the pixel intensities represent the presence of different immune cell type specific biomarkers, and using the totality of the identified immune cells of the multiple different immune cell types as the identified immune cells for computing the proximity measure.

[0071] According to embodiments, the tumor cells are identified as cells expressing at least a first tumor-type specific biomarker. The immune cells being identified as cells expressing at least a first immune cell type specific biomarker. The method further comprises:
  • analyzing the at least one received image or a further image of the tissue sample for identifying further immune cells as cells expressing a second immune cell type specific biomarker; and
  • analyzing the at least one received image or the further image of the tissue sample for identifying further tumor cells as cells expressing a second tumor type specific biomarker or using the identified immune cells as further tumor cells;
  • for each of the identified further tumor cells, determining the distance of said further tumor cell to the nearest further immune cell; alternatively, for each of the identified further immune cells, the distance of said further immune cell to the nearest further tumor cell is determined;
  • computing a further proximity measure as a function of said determined distances;
  • in dependence on the further proximity measure, sub-classifying the identified tumor cells and/or predicting the treatability of the tumor by a class of substances.


[0072] In a further aspect, the invention relates to an image analysis system for tumor classification according to claim 15.

[0073] Thus, the width of the boundary region around the tumor regions defines the area (tumor region(s) and tumor boundary region(s)) which shall be used for counting individual immune cell - tumor cell distances. Selectively identifying immune cells within the exterior outline of a boundary region around an identified tumor region may allow to ensure that the accuracy of the proximity measure is not negatively affected by including the immune cells that are normally distributed in tissue regions far away from a tumor which would never engage in an immune response against the tumor anyway. In addition, this may ensure that the accuracy of the proximity measure is not negatively affected by a too narrow boundary that would exclude immune cells in close proximity of the tumor whose position is key for predicting the inflammatory status of the tumor.

[0074] According to some embodiments, the width of the boundary region is determined for a particular cancer type empirically. The determination comprises: choosing an array of adjacent regions (e.g. circles, squares or rectangles having a predefined diameter, width and/or height e.g. in the range of 20µm to 40µm) in the image, the array of adjacent regions starting at a pixel on the tumor region outline ("starting region") and extending at least 200µm radially away from the tumor center into the non-tumor tissue region (the last one of the adjacent regions being referred to as "terminal region"; counting the number of immune cells in each of said adjacent regions; scanning the immune cell counts of the adjacent regions from the starting region in direction of the terminal region, thereby identifying the first one of the regions lying within a range of 120µm to 200µm away from the boundary of the tumor region and basically having the same immune cell count as all subsequently scanned regions. This region is considered to comprise a number of immune cells that is typical for the "normal" non-tumor tissue far away from any tumor tissue; and using the position of the identified region (e.g. a distance of any one of the pixel within said identified region from the pixel on the tumor region outline covered by the "starting region") as the boundary region width.

[0075] A "tumor" or "neoplasm" as used herein is an abnormal growth of tissue, and when also forming a mass is commonly referred to as a tumor. This abnormal growth usually but not always forms a mass.

[0076] An "inflammatory tumor" as used herein is a tumor against which a localized protective response by the immune system is observable. Said response may comprise the infiltration of plasma cells, lymphocytes, and eosinophils in the tumor.

[0077] A "non-inflammatory tumor" as used herein is a tumor where no or no significant response of the immune system against said tumor is observable.

[0078] An "immune cell" as used herein is a type of cell formed in the myelopoietic, lymphoid, and reticular portions of the reticuloendothelial system in various parts of the body, and normally present in those sites and in the circulating blood (rarely in other tissues). In particular, the immune cell can be a lymphocyte, e.g. T lymphocytes, B lymphocytes, and macrophages.

[0079] A proximity measure is a data value computed from a plurality of immune cell- tumor cell distances. For example, it can be a numerical data value, e.g. a slope of a line connecting histogram bins or a data value derived therefrom. The proximity measure may
indicate whether the majority of tumor cell - immune cell distances examined is shorter than a predefined distance threshold, e.g. the immunologically effective distance of an immune cell.

[0080] An "image analysis system" as used herein is an electronic system, e.g. a computer, configured for extracting meaningful information from digital images by means of digital image processing techniques. Image analysis tasks can comprise color deconvolution, connected component analysis and/or edge detection for identifying cells, for identifying the type of the cells (tumor or stroma cell, different types of immune cells) and distance measurements.

Brief description of the drawings



[0081] In the following embodiments of the invention are explained in greater detail, by way of example only, making reference to the drawings in which:

Detailed description



[0082] 
Figure 1
is a block diagram of an image analysis system;
Figure 2
is a flow chart of an image analysis method for classifying tumors;
Figure 3
depicts three overlay images derived from respective MSI tumor samples, the image comprising a first layer with tumor cells and a second layer with differently colored immune cells lying within a maximum distance from the nearest tumor cell;
Figure 4
depicts 8 overlay images derived from respective MSS tumor samples;
Figure 5
depicts three histograms derived from the three MSI tissue samples depicted in Fig. 3;
Figure 6
depicts 8 histograms derived from the 8 MSS tissue samples depicted in Fig. 4;
Figure 7
depicts three IHC fluorescent images of the three MSI tissue samples depicted in Fig. 3;
Figure 8
depicts 8 fluorescent images of the 8 MSS tissue samples depicted in Fig. 4;
Figure 9
depicts a comparison of the overlay image, the IHC fluorescent image and the histogram generated for a MSI sample (left column) and a MSS sample (right column); and
Figure 10
depicts a plot indicating the probability density that a particular immune cell lies within a given distance from its nearest tumor cell.
Figure 11
depicts the combination of proximity information and cell density information for capturing tumor context information for classifying the tumor.
Figure 12
depicts stained tissue samples of two patients having similar T-cell densities but different proximity measures.


[0083] Figure 1 is a block diagram of an image analysis system 100 according to an embodiment of the invention. The system comprises one or more processors 104, a main memory 106 and a non-volatile storage medium 108. The storage medium comprises one or more application programs or modules 110, 114, 112, 116 configured for performing one or more image processing tasks. For example, a first module 110 may perform a connected component analysis and edge detection routines in order to identify pixel blobs representing cells. The identification may be performed on different monochromatic fluorescent or brightfield microscopy images having been derived from the same tissue sample whereby the pixel intensities of the different monochromatic images respectively are indicative of a particular biomarker. Multiple monochromatic fluorescent images can be derived from a multispectral fluorescent image of a particular tissue sample by applying a color deconvolution algorithm.

[0084] The storage medium 108 may further comprise one or more digital images 118 of a tissue sample having been stained with one or more biomarker specific stains. Moreover, the system 100 is coupled to or comprises a display 102, e.g. an LCD display. The system uses the display 102 for displaying the digital images 118 of tissue samples of various patients (see for example figures 7 and 8), for displaying overlay images (as shown for example in figures 3 and 4) having been derived from various tissue images 118, for displaying a graphical representation of proximity measures (for example histograms depicted in figures 5 and 6) and/or for displaying a tumor classification result or treatment suggestion.

[0085] For example, a tissue sample, e.g. a biopsy sample, is used that has been stained by a plurality of biomarker specific stains. One or more of that stains selectively bind to strongly proliferating non-lymphoid cells or to proteins selectively expressed by tumor cells. Intensity signals of said stains and the corresponding monochromatic images can be analyzed by module 110 for identifying tumor cells. One or more further ones of said stains selectively bind to immune cells or specific subtypes of immune cells, e.g. cytotoxic t cells or T-helper cells. Intensity signals of said stains and the corresponding monochromatic images can be analyzed by module 110 for identifying immune cells. After having executed application program or module 110, tumor cells and immune cells within a tissue sample and the corresponding digital images have been identified.

[0086] Then, a distance measurements application program or module 112 uses the location of the pixel blobs identified as immune cells and tumor cells as input for determining the distance of tumor cells and their closest neighboring immune cells. For example, module 112 identifies the center of each cell and determines the distance between the respective cell centers. Alternatively, module 112 determines the distance as the distance between the outer boundaries of the two neighboring cells.

[0087] The totality of distances measured by module 112 is provided as input to the application program or module 114 which computes a proximity measure as a function of the totality of distances provided by module 112 for a particular tissue sample. The module 114 computes, for example a histogram for all identified distances between any of the tumor cells and its nearest immune cell neighbor and may compute a slope of the histogram bins as illustrated by figures 5, 6 and 9.

[0088] In addition, the totality of distances measured by module 112 is provided as input to the application program or module 116 which generates an overlay image as depicted, for example, in figures 3 and 4. In an overlay image tumor cells are represented in a base image layer in a first color and immune cells having been identified as being the next neighbor to at least one of the tumor cells and having in addition been identified as lying within a distance from said tumor cell that is below a distance threshold (e.g. a predefined "immunologically effective distance") are represented in a further image layer in a second color. The further layer is presented on top of the base layer. The images 118, the colorized overlay image is generated by module 116 and/or the histograms are presented to a user, e.g. a pathologist, via display 102.

[0089] Figure 2 is a flow chart of an image analysis method for classifying tumors. The methods depicted in figure 2 can be implemented, for example, by an image analysis system 100 as depicted in figure 1.

[0090] In a first step 202, the image analysis system receives a digital image 118 of a tissue sample. The digital image can be a multispectral fluorescent immunohistochemistry (IHC) image that is decomposed by the image analysis system into a plurality of monochromatic images by applying a color deconvolution algorithm. Alternatively, the image analysis system may receive a plurality of monochromatic digital images of the tissue sample. The multispectral digital image and/or the plurality of monochromatic digital images of the tissue sample can be provided via an interface with an image acquisition system. Alternatively, the image analysis system 100 may receive the digital images of the tissue sample by reading the images from a storage medium, e.g. a CD-ROM or flash drive.

[0091] For example, the tissue sample from which the images where derived can be a biopsy of a colorectal cancer tissue sample having being stained by a plurality of fluorescent stains selectively binding the following biomarkers: KI67, CD3 and CD8. Pixel regions for which a KI67 signal but no CD3 or CD8 signal was obtained are identified as proliferating non-lymphoid cells and thus are identified as tumor cells. Pixel regions for which a CD3 and a CD8 signal was obtained are identified as cytotoxic T-cells and thus are identified as immune cells. A "KI67 signal" as used herein is a light signal emitted by a stain that selectively stains the KI67 protein, whereby said light signal is represented in the form of pixel intensity values in a respective monochromatic image that selectively captures the emission spectrum of the stain used for selectively staining the KI67 protein. This definition applies analogously also to the other "biomarker signals".

[0092] For example, digital images of IHC-stained glass slides can be acquired using a Ventana iScan HT Slide Scanner. Images were viewed and organized using the Roche IRIS Platform. Ventana image analysis software VDP-SKD and the Ventana Digital Pathology Software Development Kit was used for performing most of the image analysis methods for identifying the cell boundaries and for identifying separate tissue and glass regions, tumor cells and immune cells and tumor cell regions mainly (e.g. to more than 50%) consisting of tumor cells.

[0093] In a second step 204, module 110 of the image analysis system performs one or more image analysis routines for identifying immune cells and tumor cells in the received digital image or images of the tissue sample. Image analysis routines for detecting cells, e.g. via a connected component analysis, gray scale and color segmentation techniques, intensity thresholding and the like are known in the art.

[0094] In a further step 206, module 112 of the image analysis system determines, for each of the identified tumor cells, the distance of the tumor cell to the nearest immune cell, e.g. by measuring the distance between cell boundaries.

[0095] After step 206 has completed, module 114 computes a proximity measure as a function of the determined distances in step 208. For example, the proximity measure can be a slope of two or more bins of a distance histogram can be computed, whereby a negative slope (from short distance bins to long distance bins) indicates that the majority of tumor cells can potentially be attacked by a nearby immune cell and whereby a positive slope (from short distance bins to long distance bins) indicates
that the majority of tumor cells can probably not be attacked successfully by an immune cell.

[0096] In step 210, module 114 classifies the identified tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor in dependence on the proximity measure.

[0097] In step 212, the classification result is stored on a storage medium 108. In addition, or alternatively, the classification result is displayed on a display device 102. For example, an overlay image is depicted in figures 3 and 4 can be used as graphical representation of the classification result: if the majority of tumor cells are covered by an immune cell being within the "immunologically effective distance", this is an indication that the tumor cell is depicted in the overlay image is a tumor cell of an inflammatory tumor.

[0098] Figures 3 and 4 depict overlay images generated for a group of colorectal cancer (CRC) patients. Using an immunofluorescence assay (CD3/CD4/CD8/Ki67/DAPI), surgical samples obtained from primary tumors of colorectal cancer (CRC) patients (n=23) was stained. The MSS/MSI status of said patients was known. KI67+/CD3-cells were identified as tumor cells. CD3+/CD8+ cells were identified as immune cells.

[0099] Figure 3 depicts three overlay images 302-306 computed by the image analysis system for three of the CRC patients whose microsatellite stability status was identified as "instable" ("MSI").

[0100] Each of the images comprises a first layer wherein all tumor cells are colorized with a first color, e.g. blue, irrespective of the presence or location of any immune cell. In each of said images 302-306, said first layer is overlaid with (and covered by) a second layer with pixel blobs representing immune cells having been identified as lying within a predefined maximum distance from at least one tumor cell. For example, these "potentially immunologically effective" immune cells are colorized in orange. It is important to note that the second layer does not indicate the presence of immune cells not lying within the predefined maximum distance. Thus, the second layer is not simply an overlay of layers representing immune cells and tumor cells in different colors, but rather is a computational result being indicative of potentially immunologically effective immune cells within the tumor. It has been observed that this particular form of overlay image provides a better and more accurate indication regarding the inflammatory state of a tumor than a mere superposition of a generic tumor cell image layer and a generic immune cell image layer.

[0101] As can be seen from figure 3, the majority of tumor cells in each of the overlay images 302, 304, 306 is covered by "potentially immunologically effective immune cells". The most complete coverage is observed in image 306. Thus, the tumor cells of the tissue samples corresponding to overlay images 302-306 are classified as cells of an inflammatory tumor. Thus, for those three patients, the status "MSI" is "in line" with the distance-based classification result.

[0102] Figure 4 depicts 8 overlay images derived for 8 other patients whose microsatellite stability status is "stable" (MSS). The overlay images were generated as already described for figure 3. The overlay images 402, 406-416 clearly show that the majority of tumor cells is not covered by pixels representing a "potentially immunologically effective" immune cell, i.e., an immune cell lying within a predefined maximum distance from at least one tumor cell. Thus, the tumor cells of the tissue samples corresponding to overlay images 402, 406-416 are classified as cells of a non-inflammatory tumor. For the corresponding seven patients, the status "MSS" is "in-line" with the distance-based classification result.

[0103] However, in the overlay image 404 the tumor cells are almost completely covered by "potentially immunologically effective" immune cells. Thus, the image analysis system according to embodiments of the invention the classifies the tumor cells of the tissue sample depicted in overlay image 404 as tumor cells of an inflammatory tumor. This is in contradiction to the known genetic MSS status of the corresponding patient. An in-depth analysis by a pathologist revealed that indeed the tumor depicted in overlay image 404 was an inflammatory tumor. If the MSS/MSI status would be used as a basis for a treatment decision, the corresponding patient would not have been prescribed a drug that stimulates or modulates an immune response. However, using a proximity measure as described herein for embodiments of the invention as a basis for a treatment suggestion, the patient corresponding to overlay image 404 is identified by the image analysis system as a patient benefiting from the subscription of a dragon that stimulates the immune system.

[0104] Figure 5 depicts three histograms 502, 504, 506 derived from the same three MSI tissue samples whose computed overlay images 302, 304, 306 are depicted in Fig. 3.

[0105] Each histogram depicted in figures 5, 6 and 9 comprises four distance bins. The first been covers a distance of 0 µm to 20 µm, the second been covers a distance from 21 µm to 40 µm, the third been covers a distance of 41 µm to 60 µm and the last bin covers a distance from 61 µm to 80 µm. The term "distance" relates to the distance of anyone of the identified tumor cells to its nearest identified immune cell. The distance refers to the "real" distance between respective cells in the tissue sample.

[0106] The image analysis system uses information on the resolution and/or zoom factor of the image as input for computing the cell-cell distance in µm from the numbers of pixels separating the boundaries of two cells within one image. For each bin, a corresponding bar is plotted in the histogram, whereby the height of each bar reflects the number of tumor cells whose distance to its nearest immune cell falls within said bin.

[0107] As can be inferred from the histograms 502, 504, 506, the majority of tumor cells is contained in the first and second been representing distances within the immunologically effective distance of immune cells. By plotting a straight or curved line that connects the top of the bar of the first bin with the top of the bar of the last bin and by determining the slope of that line, a negative (falling) slope will be identified. A negative slope in the distance histogram is used as a proximity measure that indicates that the tumor cells of the tumor belong to an inflammatory tumor. This proximity measure based classification result is in line with the microsatellite instability status information "MSI".

[0108] Figure 6 depicts 8 histograms derived from the 8 MSS tissue samples whose overlay images are depicted in Fig. 4.Thus, the histograms 602-616 correspond to the 8 patients whose microsatellite stability status is "stable" (MSS). The histograms were generated as already described for figure 5. The histograms, 602, 606-616 all have a positive slope, thereby clearly showing that the majority of tumor cells does not comprise a neighboring immune cell within the predefined maximum distance that reflects the "immunologically effective" immune cell distance. Thus, the tumor cells of the tissue samples corresponding to histograms 602, 606-616 are classified as tumor cells of a non-inflammatory tumor. For the corresponding seven patients, the status "MSS" is "in-line" with the slope-based classification result.

[0109] However, in histogram 604 (corresponding to the same patient as overlay image 404), a negative slope is observed and the image analysis system classifies the tumor cells of the tumor corresponding to the distance histogram 604 as an inflammatory tumor. This is in contradiction to the known genetic MSS status of the corresponding patient. As was explained above, the in-depth analysis revealed that indeed the tumor corresponding to histogram 604 was an inflammatory tumor.

[0110] In other words, the proximity measure based tumor classification, e.g. the histogram slope based classification illustrated in figures 5 and 6 a more accurate than state-of-the-art MSS/MSI status based tumor classification.

[0111] Figure 7 depicts three IHC fluorescent images 702, 704, 806 of the MSI tissue samples of the three patients for which respective overlay images were generated as depicted in Fig. 3 and for which respective distance histograms were generated as depicted in Fig. 5.

[0112] Figure 8 depicts 8 IHC fluorescent images 802-816 of the MSI tissue samples of the 8 patients for which respective overlay images were generated as depicted in Fig. 4 and for which respective distance histograms were generated as depicted in Fig. 6. IHC image 804 shows the inflammatory tumor whose IHC image looks more similar to the IHC images 702-706 than to IHC images 802, 806-816.

[0113] Figure 9 depicts a comparison of two groups of images, wherein the group depicted on the left corresponds to an inflammatory CRC tumor of a first patient and the group to the right corresponds to a non-inflammatory CRC tumor of a second patient. Tumor cells were identified as KI67+/CD3- cells. Immune cells (here: cytotoxic T-cells) were identified as CD8+/CD3+ cells. The first (topmost) row depicts overlay images computed in the same way as described for figures 3 and 4. The second row depicts the IHC fluorescent images of the respective tissue samples and the third row depicts the respectively computed distance histograms generated as described for figures 5 and 6.

[0114] For inflammatory tumor tissue (left column), the majority of tumor cells are covered in the overlay image 902 by immune cells having been identified to lie within a predefined maximum distance (immunologically effective distance) from a tumor cell. A corresponding field of view (FOV) 902 from the corresponding tissue below sows a significant degree of tumor immune infiltration. The slope of the corresponding distance histogram determined as described for figures 5 and 6 is negative.

[0115] For non-inflammatory tumor tissue (right column), the majority of tumor cells are not covered in the overlay image 904 by immune cells having been identified to lie within a predefined maximum distance (immunologically effective distance) from a tumor cell. A corresponding field of view (FOV) 908 from the corresponding tissue below sows that there is almost no tumor infiltration by immune cells will. The slope of the corresponding distance histogram determined is positive.

[0116] The difference in distribution of cytotoxic t cells engaging cancer cells between different tumor categories as expressed graphically in the overlay images 902, 904 and as expressed in the slope of the histograms 910, 912 is apparent.

[0117] Figure 10 depicts a plot indicating the probability density that a particular immune cell lies within a given distance from its nearest tumor cell.

[0118] A probability density function (PDF) is a function that describes the relative likelihood for a variable to take on a given value. The probability of the variable falling within a particular range of values is given by the integral of this variable's density over that range-that is, it is given by the area under the density function but above the horizontal axis and between the lowest and greatest values of the range. The probability density function is nonnegative everywhere, and its integral over the entire space is equal to one.

[0119] The plot depicts multiple thin MSS curves respectively representing the probability function in dependence on the distance of a tumor cell to its nearest immune cell obtained from an image of a tissue sample having been classified as MSS sample. For the MSS samples, the probability density strongly increases at a distance larger than 90µm. A bold MSS curve represents the average of all MSS curves in the plot.

[0120] The plot further depicts multiple thin MSI curves respectively representing the probability function in dependence on the distance of a tumor cell to its nearest immune cell obtained from an image of a tissue sample having been classified as MSI sample. For the MSI samples, the probability density is highest at a distance smaller than 35 µm. A bold MSI curve represents the average of all MSI curves in the plot.

[0121] In addition, the plot comprises some curves obtained for tissue samples with unknown MMS/MSI status.

[0122] The automated evaluation of tumor cell-immune cell distances in a tissue sample on a whole slide level provides a more comprehensive and accurate insight into tumor biology.

[0123] Figure 11 depicts a digital image of a stained colorectal cancer sample comprising annotations of tumor context information. A combination of proximity information and cell density information is captured as tumor context information for classifying the tumor. As was described for several embodiments of the invention, proximity measures between tumor cells and immune cells have been observed to represent valuable predictive parameters for classifying tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor. The classification result may also be used for predicting whether or not the tumor will be treatable by a drug that boosts the immune system. Thus, the distances between immune cells and tumor cells and the distance measure derived therefrom represent tumor context information that provides valuable predictive information on the likely tumor type, tumor progression and treatability.

[0124] It has been observed that the accuracy of the classification can significantly be increased by determining and using as input also the density of the tumor cells and/or the density of immune cells. Thus, according to embodiments, the tumor cell density and/or the immune cell density (measured e.g. in number of cells per 100 µm × 100 µm) is automatically determined and is fed as an additional input to the classifier. According to embodiments, the density of the immune cells is determined in an immune cell type specific manner (resulting in the computation of a density of cytotoxic t-cells, a density for macrophages, a density for B-cells, etc.).

[0125] Accordingly, in the training phase of the classifier, density information can be assigned to each tumor cell and/or to each immune cell contained in any of the training images. For example, for each tumor cell in a training image, a square-shaped or circular area of about 100µm × 100µm comprising said tumor cell in its geometrical center may be determined. Then, the number of tumor cells in said area is automatically determined and the tumor cell density is computed for said area. Then, the computed density is assigned to the tumor cell in the center of the area. This procedure may be repeated for each tumor cell. Likewise, for each immune cell in the training image, a square-shaped or circular area of about 100µm × 100µm comprising said immune cell in its geometrical center may be determined. Then, the number of immune cells in said area is automatically determined and the immune cell density is computed for said area. Then, the computed density is assigned to the immune cell in the center of the area. According to embodiments, the density of the immune cells is determined in the training phase in an immune cell type specific manner (resulting in the computation of a density of cytotoxic t-cells, a density for macrophages, a density for B-cells, etc.).

[0126] In the depicted image, several tumor cells (Ki67 positive cells) and several immune cells (CD8 positive T-cells) were identified and the distance between the tumor cells and the T-cells was determined (see the large two circles around the two small tumor cells): each of the large circle represents a distance threshold of e.g. about 35 µm. For each of the identified tumor cells, the image analysis system determines the distance of said tumor cell to the nearest immune cell and determined if said distance is below the predefined distance threshold indicated by the large circles. The ratio of tumor cells for which a "close distance" smaller than the distance threshold and tumor cells for which a "large distance" larger than the distance threshold was measured is fed in addition to e.g. immune cell density information of the respective tumor cells into the classifier for classifying a tumor as "inflammatory" or "non-inflammatory".

[0127] According to embodiments, the classifier is configured such that the classification of the identified tumor cells into "tumor cells of an inflammatory tumor" or as "tumor cells of a non-inflammatory tumor" comprising checking if the tumor cell densities of the majority of the tumor cells in the tumor depicted in the received image are within a tumor cell density range that is expected for a tumor cell of an inflammatory tumor or if the tumor cell densities of the majority of the tumor cells in the tumor depicted in the received image are within a tumor cell density range that is expected for a tumor cell of a non-inflammatory tumor. For example, the expected density ranges can be specified manually for different types of tumors (inflammatory/non-inflammatory breast cancer/colon cancer/lung cancer etc) or can be specified automatically and implicitly during the training phase by using the tumor cell densities as additional training parameters.

[0128] In addition or alternatively, the classifier is configured such that the classification of the identified tumor cells into "tumor cells of an inflammatory tumor" or as "tumor cells of a non-inflammatory tumor" comprising checking if the immune cell densities of the majority of the immune cells in spatial proximity to the tumor are within an immune cell density range that is expected in the context of an inflammatory tumor or if the immune cell densities of the majority of the immune cells in spatial proximity to the tumor are within an immune cell density range that is expected in the context of a tumor cell of a non-inflammatory tumor. As mentioned above, the expected density ranges for the immune cell densities can be specified manually for different types of tumors (inflammatory/non-inflammatory breast cancer/colon cancer/lung cancer etc) or can be specified automatically and implicitly during the training phase by using the immune cell densities as additional training parameters.

[0129] According to embodiments, the proximity measures and the cell densities are determined multiple times before, during and/or after the treatment of a patient for predicting the treatability of a tumor and for determining whether or not an applied drug had an effect on the tumor or the immune cells.

[0130] Moreover, according to embodiments, the dynamic interplay of any cell population (e.g. tumor cells or immune cells) with a given reference point (e.g. a cell of another cell type than that of the cell population like immune cells, tumor cells, stroma cells or other tissues, e.g. vessels) is determined and used as an additional predictive input parameter for training an untrained classifier and/or for feeding a trained classifier.

[0131] Figure 12 depicts stained tissue samples of two patients A, B having similar T-cell densities but different proximity measures. Different degrees of immune cell infiltration in tumor nests have been observed to represent important signs of tumor cell recognition and/or eradication by immune cells. In the two compared tissue samples, the same T-cell density but strongly deviating proximity measures were observed: while in sample A only 4% of the tumor cells were observed in close proximity to a T-cell, in sample B more than 20% of the tumor cells had at least one T-cell in close proximity, e.g. within a distance of about 35µm. It has been observed that a low degree of tumor infiltration by immune cells correlates with a microsatellite status "MSS" while a high degree of tumor infiltration (sample B) correlates with a microsatellite status "MSI". Moreover, it has been observed that while already the computation of the proximity measure provides good or even better inflammatory/non-inflammatory tumor classification results than the conventional and expensive determination of MSS/MSI state, it has been observed that the additional determination and use of density information provides significantly more accurate classification results than the use of the proximity measure alone. This is because the density of a tissue may also have an impact on the proximity measure which however, may be more related to the type of the tissue from which a tumor stems than to the status of immune cell infiltration of the tumor.


Claims

1. An image analysis method for tumor classification, the method comprising:

- receiving (202), by an image analysis system (100), at least one digital image (118) of a tissue sample;

- analyzing (204), by the image analysis system, the at least one received image for identifying immune cells and tumor cells in the at least one received image;

- for each of the identified tumor cells, determining (206), by the image analysis system, the distance of said tumor cell to the nearest immune cell;

- computing (208), by the image analysis system, a proximity measure as a function of the determined distances, the computation of the proximity measure as a function of the determined distances comprising:

• generating a histogram (502-506; 602-616; 910, 912) of the distances of the tumor cells to their respective nearest one of the identified immune cells, the histogram comprising at least two distance bins, the histogram covering a distance range of 0 µm to at least 50 µm, each of the bins corresponding to a bar of the histogram, each of the bars indicating a count of the identified tumor cells having a distance to their nearest immune cell that falls into said bin;

• connecting the upper end of the first one of the bars with the upper end of the last one of the bars with a line, the first bar corresponding to the one of the bins covering the smallest distances, the last bar corresponding to the one of the bins covering the largest distances of the distance range;

• determining the slope of the line;

• using the slope as the proximity measure, wherein in case the slope indicates that the tumor cell count of the first bar is higher than the tumor cell count of the last bar, the classification result is that the identified tumor cells belong to an inflammatory tumor;

- in dependence on the proximity measure, classifying (210), by the image analysis system, the identified tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor; and

- storing (212), by the image analysis system, the classification result on a storage medium and/or displaying (214) the classification result on a display device.


 
2. An image analysis method for tumor classification, the method comprising:

- receiving, by an image analysis system (100), a digital image (118) of a tissue sample;

- analyzing, by the image analysis system, the at least one received image for identifying tumor cells in the at least one received image;

- analyzing, by the image analysis system, the identified tumor cells for identifying tumor regions, wherein a tumor region is a tissue region whose majority of cells consist of tumor cells;

- analyzing, by the image analysis system, the at least one received image for identifying immune cells lying within one of the identified tumor regions or lying within a boundary region around one of the tumor regions, the width of the boundary being smaller than 200µm;

- for each of the identified immune cells, determining, by the image analysis system, the distance of said immune cell to the nearest tumor cell;

- computing, by the image analysis system, a proximity measure as a function of the determined distances, the computation of the proximity measure as a function of the determined distances comprising:

• generating a histogram of the distances of the immune cells to their respective nearest one of the identified tumor cells, the histogram comprising at least two distance bins, the histogram covering a distance range of 0 µm to at least 50 µm, each of the bins corresponding to a bar of the histogram, each of the bars indicating a count of the identified immune cells having a distance to their nearest tumor cell that falls into said bin;

• connecting the upper end of the first one of the bars with the upper end of the last one of the bars with a line, the first bar corresponding to the one of the bins covering the smallest distances, the last bar corresponding to the one of the bins covering the largest distances of the distance range;

• determining the slope of the line;

• using the slope as the proximity measure, wherein in case the slope indicates that the immune cell count of the first bar is higher than the immune cell count of the last bar, the classification result is that the identified tumor cells belong to an inflammatory tumor;

- in dependence on the proximity measure, classifying, by the image analysis system, the identified tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor; and

- storing, by the image analysis system, the classification result on a storage medium and/or displaying the classification result on a display device.


 
3. The method of claim 1 or 2, the displaying of the classification result comprising:

- representing all identified tumor cells as first pixel blobs having a first color; and

- representing all identified immune cells whose distance to its nearest tumor cells is below a threshold as second pixel blobs having a second color;

- displaying the second pixel blobs as an overlay of the first pixel blobs.


 
4. The image analysis method of any one of the previous claims, the tissue sample being a whole slide tissue sample and the digital image being a whole slide image.
 
5. The image analysis method of any one of the previous claims, further comprising:

- analyzing the received digital image for determining a tumor cell density of the identified tumor cells;

- wherein the classification of the identified tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor comprises inputting the proximity measure and the tumor cell density into a classifier configured to perform the classification.


 
6. The image analysis method of any one of the previous claims, further comprising:

- analyzing the received digital image for determining an immune cell density of the identified immune cells or of a particular type of immune cells;

- wherein the classification of the identified tumor cells into tumor cells of an inflammatory tumor or as tumor cells of a non-inflammatory tumor comprises inputting the proximity measure and the immune cell density into a classifier configured to perform the classification.


 
7. The image analysis method of any one of the previous claims, the identification of the tumor cells comprising:

- identifying proliferating non-lymphoid cells and using said identified cells as the tumor cells; and/or

- identifying cells expressing a set of one or more tumor-specific biomarkers and using said identified cells as the tumor cells.


 
8. The image analysis method of any one of the previous claims,

- the identification of the immune cells comprising identifying cytotoxic T-cells and using the identified cytotoxic T-cells as the identified immune cells; or

- the identification of the immune cells comprising identifying T-helper-cells and using the identified T-helper-cells as the identified immune cells; or

- the identification of the immune cells comprising identifying macrophages and using the identified macrophages as the identified immune cells; or

- the identification of the immune cells comprising identifying memory cells and using the identified memory cells as the identified immune cells; or

- the identification of the immune cells comprising identifying B- cells and using the identified memory cells as the identified immune cells; and/or

- the identification of the immune cells comprising identifying activated T-cells and using the identified activated T cells as the identified immune cells; or

- the identification of the immune cells comprising identifying PD1+ immune cells and using the identified PD1+ immune cells as the identified immune cells, the identification of the tumor cells comprising identifying PDL1+ tumor cells and using the identified PDL1+ tumor cells as the identified tumor cells.


 
9. The image analysis method of any one of the previous claims,

- the identification of the immune cells comprising identifying regulatory T-cells and identifying immune cells of at least one type of immune cells that boosts an immune response; and

- selectively using the identified immune cells that boost the immune response but not the identified regulatory T-cells as the identified immune cells.


 
10. The image analysis method of any one of the previous claims, the method further comprising:

- selectively in case the classification result indicates that the identified tumor cells belong to an inflammatory tumor, outputting a signal being indicative of a treatment recommendation to use a substance that boosts or modulates an immune response as a drug for treating the tumor.


 
11. The image analysis method of any one of the previous claims, the method further comprising:

- using the proximity measure in addition to or in replacement of an MSS "Microsatellite Stable" MSI "Microsatellite Instable" status indicator for computing a prognosis of the responsiveness of the tumor to a substance that boosts or modulates an immune response.


 
12. The image analysis method of any one of the previous claims, the identified tumor cells being colorectal cancer cells.
 
13. The image analysis method of any one of the previous claims, the identification of the immune cells comprising identifying immune cells of multiple different immune cell types by analyzing pixel intensity values in the at least one digital image representing the presence of different immune cell type specific biomarkers, and using the totality of the identified immune cells of the multiple different immune cell types as the identified immune cells for computing the proximity measure.
 
14. The image analysis method of any one of the previous claims, the tumor cells being identified as cells expressing at least a first tumor-type specific biomarker, the immune cells being identified as cells expressing at least a first immune cell type specific biomarker, the method further comprising:

- analyzing the at least one received image or a further image of the tissue sample for identifying further immune cells as cells expressing a second immune cell type specific biomarker; and

- analyzing the at least one received image or the further image of the tissue sample for identifying further tumor cells as cells expressing a second tumor type specific biomarker or using the identified immune cells as further tumor cells;

- for each of the identified further tumor cells, determining the distance of said further tumor cell to the nearest further immune cell;

- computing a further proximity measure as a function of said determined distances;

- in dependence on the further proximity measure, sub-classifying the identified tumor cells and/or predicting the treatability of the tumor by a class of substances.


 
15. An image analysis system (100) for tumor classification, the system comprising a processor (104) being configured for performing a method according to any one of the previous claims 1-14.
 


Ansprüche

1. Bildanalyseverfahren für Tumorklassifikation, wobei das Verfahren umfasst:

- Empfangen (202), durch ein Bildanalysesystem (100), mindestens eines digitalen Bildes (118) einer Gewebeprobe;

- Analysieren (204), durch das Bildanalysesystem, des mindestens einen empfangenen Bildes zum Identifizieren von Immunzellen und Tumorzellen in dem mindestens einen empfangenen Bild;

- für jede von den identifizierten Tumorzellen Feststellen (206), durch das Bildanalysesystem, der Entfernung der Tumorzelle von der nächsten Immunzelle;

- Berechnen (208), durch das Bildanalysesystem, eines Näherungsmaßes als eine Funktion der festgestellten Entfernungen, wobei die Berechnung des Näherungsmaßes als eine Funktion der festgestellten Entfernungen umfasst:

• Erzeugen eines Histogramms (502-506; 602-616; 910, 912) der Entfernungen der Tumorzellen von ihrer jeweiligen nächsten einen der identifizierten Immunzellen, wobei das Histogramm mindestens zwei Entfernungs-Bins umfasst, wobei das Histogramm einen Entfernungsbereich von 0 µm bis mindestens 50 µm abdeckt, wobei jedes der Bins einem Balken auf dem Histogramm entspricht, wobei jeder der Balken eine Anzahl der identifizierten Tumorzellen mit einer Entfernung zu ihrer nächsten Immunzelle, die in das Bin fällt, entspricht;

• Verbinden des oberen Endes des ersten ein der Balken mit dem oberen Ende des letzten einen der Balken mit einer Linie, wobei der erste Balken dem einen der Bins, das die kleinsten Entfernungen abdeckt, entspricht, wobei der letzte Balken dem einen der Bins, das die größten Entfernungen des Entfernungsbereichs abdeckt, entspricht;

• Bestimmen der Neigung der Linie;

• Verwenden der Neigung als das Näherungsmaß, wobei in dem Fall, dass die Neigung angibt, dass die Tumorzellenzahl des ersten Balkens höher als die Tumorzellenzahl des letzten Balkens ist, das Klassifikationsergebnis ist, dass die identifizierten Tumorzellen zu einem entzündlichen Tumor gehören;

- in Abhängigkeit von dem Näherungsmaß Klassifizieren (210), durch das Bildanalysesystem, der identifizierten Tumorzellen in Tumorzellen eines entzündlichen Tumors oder als Tumorzellen eines nicht entzündlichen Tumors; und

- Speichern (212), durch das Bildanalysesystem, des Klassifikationsergebnisses auf einem Speichermedium, und/oder Anzeigen (214) des Klassifikationsergebnisses auf einer Anzeigevorrichtung.


 
2. Bildanalyseverfahren für Tumorklassifikation, wobei das Verfahren umfasst:

- Empfangen, durch ein Bildanalysesystem (100), eines digitalen Bildes (118) einer Gewebeprobe;

- Analysieren, durch das Bildanalysesystem, des mindestens einen empfangenen Bildes zum Identifizieren von Tumorzellen in dem mindestens einen empfangenen Bild;

- Analysieren, durch das Bildanalysesystem, der identifizierten Tumorzellen zum Identifizieren von Tumorregionen, wobei eine Tumorregion eine Geweberegion ist, deren Mehrheit von Zellen aus Tumorzellen besteht;

- Analysieren, durch das Bildanalysesystem, des mindestens einen empfangenen Bildes zum Identifizieren von Immunzellen, die innerhalb einer der identifizierten Tumorregionen liegen oder innerhalb einer Grenzregion rings um eine der Tumorregionen liegen, wobei die Breite der Grenze kleiner als 200 µm ist;

- für jede von den identifizierten Immunzellen Feststellen, durch das Bildanalysesystem, der Entfernung von der Immunzelle zur nächsten Tumorzelle;

- Berechnen, durch das Bildanalysesystem, eines Näherungsmaßes als eine Funktion der festgestellten Entfernungen, wobei die Berechnung des Näherungsmaßes als eine Funktion der festgestellten Entfernungen umfasst:

• Erzeugen eines Histogramms der Entfernungen der Immunzellen von ihrer jeweiligen nächsten einen der identifizierten Tumorzellen, wobei das Histogramm mindestens zwei Entfernungs-Bins umfasst, wobei das Histogramm einen Entfernungsbereich von 0 µm bis mindestens 50 µm abdeckt, wobei jedes der Bins einem Balken auf dem Histogramm entspricht, wobei jeder der Balken einer Anzahl der identifizierten Immunzellen mit einer Entfernung zu ihren nächsten Tumorzellen, die in das Bin fällt, entspricht;

• Verbinden des oberen Endes des ersten ein der Balken mit dem oberen Ende des letzten einen der Balken mit einer Linie, wobei der erste Balken dem einen der Bins, das die kleinsten Entfernungen abdeckt, entspricht, wobei der letzte Balken dem einen der Bins, das die größten Entfernungen des Entfernungsbereichs abdeckt, entspricht;

• Bestimmen der Neigung der Linie;

• Verwenden der Neigung als das Näherungsmaß, wobei in dem Fall, dass die Neigung angibt, dass die Immunzellenzahl des ersten Balkens höher als die Immunzellenzahl des letzten Balkens ist, das Klassifikationsergebnis ist, dass die identifizierten Tumorzellen zu einem entzündlichen Tumor gehören;

- in Abhängigkeit von dem Näherungsmaß Klassifizieren, durch das Bildanalysesystem, der identifizierten Tumorzellen in Tumorzellen eines entzündlichen Tumors oder als Tumorzellen eines nicht entzündlichen Tumors; und

- Speichern, durch das Bildanalysesystem, des Klassifikationsergebnisses auf einem Speichermedium, und/oder Anzeigen des Klassifikationsergebnisses auf einer Anzeigevorrichtung.


 
3. Verfahren nach Anspruch 1 oder 2, wobei das Anzeigen des Klassifikationsergebnisses umfasst:

- Repräsentieren aller identifizierten Tumorzellen als erste Pixel-Blobs mit einer ersten Farbe; und

- Repräsentieren aller identifizierten Immunzellen, deren Entfernung zu ihren nächsten Tumorzellen unter einem Schwellenwert ist, als zweit Pixel-Blobs mit einer zweiten Farbe;

- Anzeigen der zweiten Pixel-Blobs als eine Überlagerung der ersten Pixel-Blobs.


 
4. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei die Gewebeprobe eine Whole-Slide-Gewebeprobe ist und das digitale Bild ein Whole-Slide-Bild ist.
 
5. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, ferner umfassend:

- Analysieren des empfangenen digitalen Bildes zum Feststellen einer Tumorzellendichte der identifizierten Tumorzellen;

- wobei die Klassifikation der identifizierten Tumorzellen in Tumorzellen eines entzündlichen Tumors oder als Tumorzellen eines nicht entzündlichen Tumors das Eingeben des Näherungsmaßes und der Tumorzellendichte in einen Klassifikator, ausgelegt zum Ausführen der Klassifikation, umfasst.


 
6. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, ferner umfassend:

- Analysieren des empfangenen digitalen Bildes zum Feststellen einer Immunzellendichte der identifizierten Immunzellen oder eines bestimmten Typs von Immunzellen;

- wobei die Klassifikation der identifizierten Tumorzellen in Tumorzellen eines entzündlichen Tumors oder als Tumorzellen eines nicht entzündlichen Tumors das Eingeben des Näherungsmaßes und der Immunzellendichte in einen Klassifikator, ausgelegt zum Ausführen der Klassifikation, umfasst.


 
7. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei die Identifikation der Tumorzellen umfasst:

- Identifizieren proliferierender, nicht lymphoider Zellen und Verwenden der identifizierten Zellen als die Tumorzellen; und/oder

- Identifizieren von Zellen, die eine Gruppe eines oder mehrerer tumorspezifischer Biomarker ausdrücken, und Verwenden der identifizierten Zellen als die Tumorzellen.


 
8. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei

- die Identifikation der Immunzellen ein Identifizieren zytotoxischer T-Zellen und Verwenden der identifizierten zytotoxischen T-Zellen als die identifizierten Immunzellen umfasst; oder

- die Identifikation der Immunzellen ein Identifizieren von T-Helfer-Zellen und Verwenden der identifizierten T-Helfer-Zellen als die identifizierten Immunzellen umfasst; oder

- die Identifikation der Immunzellen ein Identifizieren von Macrophagen und Verwenden der identifizierten Macrophagen als die identifizierten Immunzellen umfasst; oder

- die Identifikation der Immunzellen ein Identifizieren von Gedächtniszellen und Verwenden der identifizierten Gedächtniszellen als die identifizierten Immunzellen umfasst; oder

- die Identifikation der Immunzellen ein Identifizieren von B-Zellen und Verwenden der identifizierten Gedächtniszellen als die identifizierten Immunzellen umfasst; und/oder

- die Identifikation der Immunzellen ein Identifizieren aktivierter T-Zellen und Verwenden der identifizierten aktivierten T-Zellen als die identifizierten Immunzellen umfasst; oder

- die Identifikation der Immunzellen ein Identifizieren von PD-1-positiven Immunzellen und Verwenden der identifizierten PD-1-positiven Immunzellen als die identifizierten Immunzellen umfasst, wobei die Identifikation der Tumorzellen ein Identifizieren von PD-L1-positiven Tumorzellen und ein Verwenden der identifizierten PD-L1-positiven Tumorzellen als die identifizierten Tumorzellen umfasst.


 
9. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei

- die Identifikation der Immunzellen ein Identifizieren regulatorischer T-Zellen und ein Identifizieren von Immunzellen mindestens eines Typs von Immunzellen umfasst, der eine Immunantwort verstärkt; und

- die identifizierten Immunzellen, die die Immunantwort verstärken, nicht jedoch die identifizierten regulatorischen T-Zellen als die identifizierten Immunzellen selektiv verwendet werden.


 
10. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei das Verfahren ferner umfasst:

- in einem Fall, in dem das Klassifikationsergebnis angibt, dass die identifizierten Tumorzellen zu einem entzündlichen Tumor gehören, selektiv Ausgeben eines Signals, das eine Behandlungsempfehlung zur Nutzung eines Stoffes, der eine Immunantwort verstärkt oder moduliert, als ein Medikament zur Behandlung des Tumors angibt.


 
11. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei das Verfahren ferner umfasst:

- Verwenden des Näherungsmaßes zusätzlich zu oder in Ersatz von einem Statusindikator MSS ("Mikrosatellitenstabil") / MSI ("Mikrosatelliteninstabil") zum Berechnen einer Prognose der Reagibilität eines Tumors auf einen Stoff, der eine Immunantwort verstärkt oder moduliert.


 
12. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei die identifizierten Tumorzellen kolorektale Krebszellen sind.
 
13. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei die Identifikation der Immunzellen ein Identifizieren von Immunzellen mehrerer unterschiedlicher Immunzellentypen durch Analysieren von Pixelintensitätswerten in dem mindestens einen digitalen Bild, das die Präsenz unterschiedlicher immunzellentypspezifischer Biomarker repräsentiert, und Verwenden der Gesamtheit der identifizierten Immunzellen der mehreren unterschiedlichen Immunzellentypen als die identifizierten Immunzellen zum Berechnen des Näherungsmaßes umfasst.
 
14. Bildanalyseverfahren nach einem der vorangehenden Ansprüche, wobei die Tumorzellen als Zellen identifiziert werden, die mindestens einen ersten tumortypspezifischen Biomarker ausdrücken, wobei die Immunzellen als Zellen identifiziert werden, die mindestens einen ersten immunzellentypspezifischen Biomarker ausdrücken, wobei das Verfahren ferner umfasst:

- Analysieren des mindestens einen empfangenen Bildes oder eines weiteren Bildes der Gewebeprobe zum Identifizieren weiterer Immunzellen als Zellen, die einen zweiten immunzellentypspezifischen Biomarker ausdrücken; und

- Analysieren des mindestens einen empfangenen Bildes oder des weiteren Bildes der Gewebeprobe zum Identifizieren weiterer Tumorzellen, die einen zweiten tumortypspezifischen Biomarker ausdrücken, oder Verwenden der identifizierten Immunzellen als weitere Tumorzellen;

- für jede von den identifizierten weiteren Tumorzellen Feststellen der Entfernung der weiteren Tumorzelle zur nächsten weiteren Immunzelle;

- Berechnen eines weiteren Näherungsmaßes als eine Funktion der bestimmten Entfernungen;

- in Abhängigkeit von dem weiteren Näherungsmaß Unterklassifizierung der identifizierten Tumorzellen und/oder Prognostizieren der Behandelbarkeit des Tumors durch eine Klasse von Stoffen.


 
15. Bildanalysesystem (100) zur Tumorklassifikation, wobei das System einen Prozessor (104), der zum Ausführen eines Verfahrens nach einem der vorangehenden Ansprüche 1-14 konfiguriert ist, umfasst.
 


Revendications

1. Procédé d'analyse d'image permettant une classification d'une tumeur, le procédé comprenant :

- la réception (202), par un système d'analyse d'image (100), d'au moins une image (118) numérique d'un échantillon de tissu ;

- l'analyse (204), par le système d'analyse d'image, de l'au moins une image reçue pour identifier des cellules immunitaires et des cellules tumorales dans l'au moins une image reçue ;

- pour chacune des cellules tumorales identifiées, la détermination (206), par le système d'analyse d'image, de la distance de ladite cellule tumorale jusqu'à la cellule immunitaire la plus proche ;

- le calcul informatique (208), par le système d'analyse d'image, d'une mesure de proximité sous forme d'une fonction des distances déterminées, le calcul informatique de la mesure de proximité sous forme d'une fonction des distances déterminées comprenant :

• la génération d'un histogramme (502 à 506 ; 602 à 616 ; 910, 912) des distances des cellules tumorales à leur cellule la plus proche respective parmi les cellules immunitaires identifiées, l'histogramme comprenant au moins deux échelles de distance, l'histogramme couvrant une plage de distances de 0 µm à au moins 50 µm, chacune des échelles correspondant à une barre de l'histogramme, chacune des barres indiquant un comptage des cellules tumorales identifiées ayant une distance jusqu'à leur cellule immunitaire la plus proche qui tombe dans ladite échelle ;

• la connexion par une ligne droite de l'extrémité supérieure de la première des barres avec l'extrémité supérieure de la dernière des barres, la première barre correspondant à celle des échelles recouvrant les plus faibles distances, la dernière barre correspondant à celle des échelles recouvrant les distances les plus importantes de la plage de distances ;

• la détermination de la pente de la droite ;

• l'utilisation de la pente en tant que mesure de proximité, où, dans le cas où la pente indique que le comptage de cellules tumorales de la première barre est plus important que le comptage de cellules tumorales de la dernière barre, le résultat de la classification est que les cellules tumorales identifiées font partie d'une tumeur inflammatoire ;

- en fonction de la mesure de proximité, le classement (210), par le système d'analyse d'image, des cellules tumorales identifiées en cellules tumorales d'une tumeur inflammatoire ou en cellules tumorales d'une tumeur non inflammatoire ; et

- l'enregistrement (212), par le système d'analyse d'image, du résultat de classification sur un support de stockage et/ou l'affichage (214) du résultat de classification sur un dispositif d'affichage.


 
2. Procédé d'analyse d'image permettant une classification d'une tumeur, le procédé comprenant :

- la réception, par un système d'analyse d'image (100), d'une image (118) numérique d'un échantillon de tissu ;

- l'analyse, par le système d'analyse d'image, de l'au moins une image reçue pour identifier des cellules tumorales dans l'au moins une image reçue ;

- l'analyse, par le système d'analyse d'image, des cellules tumorales identifiées pour identifier des régions tumorales, où une région tumorale est une région du tissu dont la majorité des cellules est constituée de cellules tumorales,

- l'analyse, par le système d'analyse d'image, de l'au moins une image reçue pour identifier des cellules immunitaires se situant dans une des régions tumorales identifiées ou se situant dans une région limitrophe autour d'une des régions tumorales, la largeur de la limite étant inférieure à 200 µm ;

- pour chacune des cellules tumorales identifiées, la détermination, par le système d'analyse d'image, de la distance de ladite cellule immunitaire jusqu'à la cellule tumorale la plus proche ;

- le calcul informatique, par le système d'analyse d'image, d'une mesure de proximité sous forme d'une fonction des distances déterminées, le calcul informatique de la mesure de proximité sous forme d'une fonction des distances déterminées comprenant :

• la génération d'un histogramme des distances des cellules immunitaires à leur cellule la plus proche respective parmi les cellules tumorales identifiées, l'histogramme comprenant au moins deux échelles de distance, l'histogramme couvrant une plage de distances de 0 µm à au moins 50 µm, chacune des échelles correspondant à une barre de l'histogramme, chacune des barres indiquant un comptage des cellules immunitaires identifiées ayant une distance jusqu'à leur cellule tumorale la plus proche qui tombe dans ladite échelle ;

• la connexion par une droite de l'extrémité supérieure de la première des barres avec l'extrémité supérieure de la dernière des barres, la première barre correspondant à celle des échelles recouvrant les plus faibles distances, la dernière barre correspondant à celle des échelles recouvrant les distances les plus importantes de la plage de distances ;

• la détermination de la pente de la droite ;

• l'utilisation de la pente en tant que mesure de proximité, où, dans le cas où la pente indique que le comptage de cellules immunitaires de la première barre est plus important que le comptage de cellules immunitaires de la dernière barre, le résultat de la classification est que les cellules tumorales identifiées font partie d'une tumeur inflammatoire ;

- en fonction de la mesure de proximité, le classement, par le système d'analyse d'image, des cellules tumorales identifiées en cellules tumorales d'une tumeur inflammatoire ou en cellules tumorales d'une tumeur non inflammatoire ; et

- l'enregistrement, par le système d'analyse d'image, du résultat de classification sur un support de stockage et/ou l'affichage du résultat de classification sur un dispositif d'affichage.


 
3. Procédé d'analyse d'image selon la revendication 1 ou la revendication 2, l'affichage du résultat de la classification comprenant :

- la représentation de toutes les cellules tumorales identifiées en tant que premiers blobs de pixels possédant une première couleur ; et

- la représentation de toutes les cellules immunitaires dont la distance jusqu'à leurs cellules tumorales les plus proches est inférieure à un seuil en tant que deuxièmes blobs de pixels ayant une deuxième couleur ;

- l'affichage des deuxièmes blobs de pixels sous forme d'une superposition avec les premiers blobs de pixels.


 
4. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, l'échantillon de tissu étant un échantillon de tissu en lame entière et l'image numérique étant une image de lame entière.
 
5. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, comprenant en outre :

- l'analyse de l'image numérique reçue pour déterminer une densité de cellules tumorales des cellules tumorales identifiées ;

- dans lequel la classification des cellules tumorales identifiées en cellules tumorales d'une tumeur inflammatoire ou en cellules tumorales d'une tumeur non inflammatoire comprend la saisie de la mesure de proximité et de la densité de cellules tumorales dans un classifieur conçu pour effectuer la classification.


 
6. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, comprenant en outre :

- l'analyse de l'image numérique reçue pour déterminer une densité de cellules immunitaires des cellules immunitaires identifiées ou d'un type particulier de cellules immunitaires ;

- dans lequel la classification des cellules tumorales identifiées en cellules tumorales d'une tumeur inflammatoire ou en cellules tumorales d'une tumeur non inflammatoire comprend la saisie de la mesure de proximité et de la densité de cellules immunitaires dans un classifieur conçu pour effectuer la classification.


 
7. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, l'identification des cellules tumorales comprenant :

- l'identification d'une prolifération de cellules non lymphoïdes et l'utilisation desdites cellules identifiées en tant que cellules tumorales ; et/ou

- l'identification de cellules exprimant un ensemble d'un ou de plusieurs biomarqueurs spécifiques à une tumeur et l'utilisation desdites cellules identifiées en tant que cellules tumorales.


 
8. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes,

- l'identification des cellules immunitaires comprenant l'identification de cellules cytotoxiques T et l'utilisation des cellules cytotoxiques T identifiées en tant que cellules immunitaires identifiées ; ou

- l'identification des cellules immunitaires comprenant l'identification de lymphocytes auxiliaires T et l'utilisation des lymphocytes auxiliaires T identifiés en tant que cellules immunitaires identifiées ; ou

- l'identification des cellules immunitaires comprenant l'identification de macrophages et l'utilisation des macrophages identifiés en tant que cellules immunitaires identifiées ; ou

- l'identification des cellules immunitaires comprenant l'identification de cellules à mémoire et l'utilisation des cellules à mémoire identifiées en tant que cellules immunitaires identifiées ; ou

- l'identification des cellules immunitaires comprenant l'identification de cellules B et l'utilisation des cellules à mémoire identifiés en tant que cellules immunitaires identifiées ; et/ou

- l'identification des cellules immunitaires comprenant l'identification de cellules activées T et l'utilisation des cellules activées T identifiées en tant que cellules immunitaires identifiées ; ou

- l'identification des cellules immunitaires comprenant l'identification de cellules immunitaires PD1+ et l'utilisation des cellules immunitaires PD1+ identifiées en tant que cellules immunitaires identifiées, l'identification des cellules tumorales comprenant l'identification de cellules tumorales PDL1+ et l'utilisation des cellules tumorales PDL1+ identifiées en tant que cellules tumorales.


 
9. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes,

- l'identification des cellules immunitaires comprenant l'identification de cellules régulatrices T et l'identification de cellules immunitaires d'au moins un type de cellules immunitaires qui renforcent une réponse immunitaire ; et

- l'utilisation sélective des cellules immunitaires identifiées qui renforcent la réponse immunitaire mais non des cellules régulatrices T identifiées en tant que cellules immunitaires identifiées.


 
10. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, le procédé comprenant en outre :

- sélectivement, dans le cas où le résultat de la classification indique que les cellules tumorales identifiées font partie d'une tumeur inflammatoire, l'émission d'un signal étant indicateur d'une recommandation de traitement pour utiliser une substance qui renforce ou module une réponse immunitaire en tant que médicament pour traiter la tumeur.


 
11. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, le procédé comprenant en outre :

- l'utilisation de la mesure de proximité en complément ou en remplacement d'un indicateur d'état de « microsatellites stables », MSS, ou de « microsatellites instables », MSI, pour calculer informatiquement un pronostic de la réponse de la tumeur à une substance qui renforce ou module une réponse immunitaire.


 
12. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, les cellules tumorales identifiées étant des cellules de cancer colorectal.
 
13. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, l'identification des cellules immunitaires comprenant l'identification de cellules immunitaires de multiples types de cellules immunitaires différents en analysant des valeurs d'intensité de pixels dans l'au moins une image numérique représentant la présence de différents biomarqueurs spécifiques à un type de cellules immunitaires, et l'utilisation de la totalité des cellules immunitaires identifiées parmi les multiples types de cellules immunitaires différents en tant que cellules immunitaires identifiées pour le calcul informatique de la mesure de proximité.
 
14. Procédé d'analyse d'image selon l'une quelconque des revendications précédentes, les cellules tumorales étant identifiées en tant que cellules exprimant au moins un premier biomarqueur spécifique à un type de tumeur, les cellules immunitaires étant identifiées en tant que cellules exprimant au moins un premier biomarqueur spécifique à un type de cellules immunitaires, le procédé comprenant en outre :

- l'analyse de l'au moins une image reçue ou d'une autre image de l'échantillon de tissu pour identifier d'autres cellules immunitaires en tant que cellules exprimant un deuxième biomarqueur spécifique à un type de cellules immunitaires ; et

- l'analyse de l'au moins une image ou de l'autre image reçue de l'échantillon de tissu pour identifier d'autres cellules tumorales en tant que cellules exprimant un deuxième biomarqueur spécifique d'un type de tumeur ou en utilisant les cellules immunitaires identifiées en tant qu'autres cellules tumorales ;

- pour chacune des autres cellules tumorales identifiées, la détermination de la distance de ladite cellule tumorale jusqu'à la cellule immunitaire la plus proche ;

- le calcul informatique d'une nouvelle mesure de proximité sous forme d'une fonction des dites distances déterminées ;

- en fonction de la nouvelle mesure de proximité, le classement secondaire des cellules tumorales identifiées et/ou la prédiction de la possibilité de traitement de la tumeur par une classe de substances.


 
15. Système d'analyse d'image (100) pour la classification de tumeurs, le système comprenant un processeur (104) étant conçu pour l'exécution d'un procédé selon l'une quelconque des revendications précédentes 1 à 14.
 




Drawing






































Cited references

REFERENCES CITED IN THE DESCRIPTION



This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard.

Patent documents cited in the description




Non-patent literature cited in the description