(19)
(11)EP 3 572 822 A1

(12)EUROPEAN PATENT APPLICATION

(43)Date of publication:
27.11.2019 Bulletin 2019/48

(21)Application number: 18173760.2

(22)Date of filing:  23.05.2018
(51)International Patent Classification (IPC): 
G01R 33/48(2006.01)
G01R 33/56(2006.01)
G06T 11/00(2006.01)
A61N 5/10(2006.01)
(84)Designated Contracting States:
AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR
Designated Extension States:
BA ME
Designated Validation States:
KH MA MD TN

(71)Applicant: RaySearch Laboratories AB
103 65 Stockholm (SE)

(72)Inventors:
  • Fredriksson, Albin
    11869 Stockholm (SE)
  • Andersson, Sebastian
    111 34 Stockholm (SE)
  • Nilsson, Rasmus
    111 34 Stockholm (SE)
  • Nordström, Marcus
    SE-121 46 Johanneshov (SE)

  


(54)MEDICAL IMAGE CONVERSION


(57) In accordance with one or more embodiments herein, a system 100 for generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type is provided. The system 100 comprises at least one processing unit configured to: obtain original medical images of the first and second image types; obtain an initial parametrized conversion function G to convert original medical images of the first image type into converted medical images of the second image type; calculate a first penalty P1 based on at least one comparison of a first original medical image of the second image type with a first converted medical image of the second image type, that has been generated by applying a first parametrized conversion function G1, which is based on the initial parametrized conversion function G, to a first original medical image of the first image type that forms an image pair with the first original medical image of the second image type and has thereby been determined to show the same part of the same patient; calculate a second penalty P2 based on at least one comparison of an original medical image of the first image type and a converted medical image of the second image type that has been generated by applying a second parametrized conversion function G2, which is based on the initial parametrized conversion function G, to the original medical image of the first image type, after converting the original medical image of the first image type and/or the converted medical image of the second image type into images of the same image type; and generate the optimized parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first and second penalties P1 and P2.




Description

TECHNICAL FIELD



[0001] The present disclosure relates generally to systems and methods for generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type.

BACKGROUND



[0002] Computed tomography (CT) imaging is useful for many purposes. CT images may e.g. be used for dose calculation in radiation treatment planning. However, there are reasons for avoiding CT imaging if other ways of obtaining images that can be used for dose calculation can be found - CT examination takes time, and patients are exposed to radiation.

[0003] It is possible to use cone beam computed tomography (CBCT) images for dose calculation, but CBCT images are more prone to artefacts than CT images, and the intensity scale may also be different than in CT images.

[0004] The article Deep MR to CT Synthesis using Unpaired Data (Wolterink et al. 2017) describes the training of a general adversarial network (GAN) to convert MR images into CT images using a CycleGAN model and unpaired MR and CT images.

[0005] The article Medical Image Synthesis with Context-Aware Generative Adversarial Networks (Nie et al. 2016) describes the training of a fully convolutional network to convert MR images into CT images using the adversarial training strategy and an image gradient difference loss function.

PROBLEMS WITH THE PRIOR ART



[0006] In the approach described in the article Deep MR to CT Synthesis using Unpaired Data, the fact that only unpaired data is used may make the generated conversion function less accurate. In this approach there is also a risk of introducing false information, which may never be detected.

[0007] In the approach described in the article Medical Image Synthesis with Context-Aware Generative Adversarial Networks, the fact that no comparison is made between the converted image and the original image may make the generated conversion function less accurate. This approach also requires a lot of accurate paired data.

[0008] There is thus a need for an improved method of generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type.

SUMMARY



[0009] The above described problem is addressed by the claimed system for generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type. The system may comprise at least one processing unit configured to: obtain original medical images of the first and second image types; obtain an initial parametrized conversion function G to convert original medical images of the first image type into converted medical images of the second image type; calculate a first penalty P1 based on at least one comparison of a first original medical image of the second image type with a first converted medical image of the second image type, that has been generated by applying a first parametrized conversion function G1, which is based on the initial parametrized conversion function G, to a first original medical image of the first image type that forms an image pair with the first original medical image of the second image type and has thereby been determined to show the same part of the same patient; calculate a second penalty P2 based on at least one comparison of an original medical image of the first image type and a converted medical image of the second image type that has been generated by applying a second parametrized conversion function G2, which is based on the initial parametrized conversion function G, to the original medical image of the first image type, after converting the original medical image of the first image type and/or the converted medical image of the second image type into images of the same image type; and generate the optimized parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first and second penalties P1 and P2., e.g. by using estimates of the gradients of said first and second penalties P1 and P2, e.g. to adapt the parameters to minimize said first and second penalties P1 and P2. Such a system enables the generation of a very accurate optimized parametrized conversion function T.

[0010] In embodiments, the at least one processing unit is further configured to: obtain a first parametrized discriminator function D1 to discriminate between original medical images of the second image type and converted medical images of the second image type; and calculate a third penalty P3 based on a classification error of the first parametrized discriminator function D1 when trying to discriminate between at least one original medical image of the second image type and at least one converted medical image of the second image type that has been generated by applying a third parametrized conversion function G3, which is based on the initial parametrized conversion function G, to at least one original medical image of the first image type. The at least one processing unit may in this case generate the parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first, second and third penalties P1, P2 and P3, e.g. by using estimates of the gradients of said first, second and third penalties P1, P2 and P3, e.g. to adapt the parameters to minimize said first, second and third penalties P1, P2 and P3. The at least one processing unit may e.g. obtain the first parametrized discriminator function D1 based on the parameters of the parametrized discriminator function D1 used for generating said optimized parametrized conversion function T and at least said third penalty P3, e.g. by using estimates of the gradient of said third penalty P3, e.g. to adapt the parameters to minimize said third penalty P3, so that the first parametrized discriminator function D1 is iteratively optimized. This further improves the accuracy of the optimized parametrized conversion function T.

[0011] In embodiments, the at least one processing unit is further configured to: obtain a second parametrized discriminator function D2 to discriminate between a) image pairs with corresponding original medical images, an original medical image of the first image type and an original medical image of the second image type, which have been paired and thereby determined to show the same part of the same patient; and b) image pairs of an original medical image of the first image type and a corresponding converted medical image of the second image type; and calculate a fourth penalty P4 based on a classification error of the second parametrized discriminator function D2. The at least one processing unit may in this case generate the parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first, second and fourth penalties P1, P2 and P4, e.g. by using estimates of the gradients of said first, second and fourth penalties P1, P2 and P4, e.g. to adapt the parameters to minimize said first, second and fourth penalties P1, P2 and P4. The at least one processing unit may e.g. obtain the second parametrized discriminator function D2 based on the parameters of the parametrized discriminator function D2 used for generating said optimized parametrized conversion function T and at least said fourth penalty P4, e.g. by using estimates of the gradient of said fourth penalty P4, e.g. to adapt the parameters to minimize said fourth penalty P4, so that the second parametrized discriminator function D2 is iteratively optimized. This further improves the accuracy of the optimized parametrized conversion function T.

[0012] The third and fourth penalties may of course also be combined, so that the at least one processing unit generates the parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first, second, third and fourth penalties P1, P2, P3 and P4, e.g. by using estimates of the gradient of said first, second, third and fourth penalties P1, P2, P3 and P4, e.g. to adapt the parameters to minimize said first, second, third and fourth penalties P1, P2, P3 and P4. This further improves the accuracy of the optimized parametrized conversion function T.

[0013] The at least one processing unit may e.g. obtain the initial parametrized conversion function G by generating it using a machine learning algorithm such as a neural network, a random forest, or a support vector machine. The machine learning algorithm may e.g. be a Generative Adversarial Network (GAN). The at least one processing unit may in further iterations obtain the initial parametrized conversion function G by using a previously generated optimized parametrized conversion function T as the initial parametrized conversion function G, so that the parametrized conversion function T is iteratively optimized.

[0014] The at least one processing unit may e.g. compare the first original medical image with the corresponding first converted medical image by voxel wise comparing the images, after first converting both images into a third image type such as e.g. a gradient image, a segmented image, or the output from a pre-trained machine learning algorithm such as a convolutional neural network, wherein the third image type amplifies certain features such as contours, regions-of-interest (ROI) and/or structures in the images so that they may more easily be compared. Alternatively, the at least one processing unit may convert the original medical image of the first image type into the second image type, or the converted medical image of the second image type into the first image type, before the comparison of the images.

[0015] The at least one processing unit may be one processing unit, or comprise several different processing units which together perform the claimed functions.

[0016] The above described problem is further addressed by the claimed processor-implemented method of generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type. The method may comprise: obtaining original medical images of the first and second image types; obtaining an initial parametrized conversion function G to convert original medical images of the first image type into converted medical images of the second image type; calculating a first penalty P1 based on at least one comparison of a first original medical image of the second image type with a first converted medical image of the second image type, that has been generated by applying a first parametrized conversion function G1, which is based on the initial parametrized conversion function G, to a first original medical image of the first image type that forms an image pair with the first original medical image of the second image type and has thereby been determined to show the same part of the same patient; calculating a second penalty P2 based on at least one comparison of an original medical image of the first image type and a converted medical image of the second image type that has been generated by applying a second parametrized conversion function G2, which is based on the initial parametrized conversion function G, to the original medical image of the first image type, after converting the original medical image of the first image type and/or the converted medical image of the second image type into images of the same image type; and generating the optimized parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first and second penalties P1 and P2., e.g. by using estimates of the gradients of said first and second penalties P1 and P2, e.g. to adapt the parameters to minimize said first and second penalties P1 and P2. Such a method enables the generation of a very accurate optimized parametrized conversion function T.

[0017] In embodiments, the method further comprises: obtaining a first parametrized discriminator function D1 to discriminate between original medical images of the second image type and converted medical images of the second image type; and calculating a third penalty P3 based on a classification error of the first parametrized discriminator function D1 when trying to discriminate between at least one original medical image of the second image type and at least one converted medical image of the second image type that has been generated by applying a third parametrized conversion function G3, which is based on the initial parametrized conversion function G, to at least one original medical image of the first image type. The parametrized conversion function T may in this case be generated based on the parameters of the initial parametrized conversion function G and at least said first, second and third penalties P1, P2 and P3, e.g. by using estimates of the gradients of said first, second and third penalties P1, P2 and P3, e.g. to adapt the parameters to minimize said first, second and third penalties P1, P2 and P3. The first parametrized discriminator function D1 may e.g. be obtained based on the parameters of the parametrized discriminator function D1 used for generating said optimized parametrized conversion function T and at least said third penalty P3, e.g. by using estimates of the gradient of said third penalty P3, e.g. to adapt the parameters to minimize said third penalty P3, so that the first parametrized discriminator function D1 is iteratively optimized. This further improves the accuracy of the optimized parametrized conversion function T.

[0018] In embodiments, the method further comprises: obtaining a second parametrized discriminator function D2 to discriminate between a) image pairs with corresponding original medical images, an original medical image of the first image type and an original medical image of the second image type, which have been paired and thereby determined to show the same part of the same patient; and b) image pairs of an original medical image of the first image type and a corresponding converted medical image of the second image type; and calculating a fourth penalty P4 based on a classification error of the second parametrized discriminator function D2. The parametrized conversion function T may in this case be generated based on the parameters of the initial parametrized conversion function G and at least said first, second and fourth penalties P1, P2 and P4, e.g. by using estimates of the gradients of said first, second and fourth penalties P1, P2 and P4, e.g. to adapt the parameters to minimize said first, second and fourth penalties P1, P2 and P4. The second parametrized discriminator function D2 may e.g. be obtained based on the parameters of the parametrized discriminator function D2 used for generating said optimized parametrized conversion function T and at least said fourth penalty P4, e.g. by using estimates of the gradient of said fourth penalty P4, e.g. to adapt the parameters to minimize said fourth penalty P4, so that the second parametrized discriminator function D2 is iteratively optimized. This further improves the accuracy of the optimized parametrized conversion function T.

[0019] The third and fourth penalties may of course also be combined, so that the parametrized conversion function T is generated based on the parameters of the initial parametrized conversion function G and at least said first, second, third and fourth penalties P1, P2, P3 and P4, e.g. by using estimates of the gradients of said first, second, third and fourth penalties P1, P2, P3 and P4, e.g. to adapt the parameters to minimize said first, second, third and fourth penalties P1, P2, P3 and P4. This further improves the accuracy of the optimized parametrized conversion function T.

[0020] The initial parametrized conversion function G may e.g. be obtained by being generated using a machine learning algorithm such as a neural network, a random forest, or a support vector machine. The machine learning algorithm may e.g. be a Generative Adversarial Network (GAN). In further iterations, the initial parametrized conversion function G may be obtained by a previously generated optimized parametrized conversion function T being used as the initial parametrized conversion function G, so that the parametrized conversion function T is iteratively optimized.

[0021] In embodiments, the penalties are combined in the form of a weighted sum of the penalties, so that the parameters may e.g. be adapted based on the gradient of the weighted sum of the penalties. In other embodiments, the penalties are instead applied one after the other, so that the parameters are adapted based on one penalty at a time.

[0022] The comparing of the first original medical image with the corresponding first converted medical image may e.g. voxel wise compare the images, after first converting both images into a third image type such as e.g. a gradient image, a segmented image, or the output from a pre-trained machine learning algorithm such as a convolutional neural network, wherein the third image type amplifies certain features such as contours, regions-of-interest (ROI) and/or structures in the images so that they may more easily be compared. Alternatively, the original medical image of the first image type may be converted into the second image type, or the converted medical image of the second image type may be converted into the first image type, before the comparison of the images.

[0023] The first parametrized conversion function G1 and/or the second parametrized conversion function G2 may e.g. be the same as the initial parametrized conversion function G. They may however also be different, e.g. by the first parametrized conversion function G1 being based on the parameters of the initial parametrized conversion function G and the first penalty P1, and the second parametrized conversion function G2 being based on the parameters of either the initial parametrized conversion function G or the first parametrized conversion function G1 and the second penalty P2. The same may apply to the third and the fourth parametrized conversion functions G3 and G4. If the penalties are applied one after the other, the parametrized conversion function adapted based on one penalty may thus e.g. be used when calculating the subsequent penalty.

[0024] The image type may e.g. be a medical image modality, such as e.g. MR, CT or CBCT. In other embodiments, the image type is the output from a pre-trained machine learning algorithm such as a convolutional neural network. In other embodiments, the medical images are converted into a third image type by being segmented, by being transferred into another domain by e.g. the gradient or the frequency content in the images being calculated, or by the resolution the images being changed. In other embodiments, the first image type is a medical image of a certain medical image modality, and the second image type is an image showing the density, the stopping power, the material composition, and/or any other feature that may be used for dose calculation. In other embodiments, the first image type is a segmented image and the second image type is a medical image of a certain medical image modality. This may be useful e.g. for creating hypothetical patient images in order to simulate radiation treatment based on such hypothetical patient images.

[0025] The above described problem is further addressed by the optimized parametrized conversion function T generated by any one of the methods described above, and by the use of the optimized parametrized conversion function T generated by any one of the methods described above for converting an original medical image of a first image type into a converted medical image of a second image type, which may e.g. be used for dose calculation in radiation treatment planning, where machine instructions e.g. in the form of a control signal may be sent to radiation treatment equipment based on an analysis of the converted medical image of the second image type.

[0026] The scope of the invention is defined by the claims, which are incorporated into this section by reference. A more complete understanding of embodiments of the invention will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS



[0027] 

Figure 1 schematically illustrates a system for generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type, in accordance with one or more embodiments described herein.

Figure 2 schematically illustrates a calculation of a first penalty P1 and a second penalty P2, in accordance with one or more embodiments described herein.

Figure 3 schematically illustrates a calculation of a third penalty P3, in accordance with one or more embodiments described herein.

Figure 4 schematically illustrates a calculation of a fourth penalty P4, in accordance with one or more embodiments described herein.

Figure 5 schematically illustrates a calculation of a second penalty P2, in accordance with one or more embodiments described herein.

Figure 6 schematically illustrates a processor-implemented method of generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type, in accordance with one or more embodiments described herein.

Figure 7 is a flow diagram exemplifying the steps of the method.



[0028] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures.

DETAILED DESCRIPTION


Introduction



[0029] As explained above, a number of different methods of training a machine learning algorithm to generate CT images based on MR images are known. However, the known methods all rely on either using paired medical images to train the algorithm, or using a feedback model, such as e.g. a CycleGAN model, to train the network using unpaired medical images. It is e.g. explained in the article Deep MR to CT Synthesis using Unpaired Data, which refers to the article Medical Image Synthesis with Context-Aware Generative Adversarial Networks, that with the use of a CycleGAN model it is not necessary to use paired medical images for training.

[0030] The present inventors have however discovered that training of the machine learning algorithm based on a combination of paired medical images and comparisons of the converted medical image with the original medical image is even more powerful, and leads to an improved machine learning algorithm. The use of paired data greatly reduces the risk of introducing false information. The machine learning algorithm may be even further improved by also being trained based on the use of discriminator functions.

[0031] Paired medical images are original medical images of different image types which have been determined to show the same part of the same patient, such as e.g. the head or any other body part of the patient. The image type may e.g. be a medical image modality, such as e.g. magnetic resonance imaging (MR) images, computed tomography (CT) images, cone beam computed tomography (CBCT) images or positron emission tomography (PET) images. In other embodiments, the image type is the output from a pre-trained machine learning algorithm such as a convolutional neural network. In other embodiments, the medical images are converted into a third image type by being segmented, by being transferred into another domain by e.g. the gradient or the frequency content in the images being calculated, or by the resolution the images being changed. In other embodiments, the first image type is a medical image of a certain medical image modality, and the second image type is an image showing the density, the stopping power, the material composition, and/or any other feature that may be used for dose calculation. In other embodiments, the first image type is a segmented image and the second image type is a medical image of a certain medical image modality.

[0032] The system and method may e.g. be used to convert images of a first image type into images of a second image type, e.g. MR images into CT images, or CBCT images into CT images, or any first image type into a second image type showing the density, the stopping power, the material composition, and/or any other feature that may be used for dose calculation. The converted images may e.g. be used for dose calculation in radiation treatment planning, and machine instructions e.g. in the form of a control signal may e.g. be sent to radiation treatment equipment based on an analysis of the converted images. The system and method may also, or alternatively, be used for removing artefacts in images.

[0033] The present disclosure relates generally to systems and methods for generating a parametrized conversion function for converting an original medical image of a first image type into a converted medical image of a second image type.

[0034] Embodiments of the disclosed solution are presented in more detail in connection with the figures.

System architecture



[0035] Figure 1 schematically illustrates a system 100 for generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type. According to embodiments, the system comprises a storage media 110, configured to receive and store images and parameters. In some embodiments, the system 100 is communicatively coupled, as indicated by the dashed arrow in figure 1, to at least one imaging system 130. The imaging system 130 may be configured to capture or generate original medical images, such as for example X-ray images, ultrasound images, computed tomography (CT) images, cone beam computed tomography (CBCT) images, nuclear medicine including positron emission tomography (PET) images, and/or magnetic resonance imaging (MR) images. The storage media 110 may be configured to receive original medical images from the at least one imaging system 130 and store the received original medical images.

[0036] The system 100 further comprises at least one processing unit 120 configured to generate an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type. The at least one processing unit 120 may for example be a general data processor, or other circuit or integrated circuit capable of executing instructions to perform various processing operations. The at least one processing unit 120 may be one processing unit, or comprise several different processing units which together perform the claimed functions.

[0037] In one or more embodiments, the at least one processing unit 120 is configured to: obtain original medical images of the first and second image types, e.g. from the storage media 110; obtain an initial parametrized conversion function G to convert original medical images of the first image type into converted medical images of the second image type, e.g. from the storage media 110; calculate a first penalty P1 based on at least one comparison of a first original medical image of the second image type with a first converted medical image of the second image type, that has been generated by applying a first parametrized conversion function G1, which is based on the initial parametrized conversion function G, to a first original medical image of the first image type that forms an image pair with the first original medical image of the second image type and has thereby been determined to show the same part of the same patient; calculate a second penalty P2 based on at least one comparison of an original medical image of the first image type and a converted medical image of the second image type that has been generated by applying a second parametrized conversion function G2, which is based on the initial parametrized conversion function G, to the original medical image of the first image type, after converting the original medical image of the first image type and/or the converted medical image of the second image type into images of the same image type; and generate the optimized parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first and second penalties P1 and P2, e.g. by using estimates of the gradients of said first and second penalties P1 and P2, e.g. to adapt the parameters to minimize said first and second penalties P1 and P2. The optimized parametrized conversion function T may e.g. be stored in the storage media 110.

[0038] The first penalty P1 is thus based on paired images, by comparing the converted medical image of the second image type with the paired original medical image of the second image type. What may be compared, e.g. voxel wise, may e.g. be features such as contours, regions-of-interest (ROI) and/or structures in the images, e.g. using a pre-trained machine learning algorithm such as a convolutional neural network. If the converted medical image is very similar to the paired original medical image, the first parametrized conversion function G1 has very accurately converted the original medical image of the first image type into a converted medical image of the second image type, and the penalty P1 will thus be low. If the converted medical image is very different from the paired original medical image, the first parametrized conversion function G1 has not accurately converted the original medical image of the first image type into a converted medical image of the second image type, and the penalty P1 will thus be high. In this case, the at least one processing unit 120 may adapt the parameters of the optimized parametrized conversion function T, e.g. to minimize the penalty P1. This may be done for any number of paired images.

[0039] The second penalty P2 is based on comparisons, e.g. voxel wise, between converted medical images and original medical images, after converting them into images of the same image type. This may be done by either converting one of the images into an image of the other image type, or converting both images into images of a third image type. In order to voxel wise compare the images, they must however be of the same image type.

[0040] The image type does not have to be a medical image modality. In other embodiments, the image type is the output from a pre-trained machine learning algorithm such as a convolutional neural network. In other embodiments, the medical images are converted into a third image type by being segmented, by being transferred into another domain by e.g. the gradient or the frequency content in the images being calculated, or by the resolution the images being changed. In other embodiments, the first image type is a medical image of a certain medical image modality, and the second image type is an image showing the density, the stopping power, the material composition, and/or any other feature that may be used for dose calculation. In other embodiments, the first image type is a segmented image and the second image type is a medical image of a certain medical image modality. This may be useful e.g. for creating hypothetical patient images in order to simulate radiation treatment based on such hypothetical patient images.

[0041] What may be compared may e.g. be features such as contours, regions-of-interest (ROI) and/or structures in the images, e.g. using a pre-trained machine learning algorithm such as a convolutional neural network. If these properties of the two images are very similar, the second parametrized conversion function G2 has very accurately converted the original medical image of the first image type into a converted medical image of the second image type, and the penalty P2 will thus be low. If these properties of the two images are very different, the second parametrized conversion function G2 has not accurately converted the original medical image of the first image type into a converted medical image of the second image type, and the penalty P2 will thus be high. In this case, the at least one processing unit 120 may adapt the parameters of the optimized parametrized conversion function T, e.g. to minimize the penalty P2. This may be done for any number of converted medical images.

[0042] Figure 2 schematically illustrates a calculation of the first and second penalties P1 and P2 in a situation where a CT image is converted into an MR image (to the left in the figure) and an MR image is converted into a CT image (to the right in the figure) - each half of figure 2 is an independent process. CTp and MRp are paired original medical images, and CTr and MRr are unpaired original medical images. MRr* and MRp* are medical images which have been converted from CT images into MR images using a conversion function GMR, and CTp* and CTr* are medical images which have been converted from MR images into CT images using a conversion function GCT. CTcyc is a medical image which has been converted from a CT image into an MR image using a conversion function GMR and then converted back into a CT image using another conversion function HCT, and MRcyc is a medical image which has been converted from an MR image into a CT image using a conversion function GCT and then converted back into an MR image using another conversion function HMR. In the embodiment described in figure 2, GMR and HMR are typically the same function, and GCT and HCT are typically the same function.

[0043] P1 is in this illustration labelled "paired loss", since P1 is determined based on a comparison with a paired image. P1 may according to figure 2 be determined either by comparing MRp* with MRp, or by comparing CTp* with CTp. What may be compared, e.g. voxel wise, may e.g. be features such as contours, regions-of-interest (ROI) and/or structures in the images, e.g. using a pre-trained machine learning algorithm such as a convolutional neural network.

[0044] P2 is in this illustration labelled cyclic loss, since P2 may be determined based on a comparison between the original image and the converted image, after converting the converted image back into its original image type, in a cyclic manner. P2 may in this case determine the penalty to be used for optimizing the parameters of both G and H. P2 may according to figure 2 be determined by comparing CTr with CTcyc, which is CTr converted using a conversion function GMR and converted back using another conversion function HCT. P2 may according to figure 2 also or alternatively be determined by comparing MRr with MRcyc, which is MRr converted using a conversion function GCT and converted back using another conversion function HMR. The comparison may e.g. be voxel wise.

[0045] In one or more embodiments, the at least one processing unit 120 is further configured to: obtain a first parametrized discriminator function D1 to discriminate between original medical images of the second image type and converted medical images of the second image type; and calculate a third penalty P3 based on a classification error of the first parametrized discriminator function D1 when trying to discriminate between at least one original medical image of the second image type and at least one converted medical image of the second image type that has been generated by applying a third parametrized conversion function G3, which is based on the initial parametrized conversion function G, to at least one original medical image of the first image type.

[0046] The at least one processing unit 120 may in this case generate the parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first, second and third penalties P1, P2 and P3, e.g. by using estimates of the gradients of said first, second and third penalties P1, P2 and P3, e.g. to adapt the parameters to minimize said first, second and third penalties P1, P2 and P3. The at least one processing unit may e.g. obtain the first parametrized discriminator function D1 based on the parameters of the parametrized discriminator function D1 used for generating said optimized parametrized conversion function T and at least said third penalty P3, e.g. by using estimates of the gradient of said third penalty P3, e.g. to adapt the parameters to minimize said third penalty P3, so that the first parametrized discriminator function D1 is iteratively optimized.

[0047] The third penalty P3 is thus based on a classification error of a discriminator function when trying to discriminate between original and converted medical images of the same image type. The original medical image used may be any original medical image, it does not have to be paired with the original medical image on which the converted medical image is based. Such discriminator functions are often used for the training of adversarial networks. The conversion function tries to produce converted images that are realistic enough to fool the discriminator function, and the discriminator function tries to distinguish original and converted images. One way of doing this in a GAN framework is described in the article Medical Image Synthesis with Context-Aware Generative Adversarial Networks. The discriminator function is preferably iteratively trained based on at least the classification error P3, e.g. by using estimates of the gradient of the classification error P3, e.g. to adapt the parameters to minimize said third penalty P3.

[0048] Figure 3 schematically illustrates a calculation of the third penalty P3. P3 may according to figure 3 be determined based on a classification error of a first discriminator function D1CT when trying to distinguish between CTr and CTr*, and/or a classification error of a first discriminator function D1MR when trying to distinguish between MRr and MRr*. The classification error may e.g. be determined as a number between 0 and 1, representing the percentage or likelihood of a classification error.

[0049] In one or more embodiments, the at least one processing unit 120 is further configured to: obtain a second parametrized discriminator function D2 to discriminate between a) image pairs with corresponding original medical images, an original medical image of the first image type and an original medical image of the second image type, which have been paired and thereby determined to show the same part of the same patient; and b) image pairs of an original medical image of the first image type and a corresponding converted medical image of the second image type; and calculate a fourth penalty P4 based on a classification error of the second parametrized discriminator function D2. The at least one processing unit 120 may in this case generate the parametrized conversion function T based on the parameters of the initial parametrized conversion function G and of at least said first, second and fourth penalties P1, P2 and P4, e.g. by using estimates of the gradients of said first, second and fourth penalties P1, P2 and P4, e.g. to adapt the parameters to minimize said first, second and fourth penalties P1, P2 and P4. The at least one processing unit may e.g. obtain the second parametrized discriminator function D2 based on the parameters of the parametrized discriminator function D2 used for generating said optimized parametrized conversion function T and at least said fourth penalty P4, e.g. by using estimates of the gradient of said fourth penalty P4, e.g. to adapt the parameters to minimize said fourth penalty P4, so that the second parametrized discriminator function D2 is iteratively optimized.

[0050] The fourth penalty P4 is thus based on a classification error of a discriminator function when trying to discriminate between pairs of original paired medical images and pairs of original and corresponding converted medical images.

[0051] Figure 4 schematically illustrates a calculation of the fourth penalty P4. P4 may according to figure 4 be determined based on a classification error of the second discriminator function D2 when trying to distinguish between the image pair CTp and MRp and the image pair CTr* and MRr. A corresponding penalty P4 may of course also be calculated in the process to the left in figure 4. The classification error may e.g. be determined as a number between 0 and 1, representing the percentage or likelihood of a classification error.

[0052] The third and fourth penalties may of course also be combined, so that the at least one processing unit generates the parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first, second, third and fourth penalties P1, P2, P3 and P4, e.g. by using estimates of the gradients of said first, second, third and fourth penalties P1, P2, P3 and P4, e.g. to adapt the parameters to minimize said first, second, third and fourth penalties P1, P2 P3 and P4. This further improves the accuracy of the optimized parametrized conversion function T.

[0053] In embodiments, the penalties are combined in the form of a weighted sum of the penalties, so that the parameters are adapted e.g. based on the gradient of the weighted sum of the penalties. In other embodiments, the penalties are instead applied one after the other, so that the parameters are adapted based on one penalty at a time.

[0054] Figure 5 schematically illustrates a calculation of a second penalty P2 in an embodiment where both the images used for calculating P2 are converted into a third image type before they are compared. CTgrad in figure 5 corresponds to an original medical image MRr of a first image type (MR image) which has been converted into a converted medical image CTr* of a second image type (CT image) using a conversion function GCT, and then converted again, into an image CTgrad of a third image type, in this example a gradient image, using another conversion function FCT. MRgrad in figure 5 corresponds to the same original medical image MRr of the first image type (MR image) which has been converted into an image MRgrad of the same third image type, in this example a gradient image, using a conversion function FMR. In the example shown in figure 5, the conversion functions FCT and FMR is the same function, but the conversion functions FCT and FMR may also be different functions.

[0055] The at least one processing unit 120 may e.g. obtain the initial parametrized conversion function G by generating it using a machine learning algorithm such as a neural network, a random forest, or a support vector machine. The machine learning algorithm may e.g. be a Generative Adversarial Network (GAN). This is a useful way of obtaining an initial parametrized conversion function G that may then be optimized by the system.

[0056] The at least one processing unit 120 may in further iterations obtain the initial parametrized conversion function G by using a previously generated optimized parametrized conversion function T as the initial parametrized conversion function G. This allows for iterative optimization of the parametrized conversion function T.

[0057] The first parametrized conversion function G1 and/or the second parametrized conversion function G2 may e.g. be the same as the initial parametrized conversion function G. They may however also be different, e.g. by the first parametrized conversion function G1 being based on the parameters of the initial parametrized conversion function G and the first penalty P1, and the second parametrized conversion function G2 being based on the parameters of either the initial parametrized conversion function G or the first parametrized conversion function G1 and the second penalty P2. The same may apply to the third and the fourth parametrized conversion functions G3 and G4. If the penalties are applied one after the other, the parametrized conversion function adapted based on one penalty may thus e.g. be used when calculating the subsequent penalty.

[0058] The image type may e.g. be a medical image modality, such as e.g. MR, CT or CBCT. In other embodiments, the image type is the output from a pre-trained machine learning algorithm such as a convolutional neural network. In other embodiments, the medical images are converted into a third image type by being segmented, by being transferred into another domain by e.g. the gradient or the frequency content in the images being calculated, or by the resolution the images being changed. In other embodiments, the first image type is a medical image of a certain medical image modality, and the second image type is an image showing the density, the stopping power, the material composition, and/or any other feature that may be used for dose calculation. In other embodiments, the first image type is a segmented image and the second image type is a medical image of a certain medical image modality. This may be useful e.g. for creating hypothetical patient images in order to simulate radiation treatment based on such hypothetical patient images.

[0059] In one or more embodiments, the system 100 may optionally comprise a display 140 configured to display images. The display 140 may be configured to receive images for display via the at least one processing unit 120, and/or to retrieve images for display directly from the storage media 110, possibly in response to a control signal received from the at least one processing unit 120 or an inputter 150, which is further presented below.

[0060] In some embodiments, the system 100 may further optionally comprise one or more inputters 150 configured to receive user input. The inputter 150 is typically configured to interpret received user input and to generate control signals in response to said received user input. The display 140 and the inputter 150 may be integrated in, connected to or communicatively coupled to the system 100. The inputter 150 may for instance be configured to interpret received user input that is being input in connection with a displayed image, and generate control signals in response to said received user input, to trigger display of an image or manipulation of images being displayed, wherein the manipulations may be temporary or permanent. Such manipulations may for example include providing annotations, moving or changing an image or part of an image, changing the viewing perspective, zooming in or out, and/or any other suitable form of manipulation that enables the user to view and analyze the displayed image data in an improved manner. An inputter 150 may for example comprise a selection of a keyboard, a computer mouse, one or more buttons, touch functionality, a joystick, and/or any other suitable input device. In some embodiments, the processor 120 may be configured to receive a control signal from the inputter 150 and to process an image that is being displayed, or in other words manipulate a displayed image, in response to the received control signal.

[0061] The at least one processing unit 120 may further be configured to perform the method steps of any or all of the embodiments presented herein.

Method embodiments



[0062] Figure 6 schematically illustrates a processor-implemented method of generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type, in accordance with one or more embodiments described herein. The method 600 may comprise:
Step 610: obtaining original medical images of the first and second image types.

[0063] Step 620: obtaining an initial parametrized conversion function G to convert original medical images of the first image type into converted medical images of the second image type.

[0064] Step 630: calculating a first penalty P1 based on at least one comparison of a first original medical image of the second image type with a first converted medical image of the second image type, that has been generated by applying a first parametrized conversion function G1, which is based on the initial parametrized conversion function G, to a first original medical image of the first image type that forms an image pair with the first original medical image of the second image type and has thereby been determined to show the same part of the same patient.

[0065] Step 640: calculating a second penalty P2 based on at least one comparison of an original medical image of the first image type and a converted medical image of the second image type that has been generated by applying a second parametrized conversion function G2, which is based on the initial parametrized conversion function G, to the original medical image of the first image type, after converting the original medical image of the first image type and/or the converted medical image of the second image type into images of the same image type.

[0066] Step 650: generating the optimized parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first and second penalties P1 and P2.

[0067] The initial parametrized conversion function G may e.g. be obtained 620 by being generated using a machine learning algorithm such as a neural network, a random forest, or a support vector machine. The machine learning algorithm may e.g. be a Generative Adversarial Network (GAN). This is a useful way of obtaining 620 an initial parametrized conversion function G that may then be optimized.

[0068] In further iterations, the initial parametrized conversion function G may be obtained 620 by a previously generated optimized parametrized conversion function T being used as the initial parametrized conversion function G. This allows for iterative optimization of the parametrized conversion function T.

[0069] The comparing of the first original medical image with the corresponding first converted medical image may e.g. voxel wise compare the images. Before the comparison, the original medical image of the first image type may be converted into the second image type, the converted medical image of the second image type may be converted into the first image type, or both images may be converted into a third image type. The third image type may e.g. amplify certain features such as contours, regions-of-interest (ROI) and/or structures in the images so that they may more easily be compared. The third image type may e.g. be a gradient image, a segmented image or the output from a pre-trained machine learning algorithm such as a convolutional neural network.

[0070] The first parametrized conversion function G1 and/or the second parametrized conversion function G2 may e.g. be the same as the initial parametrized conversion function G. They may however also be different, e.g. by the first parametrized conversion function G1 being based on the parameters of the initial parametrized conversion function G and the first penalty P1, and the second parametrized conversion function G2 being based on the parameters of either the initial parametrized conversion function G or the first parametrized conversion function G1 and the second penalty P2. The same may apply to the third and the fourth parametrized conversion functions G3 and G4. If the penalties are applied one after the other, the parametrized conversion function adapted based on one penalty may thus e.g. be used when calculating the subsequent penalty.

[0071] The image type may e.g. be a medical image modality, such as e.g. MR, CT or CBCT. In other embodiments, the image type is the output from a pre-trained machine learning algorithm such as a convolutional neural network. In other embodiments, the medical images are converted into a third image type by being segmented, by being transferred into another domain by e.g. the gradient or the frequency content in the images being calculated, or by the resolution the images being changed. In other embodiments, the first image type is a medical image of a certain medical image modality, and the second image type is an image showing the density, the stopping power, the material composition, and/or any other feature that may be used for dose calculation. In other embodiments, the first image type is a segmented image and the second image type is a medical image of a certain medical image modality. This may be useful e.g. for creating hypothetical patient images in order to simulate radiation treatment based on such hypothetical patient images.

[0072] In embodiments, the method 600 further comprises:
Step 660: obtaining a first parametrized discriminator function D1 to discriminate between original medical images of the second image type and converted medical images of the second image type.

[0073] Step 670: calculating a third penalty P3 based on a classification error of the first parametrized discriminator function D1 when trying to discriminate between at least one original medical image of the second image type and at least one converted medical image of the second image type that has been generated by applying a third parametrized conversion function G3, which is based on the initial parametrized conversion function G, to at least one original medical image of the first image type.

[0074] The optimized parametrized conversion function T may in this case be generated 650 based on the parameters of the initial parametrized conversion function G and at least said first, second and third penalties P1, P2 and P3.

[0075] The first parametrized discriminator function D1 may e.g. be obtained 660 based on the parameters of the parametrized discriminator function D1 used for generating said optimized parametrized conversion function T and at least said third penalty P3, so that the first parametrized discriminator function D1 is iteratively optimized. This further improves the accuracy of the optimized parametrized conversion function T.

[0076] In embodiments, the method 600 further comprises:
Step 680: obtaining a second parametrized discriminator function D2 to discriminate between
  1. a) image pairs with corresponding original medical images, an original medical image of the first image type and an original medical image of the second image type, which have been paired and thereby determined to show the same part of the same patient; and
  2. b) image pairs of an original medical image of the first image type and a corresponding converted medical image of the second image type.


[0077] Step 690: calculating a fourth penalty P4 based on a classification error of the second parametrized discriminator function D2.

[0078] The parametrized conversion function T may in this case be generated 650 based on the parameters of the initial parametrized conversion function G and at least said first, second and fourth penalties P1, P2 and P4.

[0079] The third and fourth penalties may of course also be combined, so that the parametrized conversion function T is generated based on the parameters of the initial parametrized conversion function G and at least said first, second, third and fourth penalties P1, P2, P3 and P4.

[0080] In embodiments, the penalties are combined in the form of a weighted sum of the penalties, so that the parameters are adapted e.g. based on the gradient of the weighted sum of the penalties. In other embodiments, the penalties are instead applied one after the other, so that the parameters are adapted based on one penalty at a time.

Use case embodiment



[0081] To set the presently disclosed systems and methods in a larger context, the generating of an optimized parametrized conversion function T may in use case embodiments be preceded by the generating of original medical images.

[0082] Figure 7 is a flow diagram exemplifying such a larger context, including the steps of: obtaining original medical images; obtaining an initial parametrized conversion function; calculating a first penalty P1 based on paired images, by comparing the converted medical image of the second image type with the paired original medical image of the second image type; calculating a second penalty P2 based on comparisons between converted medical images and original medical images, after converting them into images of the same image type; and generating an optimized parametrized conversion function based on the parameters of the initial parametrized conversion function and at least said first and second penalties P1 and P2. Figure 7 further includes steps of: generating the original medical images, e.g. by capturing them using image capturing systems such as magnetic resonance imaging (MR), computed tomography (CT), cone beam computed tomography (CBCT) and/or positron emission tomography (PET); calculating third and fourth penalties P3 and P4, and feeding back the conversion function for iteration. In figure 7, everything except the obtaining of original medical images, the obtaining of an initial parametrized conversion function, the calculating of a first penalty P1, the calculating of a second penalty P2, and optimizing the parametrized conversion function, is marked with dashed lines to clarify that they are optional steps shown in the figure to provide context only, and not essential to any of the embodiments presented herein. Especially, steps 710 and 720 relating to the generation of original medical images are not part of the embodiments presented herein.

[0083] According to the example shown in figure 7, an original medical image of a first image type may be generated in a step 710, e.g. by being captured by an image capturing system 130 using magnetic resonance imaging (MR), cone beam computed tomography (CBCT) and/or positron emission tomography

[0084] (PET). An original medical image of a second image type may be generated in a step 720, e.g. by being captured using computed tomography (CT). These original medical images may be paired if they have been determined to show the same part of the same patient.

[0085] The original medical images may in step 730 be obtained either directly from the image capturing system 130 or from storage media 110. The initial parametrized conversion function G may in step 740 be obtained either by being determined by the at least one processing unit 120, e.g. using a machine learning algorithm, by being retrieved from storage media 110, or by feedback through using a previously generated parametrized conversion function T as the initial parametrized conversion function G, thereby iteratively optimizing the parametrized conversion function T.

[0086] In step 750, the initial parametrized conversion function G, or an adapted parametrized conversion function G1, is applied to an original medical image of the first image type so that a converted medical image of the second image type is generated. This converted medical image is then compared to an original medical image of the second image type that forms an image pair with the original medical image of the first image type, so that it can be determined how closely the converted medical image of the second image type matches the original medical image of the second image type that shows the same the same part of the same patient. A first penalty P1 is then calculated based on this comparison, which may e.g. be done voxel wise, and may e.g. compare features such as contours, regions-of-interest (ROI) and/or structures in the images, e.g. using a pre-trained machine learning algorithm such as a convolutional neural network.

[0087] In step 760, the initial parametrized conversion function G, or an adapted parametrized conversion function G2, is applied to an original medical image of the first image type so that a converted medical image of the second image type is generated. The images are then converted into the same image type. This may be done by either converting one of the images into an image of the other image type, or converting both images into images of a third image type. In order to compare the images, they must however be of the same image type. The image type does not have to be a medical image modality. In other embodiments, the image type is the output from a pre-trained machine learning algorithm such as a convolutional neural network. In other embodiments, the medical images are converted into a third image type by being segmented, by being transferred into another domain by e.g. the gradient or the frequency content in the images being calculated, or by the resolution the images being changed. In other embodiments, the first image type is a medical image of a certain medical image modality, and the second image type is an image showing the density, the stopping power, the material composition, and/or any other feature that may be used for dose calculation. In other embodiments, the first image type is a segmented image and the second image type is a medical image of a certain medical image modality. What may be compared, e.g. voxel wise, may e.g. be features such as contours, regions-of-interest (ROI) and/or structures in the images, e.g. using a pre-trained machine learning algorithm such as a convolutional neural network. A second penalty P2 is then calculated based on this comparison.

[0088] In step 770, which is an optional step, the initial parametrized conversion function G, or an adapted parametrized conversion function G3, is applied to an original medical image of the first image type so that a converted medical image of the second image type is generated. A first discriminator function D1 then tries to discriminate between the converted medical image of the second image type and an original medical image of the second image type. The original medical image used may be any original medical image, it does not have to be paired with the original medical image on which the converted medical image is based. A third penalty P3 is then calculated based on a classification error of the first discriminator function D1.

[0089] In step 780, which is also an optional step, the initial parametrized conversion function G, or an adapted parametrized conversion function G4, is applied to an original medical image of the first image type so that a converted medical image of the second image type is generated. A second discriminator function D2 then tries to discriminate between pairs of original paired medical images and pairs of original and corresponding converted medical images. A fourth penalty P4 is then calculated based on a classification error of the discriminator function D2.

[0090] In step 790, an optimized parametrized conversion function T is generated based on the parameters of the initial parametrized conversion function G and at least the first and second penalties P1 and P2, and optionally also the third penalty P3 and/or the fourth penalty P4. The penalties may be combined in the form of a weighted sum of the penalties, so that the parameters of the parametrized conversion function T are adapted based on the weighted sum of the penalties, or applied one after the other, so that the parameters of the parametrized conversion function T are adapted based on one penalty at a time.

[0091] In a use case embodiment, the optimized parametrized conversion function T may then be used for converting an original medical image of a first image type into a converted medical image of a second image type, which may e.g. be used for dose calculation in radiation treatment planning, where machine instructions e.g. in the form of a control signal may be sent to radiation treatment equipment based on an analysis of the converted medical image of the second image type.

Further embodiments



[0092] Where applicable, various embodiments provided by the present disclosure can be implemented using hardware, software, or combinations of hardware and software. Also where applicable, the various hardware components and/or software components set forth herein can be combined into composite components comprising software, hardware, and/or both without departing from the claimed scope of the present disclosure. Where applicable, the various hardware components and/or software components set forth herein can be separated into sub-components comprising software, hardware, or both without departing from the claimed scope of the present disclosure. In addition, where applicable, it is contemplated that software components can be implemented as hardware components, and vice-versa. The method steps of one or more embodiments described herein may be performed automatically, by any suitable processing unit, or one or more steps may be performed manually. Where applicable, the ordering of various steps described herein can be changed, combined into composite steps, and/or separated into sub-steps to provide features described herein.

[0093] Software in accordance with the present disclosure, such as program code and/or data, can be stored in non-transitory form on one or more machine-readable mediums. It is also contemplated that software identified herein can be implemented using one or more general purpose or specific purpose computers and/or computer systems, networked and/or otherwise.

[0094] In embodiments, there are provided a computer program product comprising computer readable code configured to, when executed in a processor, perform any or all of the method steps described herein. In some embodiments, there are provided a non-transitory computer readable memory on which is stored computer readable and computer executable code configured to, when executed in a processor, perform any or all of the method steps described herein.

[0095] In one or more embodiments, there is provided a non-transitory machine-readable medium on which is stored machine-readable code which, when executed by a processor, controls the processor to perform the method of any or all of the method embodiments presented herein.

[0096] The foregoing disclosure is not intended to limit the present invention to the precise forms or particular fields of use disclosed. It is contemplated that various alternate embodiments and/or modifications to the present invention, whether explicitly described or implied herein, are possible in light of the disclosure. Accordingly, the scope of the invention is defined only by the claims.


Claims

1. A system (100) for generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type, the system comprising at least one processing unit configured to:

obtain original medical images of the first and second image types;

obtain an initial parametrized conversion function G to convert original medical images of the first image type into converted medical images of the second image type;

calculate a first penalty P1 based on at least one comparison of a first original medical image of the second image type with a first converted medical image of the second image type, that has been generated by applying a first parametrized conversion function G1, which is based on the initial parametrized conversion function G, to a first original medical image of the first image type that forms an image pair with the first original medical image of the second image type and has thereby been determined to show the same part of the same patient;

calculate a second penalty P2 based on at least one comparison of an original medical image of the first image type and a converted medical image of the second image type that has been generated by applying a second parametrized conversion function G2, which is based on the initial parametrized conversion function G, to the original medical image of the first image type, after converting the original medical image of the first image type and/or the converted medical image of the second image type into images of the same image type; and

generate the optimized parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first and second penalties P1 and P2.


 
2. A system according to claim 1, wherein the at least one processing unit is further configured to:

obtain a first parametrized discriminator function D1 to discriminate between original medical images of the second image type and converted medical images of the second image type; and

calculate a third penalty P3 based on a classification error of the first parametrized discriminator function D1 when trying to discriminate between at least one original medical image of the second image type and at least one converted medical image of the second image type that has been generated by applying a third parametrized conversion function G3, which is based on the initial parametrized conversion function G, to at least one original medical image of the first image type;

wherein the at least one processing unit generates the parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first, second and third penalties P1, P2 and P3.
 
3. A system according to claim 1 or 2, wherein the at least one processing unit is further configured to:

obtain a second parametrized discriminator function D2 to discriminate between

a) image pairs with corresponding original medical images, an original medical image of the first image type and an original medical image of the second image type, which have been paired and thereby determined to show the same part of the same patient; and

b) image pairs of an original medical image of the first image type and a corresponding converted medical image of the second image type; and

calculate a fourth penalty P4 based on a classification error of the second parametrized discriminator function D2;

wherein the at least one processing unit generates the parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first, second and fourth penalties P1, P2 and P4.
 
4. A system according to any one of claims 1-3, wherein the at least one processing unit obtains the initial parametrized conversion function G by using a previously generated optimized parametrized conversion function T as the initial parametrized conversion function G, so that the parametrized conversion function T is iteratively optimized.
 
5. A processor-implemented method (600) of generating an optimized parametrized conversion function T for converting an original medical image of a first image type into a converted medical image of a second image type, the method comprising:

obtaining (610) original medical images of the first and second image types;

obtaining (620) an initial parametrized conversion function G to convert original medical images of the first image type into converted medical images of the second image type;

calculating (630) a first penalty P1 based on at least one comparison of a first original medical image of the second image type with a first converted medical image of the second image type, that has been generated by applying a first parametrized conversion function G1, which is based on the initial parametrized conversion function G, to a first original medical image of the first image type that forms an image pair with the first original medical image of the second image type and has thereby been determined to show the same part of the same patient;

calculating (640) a second penalty P2 based on at least one comparison of an original medical image of the first image type and a converted medical image of the second image type that has been generated by applying a second parametrized conversion function G2, which is based on the initial parametrized conversion function G, to the original medical image of the first image type, after converting the original medical image of the first image type and/or the converted medical image of the second image type into images of the same image type; and

generating (650) the optimized parametrized conversion function T based on the parameters of the initial parametrized conversion function G and at least said first and second penalties P1 and P2.


 
6. A method according to claim 5, further comprising:

obtaining (660) a first parametrized discriminator function D1 to discriminate between original medical images of the second image type and converted medical images of the second image type; and

calculating (670) a third penalty P3 based on a classification error of the first parametrized discriminator function D1 when trying to discriminate between at least one original medical image of the second image type and at least one converted medical image of the second image type that has been generated by applying a third parametrized conversion function G3, which is based on the initial parametrized conversion function G, to at least one original medical image of the first image type;

wherein the parametrized conversion function T is generated (640) based on the parameters of the initial parametrized conversion function G and at least said first, second and third penalties P1, P2 and P3.


 
7. A method according to claim 6, further comprising obtaining (660) the first parametrized discriminator function D1 based on the parameters of the parametrized discriminator function D1 used for generating said optimized parametrized conversion function T and at least said third penalty P3, so that the first parametrized discriminator function D1 is iteratively optimized.
 
8. A method according to any one of claims 5-7, further comprising:

obtaining (680) a second parametrized discriminator function D2 to discriminate between

a) image pairs with corresponding original medical images, an original medical image of the first image type and an original medical image of the second image type, which have been paired and thereby determined to show the same part of the same patient; and

b) image pairs of an original medical image of the first image type and a corresponding converted medical image of the second image type; and

calculating (690) a fourth penalty P4 based on a classification error of the second parametrized discriminator function D2;

wherein the parametrized conversion function T is generated (650) based on the parameters of the initial parametrized conversion function G and at least said first, second and fourth penalties P1, P2 and P4.
 
9. A method according to any one of claims 5-8, further comprising obtaining (620) the initial parametrized conversion function G by generating it using a machine learning algorithm such as a neural network, a random forest, or a support vector machine.
 
10. A method according to any one of claims 5-8, wherein the initial parametrized conversion function G is obtained (620) by a previously generated optimized parametrized conversion function T being used as the initial parametrized conversion function G, so that the parametrized conversion function T is iteratively optimized.
 
11. A method according to any one of claims 5-10, wherein the comparing of the first original medical image with the corresponding first converted medical image voxel wise compares the images, after first converting both images into a third image type such as e.g. a gradient image, a segmented image, or the output from a pre-trained machine learning algorithm such as a convolutional neural network, wherein the third image type amplifies certain features such as contours, regions-of-interest (ROI) and/or structures in the images so that they may more easily be compared.
 
12. A method according to any one of claims 5-11, wherein the first parametrized conversion function G1 and/or the second parametrized conversion function G2 are the same as the initial parametrized conversion function G.
 
13. A method according to any one of claims 5-12, wherein the image type is a medical image modality, such as e.g. MR, CT or CBCT.
 
14. The optimized parametrized conversion function T generated by the method of any one of claims 5-13.
 
15. Use of the optimized parametrized conversion function T generated by the method of any one of claims 5-13 for converting an original medical image of a first image type into a converted medical image of a second image type, which may e.g. be used for dose calculation in radiation treatment planning, where machine instructions e.g. in the form of a control signal may be sent to radiation treatment equipment based on an analysis of the converted medical image of the second image type.
 
16. A system according to claim 2, wherein the at least one processing unit obtains the first parametrized discriminator function D1 based on the parameters of the parametrized discriminator function D1 used for generating said optimized parametrized conversion function T and at least said third penalty P3, so that the first parametrized discriminator function D1 is iteratively optimized.
 
17. A system according to any one of claims 1-3 or 16, wherein the at least one processing unit obtains the initial parametrized conversion function G by generating it using a machine learning algorithm such as a neural network, a random forest, or a support vector machine.
 
18. A system according to any one of claims 1-4, 16 or 17, wherein the at least one processing unit compares the first original medical image with the corresponding first converted medical image by voxel wise comparing the images, after first converting both images into a third image type such as e.g. a gradient image, a segmented image, or the output from a pre-trained machine learning algorithm such as a convolutional neural network, wherein the third image type amplifies certain features such as contours, regions-of-interest (ROI) and/or structures in the images so that they may more easily be compared.
 
19. A system according to any one of claims 1-4 or 16-18, wherein the first parametrized conversion function G1 and/or the second parametrized conversion function G2 are the same as the initial parametrized conversion function G.
 
20. A system according to anyone of claims 1-4 or 16-19, wherein the image type is a medical image modality, such as e.g. MR, CT or CBCT.
 




Drawing

























Search report












Search report




Cited references

REFERENCES CITED IN THE DESCRIPTION



This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard.

Non-patent literature cited in the description