Realistic cross-domain microscopy via conditional generative adversarial networks: Converting immunofluorescence to Hematoxylin and Eosin

Hematoxylin and Eosin (H&E) is a widely-used stain for diagnosis and prognosis in clinical pathology; it is a non-specific stain, binding to all cell types. Immunofluorescent (IF) staining is highly specific, binding only to targeted proteins in a sample to identify specific cellular and sub-cellular structures. IF images are costlier and more technically difficult compared with H&E, so are rarely used in routine clinical workup, but can be used for identification of diagnostically significant cell types. In previous work, we used registered IF and H&E images to generate class labels for training a deep learning H&E segmentation algorithm. In this work, we leverage this dataset to train a Conditional Generative Adversarial Network (cGAN) to generate realistic-looking H&E images from IF images stained for DAPI and ribosomal S6. Using these generated images, we trained a semantic segmentation algorithm to identify nuclei, cytoplasm, and membrane classes by thresholding the original IF stains for use as class labels on the generated H&E. The trained classifier was then used to segment a holdout dataset of real H&E images. We found that the semantic segmentation models trained on the generated H&E images (Dice score: 0.539) performed similarly to models trained on real H&E (Dice score: 0.503), suggesting that cGAN generated samples can be used as a viable training set for deep learning models that are intended to be applied on real H&E data.