Live cells in tissue are plastic, phenotypically dynamic, and modify their function in response to genetic and environmental perturbations. To unleash the power of live-cell imaging to identify phenotype-genotype-function coupling over time, we report the development of a standardized Shape-Appearance-Motion (SAM) "phenome" and SAM-Phenotype-Observation-Tool (SPOT), that act as an image-"transcriptome" and image-"transcriptome analyzer" respectively, and provide an unbiased and comprehensive description of morpho-dynamic phenotypes without prior knowledge. We apply SAM-SPOT to our simulated organoids database with known ground-truth and >1.6 million mouse and human organoid instances with defined genetic and chemical perturbations. SAM-SPOT can effectively and robustly characterize 3D morpho-dynamics from 2D projection videos. Combined with single-cell RNA sequencing, SAM-SPOT reveals that altered WNT signaling, but not mutant RAS or p53, predisposes intestinal organoids to irregular morphogenesis. SAM-SPOT advances biomedical discovery by empowering live-cell imaging to identify phenotype-genotype-function relationships through large-scale and cost-effective label-free live-cell imaging.