Effective treatments for pancreatic ductal adenocarcinoma (PDAC) are lacking, and targeted agents have demonstrated limited efficacy. It has been speculated that a rare population of cancer stem cells (CSCs) drives growth, therapy resistance, and rapid metastatic progression in PDAC. These CSCs demonstrate high clonogenicity in vitro and tumorigenic potential in vivo. However, their relevance in established PDAC tissue has not been determined. Here, we use marker-independent stochastic clonal labeling, combined with quantitative modeling of tumor expansion, to uncover PDAC tissue growth dynamics. We find that in contrast to the CSC model, all PDAC cells display clonogenic potential in situ. Furthermore, the proximity to activated cancer-associated fibroblasts determines tumor cell clonogenicity. This means that the microenvironment is dominant in defining the clonogenic activity of PDAC cells. Indeed, manipulating the stroma by Hedgehog pathway inhibition alters the tumor growth mode, revealing that tumor-stroma crosstalk shapes tumor growth dynamics and clonal architecture.