The footprint left by early cancer dynamics on the spatial arrangement of tumour cells is poorly understood, and yet could encode information about how therapy resistant sub-clones grew within the expanding tumour. Novel methods of quantifying spatial tumour data at the cellular scale are required to link evolutionary dynamics to the resulting spatial architecture of the tumour. Here, we propose a framework using first passage times of random walks to quantify the complex spatial patterns of tumour cell population mixing. First, using a toy model of cell mixing we demonstrate how first passage time statistics can distinguish between different pattern structures. We then apply our method to simulated patterns of wild-type and mutated tumour cell population mixing, generated using an agent-based model of expanding tumours, to explore how first passage times reflect mutant cell replicative advantage, time of emergence and strength of cell pushing. Finally, we analyse experimentally measured patterns of genetic point mutations in human colorectal cancer, and estimate parameters of early sub-clonal dynamics using our spatial computational model. We uncover a wide range of mutant cell replicative advantages and timings, with the majority of sampled tumours consistent with boundary driven growth or short-range cell pushing. By analysing multiple sub-sampled regions in a small number of samples, we explore how the distribution of inferred dynamics could inform about the initial mutational event. Our results demonstrate the efficacy of first passage time analysis as a new methodology for quantifying cell mixing patterns in vivo, and suggest that patterns of sub-clonal mixing can provide insights into early cancer dynamics.