Improved prediction of progression to Alzheimer's Disease (AD) among older individuals with mild cognitive impairment (MCI) is of high clinical and societal importance. We recently developed a polygenic hazard score (PHS) that predicted age of AD onset above and beyond APOE. Here, we used data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) to further explore the potential clinical utility of PHS for predicting AD development in older adults with MCI. We examined the predictive value of PHS alone and in combination with baseline structural magnetic resonance imaging (MRI) data on performance on the Mini-Mental State Exam (MMSE). In survival analyses, PHS significantly predicted time to progression from MCI to AD over 120 months (p = 1.07e-5), and PHS was significantly more predictive than APOE alone (p = 0.015). Combining PHS with baseline brain atrophy score and/or MMSE score significantly improved prediction compared to models without PHS (three-factor model p = 4.28e-17). Prediction model accuracies, sensitivities and area under the curve were also improved by including PHS in the model, compared to only using atrophy score and MMSE. Further, using linear mixed-effect modeling, PHS improved the prediction of change in the Clinical Dementia Rating—Sum of Boxes (CDR-SB) score and MMSE over 36 months in patients with MCI at baseline, beyond both APOE and baseline levels of brain atrophy. These results illustrate the potential clinical utility of PHS for assessment of risk for AD progression among individuals with MCI both alone, or in conjunction with clinical measures of prodromal disease including measures of cognitive function and regional brain atrophy.