In this paper, we present Rapid Activity Prediction Through Object-oriented Regression (RAPTOR), a scalable method for performing rapid, real-time activity recognition and prediction that achieves state-of-the-art classification accuracy on both a generic human activity dataset and two domainspecific collaborative robotics manufacturing datasets. Our approach is designed to be human-interpretable: able to provide explanations for its reasoning such that non-experts can better understand and improve its activity models. We incorporate methods to increase RAPTOR’s resilience against confusion due to temporal variations, as well as against learning false correlations between features. We report full and partial trajectory classification results across three datasets and conclude by demonstrating our model’s ability to provide interpretable explanations of its reasoning using outlier detection techniques.