We are interested in designing collaborative robots that can seamlessly interact in complex domains - such as, collaborative manufacturing and disaster response - with human teammates. To be successful teammates, such robots need the ability to model, predict and adapt to their human collaborators. However, modeling the behavior of human teammates is challenging - since human decisions often depend on factors that are latent and difficult to specify. In this extended abstract, we describe this challenge for modeling humans for designing robot collaborators, summarize our algorithmic solutions towards this problem and conclude with a description of a human-robot collaboration scenario designed to evaluate our approach.