Judicial impartiality is a cornerstone of well-functioning legal systems. We assemble a dataset of 112,312 civil lawsuits in U.S. District Courts to study the effect of extraneous factors on judicial decision making. We show that cases are randomly assigned to judges and that biographical judge features are predictive of judicial decisions. We use low-dimensional representations of judges' early-career citation records as generic representations of judicial idiosyncrasies. These predict future judgments with accuracies exceeding 65% for high-confidence predictions on balanced out-of-sample test cases. For 6-8% of judges, these representations are significant predictors across all judgments. These findings indicate that a small but significant group of judges routinely relies on extraneous factors and careful vetting of judges prior to appointment may partially address this issue. Our use of low-dimensional representations of citation records may also be generalized to other jurisdictions or to study other aspects of judicial decision making.