Humans and robots can recognize materials with distinct thermal effusivities by making physical contact and observing temperatures during heat transfer. This works well with room temperature materials and humans and robots at human body temperatures. Past research has shown that cooling or heating a material can result in temperatures that are similar to contact with another material. To thoroughly investigate this perceptual ambiguity, we designed a psychophysical experiment in which a participant discriminates between two materials given ambiguous initial conditions. We conducted a study with 32 human participants and a robot. Humans and the robot confused the materials. We also found that robots can overcome this ambiguity using two temperature sensors with different temperatures prior to contact. We support this conclusion based on a mathematical proof using a heat transfer model and empirical results in which a robot achieved 100% accuracy compared to 5% human accuracy. Our results also indicate that robots can use subtle cues to distinguish thermally ambiguous materials with a single temperature sensor. Overall, our work provides insights into challenging conditions for material recognition via heat transfer, and suggests methods by which robots can overcome these challenges to outperform humans.