We study the palindromic length of factors of infinite words fixed by morphisms of the so-called class $\mathcal{P}$ introduced by Hof, Knill and Simon. We show that it grows at most logarithmically with the length of the factor. For the Fibonacci word and the Thue-Morse word we provide estimates on the constants of the growth. We also construct an infinite word rich in palindromes for which the palindromic length grows as $\sqrt{n}$.