This paper is devoted to the convergence analysis of stochastic approximation algorithms of the form $θ\_{n+1} = θ\_n + γ\_{n+1} H\_{θ\_n}(X\_{n+1})$ where $\{θ\_nn, n \geq 0\}$ is a $R^d$-valued sequence, $\{γ, n \geq 0\}$ is a deterministic step-size sequence and $\{X\_n, n \geq 0\}$ is a controlled Markov chain. We study the convergence under weak assumptions on smoothness-in-$θ$ of the function $θ\mapsto H\_θ(x)$. It is usually assumed that this function is continuous for any $x$; in this work, we relax this condition. Our results are illustrated by considering stochastic approximation algorithms for (adaptive) quantile estimation and a penalized version of the vector quantization.