It is known that any $n$-variable function on a finite prime field of characteristic $p$ can be expressed as a polynomial over the same field with at most $p^n$ monomials. However, it is not obvious to determine the polynomial for a given concrete function. In this paper, we study the concrete polynomial expressions of the carries in addition and multiplication of $p$-ary integers. For the case of addition, our result gives a new family of symmetric polynomials, which generalizes the known result for the binary case $p = 2$ where the carries are given by elementary symmetric polynomials. On the other hand, for the case of multiplication of $n$ single-digit integers, we give a simple formula of the polynomial expression for the carry to the next digit using the Bernoulli numbers, and show that it has only $(n+1)(p-1)/2 + 1$ monomials, which is significantly fewer than the worst-case number $p^n$ of monomials for general functions. We also discuss applications of our results to cryptographic computation on encrypted data.