Combinatorial $t$-designs have nice applications in coding theory, finite geometries and several engineering areas. There are two major methods of constructing $t$-designs. One of them is via group actions of certain permutation groups which are $t$-transitive or $t$-homogeneous on some point set. The other is a coding-theoretical one. The objectives of this paper are to introduce two constructions of $t$-designs with special polynomials over finite fields GF$(q)$, and obtain $2$-designs and $3$-designs with interesting parameters. A type of d-polynomials is defined and used to construct $2$-designs. Under the framework of the first construction, it is shown that every o-polynomial over GF$(2^m)$ gives a $2$-design, and every o-monomial over GF$(2^m)$ yields a $3$-design. Under the second construction, every $o$-polynomial gives a $3$-design. Some open problems and conjectures are also presented in this paper.