We provide a method to construct $t$-designs from weighing matrices and association schemes. One instance of our method can produce a $3$-design from any (symmetric or skew-symmetric) conference matrix, thereby providing a partial answer to a question of Gunderson and Semeraro JCTB 2017. We explore variations of our method on some matrices that satisfy certain combinatorial restrictions. In particular, we show that there exist various infinite families of partially balanced incomplete block designs with block size four on the binary Hamming schemes and the $3$-class association schemes attached to symmetric designs, and regular pairwise balanced designs with block sizes three and four.