Strongly Koszul algebras were introduced by Herzog, Hibi and Restuccia in 2000. The goal of the present paper is to provide an in-depth study of such algebras and to investigate how strong Koszulness interacts with the existence of a quadratic Gröbner basis for the defining ideal. Firstly, we prove that the existence of a quadratic revlex-universal Gröbner basis with a strong sparsity condition (that we name "tidiness") is a sufficient condition for strong Koszulness, and exhibit several concrete examples arising from determinantal objects and Macaulay's inverse system. We then prove that there exist standard graded algebras that are strongly Koszul but do not admit a Gröbner basis of quadrics even after a linear change of coordinates, thus answering negatively a question posed by Conca, De Negri and Rossi. As a bonus, we prove that strong Koszulness behaves well under tensor and fiber products of algebras and illustrate how Severi varieties and Macaulay's inverse system interact to produce examples of strongly Koszul algebras with a geometric flavor.