Graph algorithms play an important role in many computer science areas. In order to solve problems that can be modeled using graphs, it is necessary to use a data structure that can represent those graphs in an efficient manner. On top of this, an infrastructure should be build that will assist in implementing common algorithms or developing specialized ones. Here, a new Java library is introduced, called Graph4J, that uses a different approach when compared to existing, well-known Java libraries such as JGraphT, JUNG and Guava Graph. Instead of using object-oriented data structures for graph representation, a lower-level model based on arrays of primitive values is utilized, that drastically reduces the required memory and the running times of the algorithm implementations. The design of the library, the space complexity of the graph structures and the time complexity of the most common graph operations are presented in detail, along with an experimental study that evaluates its performance, when compared to the other libraries. Emphasis is given to infrastructure related aspects, that is graph creation, inspection, alteration and traversal. The improvements obtained for other implemented algorithms are also analyzed and it is shown that the proposed library significantly outperforms the existing ones.