Efficient compilation of quantum algorithms is vital in the era of Noisy Intermediate-Scale Quantum (NISQ) devices. While multiple open-source quantum compilation and circuit optimization frameworks are available, e.g. IBM Qiskit, CQC Tket, Google Cirq, Rigetti Quilc, PyZX, their relative performance is not always clear to a quantum programmer. The growth of complexity and diversity of quantum circuit compilation algorithms creates a demand for a dedicated tool for cross-benchmarking and profiling of inner workflow of the quantum compilation stack. We present an open-source software package, Arline Benchmarks, that is designed to perform automated benchmarking of quantum compilers with the focus on NISQ applications. The name "Arline" was given in honour of Arline Greenbaum Feynman, the first wife of Richard Feynman, the pioneer of quantum computing. We compared several quantum compilation frameworks based on a set of important metrics such as post-optimization gate counts, circuit depth, hardware-dependent circuit cost function, compiler run time etc. with a detailed analysis of metrics for each compilation stage. We performed a variety of compiler tests for random circuits and structured quantum algorithms (VQE, Trotter decomposition, Grover search, Option Pricing via Amplitude Estimation) for several popular quantum hardware architectures. Leveraging cross-platform functionality of Arline, we propose a concept of composite compilation pipeline that combines compiler-specific circuit optimization subroutines in a single compilation stack and finds an optimized sequence of compilation passes. By providing detailed insights into the compilation flow of quantum compilers, Arline Benchmarks offers a valuable toolkit for quantum computing researchers and software developers to gain additional insights into compilers' characteristics.