A new method to construct task graphs for \mcH-matrix arithmetic is introduced, which uses the information associated with all tasks of the standard recursive \mcH-matrix algorithms, e.g., the block index set of the matrix blocks involved in the computation. Task refinement, i.e., the replacement of tasks by sub-computations, is then used to proceed in the \mcH-matrix hierarchy until the matrix blocks containing the actual matrix data are reached. This process is a natural extension of the classical, recursive way in which \mcH-matrix arithmetic is defined and thereby simplifies the efficient usage of many-core systems. Examples for standard and accumulator based \mcH-arithmetic are shown for model problems with different block structures.