A graph labeling assigns values to the components of a graph (vertices, edges, etc.). In particular, distance magic labelings have been widely studied in undirected graphs. In such a labeling, the vertices are labeled with unique values from one up to the number of vertices so that the sum of labels on the neighbors of any vertex is the same across all vertices. For oriented graphs, a related concept of distance difference magic has been studied. In a distance difference magic labeling, each vertex is given a unique value from one up to the number of vertices such that for each vertex the sums of the labels of vertices in the in-neighborhood minus the sums of the labels of vertices in the out-neighborhood equals zero. In this paper, we expand on this concept by showing a connected difference distance magic oriented graph on $n$ vertices exists for each integer $n \geq 5$. We also construct arbitrarily large difference distance magic oriented graphs from smaller ones using a new graph sum and exhibit a connection between linear algebra and this type of labeling.