In this paper, we generate a transmit power pool (PP) for Internet of things (IoT) networks with semi-grant-free non-orthogonal multiple access (SGF-NOMA) via multi-agent deep reinforcement learning (MA-DRL) to enable open loop power control (PC). The PP is mapped with each resource block (RB) to achieve distributed power control (DPC). We first formulate the resource allocation problem as stochastic Markov game, and then solve it using two MA-DRL algorithms, namely double deep Q network (DDQN) and Dueling DDQN. Each GF user as an agent tries to find out the optimal transmit power level and RB to form the desired PP. With the aid of dueling processes, the learning process can be enhanced by evaluating the valuable state without considering the effect of each action at each state. Therefore, DDQN is designed for communication scenarios with a small-size action-state space, while Dueling DDQN is for a large-size case. Moreover, to decrease the training time, we reduce the action space by eliminating invalid actions. To control the interference and guarantee the quality-of-service requirements of grant-based users, we determine the optimal number of GF users for each sub-channel. We show that the PC approach has a strong impact on data rates of both grant-based and GF users. We demonstrate that the proposed algorithm is computationally scalable to large-scale IoT networks and produce minimal signalling overhead. Our results show that the proposed MA-Dueling DDQN based SGF-NOMA with DPC outperforms the existing SGF-NOMA system and networks with pure GF protocols with 17.5\% and 22.2\% gain in terms of the system throughput, respectively. Finally, we show that our proposed algorithm outperforms the conventional open loop PC mechanism.