Existing side-channel analysis techniques require a leakage model, in the form of a prior knowledge or a set of training data, to establish a relationship between the secret data and the measurements. We introduce side-channel analysis with reinforcement learning (SCARL) capable of extracting data-dependent features of the measurements in an unsupervised learning approach without requiring a prior knowledge on the leakage model. SCARL consists of an auto-encoder to encode the information of power measurements into an internal representation, and a reinforcement learning algorithm to extract information about the secret data. We employ a reinforcement learning algorithm with actor-critic networks, to identify the proper leakage model that results in maximum inter-cluster separation of the auto-encoder representation. SCARL assumes that the lower order components of a generic non-linear leakage model have larger contribution to the leakage of sensitive data. On a lightweight implementation of the Ascon authenticated cipher on the Artix-7 FPGA, SCARL is able to recover the secret key using 24K power traces during the key insertion, or Initialization Stage, of the cipher. We also demonstrate that classical techniques such as DPA and CPA fail to identify the correct key using traditional linear leakage models and more than 40K power traces.