Learning difficulties pose significant challenges for students, impacting their academic performance and overall educational experience. These difficulties could sometimes put students into a downward spiral that lack of educational resources for personalized support consistently led to under-accommodation of students special needs, and the student lose opportunities in the longer term academic and work development. This research aims to propose a conceptual framework for an adaptive AI-based virtual tutor system that incorporates psychometric assessment to support students with learning difficulties. This process involves the careful selection and integration of validated current mature psychometric scales that assess key dimensions of learning, such as cognitive abilities, learning styles, and academic skills. By incorporating scales that specifically assess these difficulties, the psychometric test will provide a comprehensive understanding of each students unique learning profile and inform targeted interventions within the adaptive tutoring system. The paper also proposes using autoencoders to identify the latent patterns to generate the students profile vector for collection of psychometric data, defining state space and action space representing the students desired combination of images, sound and text engagements, employing extended Bayesian knowledge tracing and hierarchical model and Metropolis-Hastings to continuously estimate and monitor the students performance in various psychometric constructs. The proposed system will leverage the capabilities of LLMs, visual generation models, and psychometric assessments to provide personalized instruction and support tailored to each students unique learning characteristics and needs.