Value at Risk (VaR) and stress testing are two of the most widely used approaches in portfolio risk management to estimate potential market value losses under adverse market moves. VaR quantifies potential loss in value over a specified horizon (such as one day or ten days) at a desired confidence level (such as 95'th percentile). In scenario design and stress testing, the goal is to construct extreme market scenarios such as those involving severe recession or a specific event of concern (such as a rapid increase in rates or a geopolitical event), and quantify potential impact of such scenarios on the portfolio. The goal of this paper is to propose an approach for incorporating prevailing market conditions in stress scenario design and estimation of VaR so that they provide more accurate and realistic insights about portfolio risk over the near term. The proposed approach is based on historical data where historical observations of market changes are given more weight if a certain period in history is "more similar" to the prevailing market conditions. Clusters of market conditions are identified using a Machine Learning approach called Variational Inference (VI) where for each cluster future changes in portfolio value are similar. VI based algorithm uses optimization techniques to obtain analytical approximations of the posterior probability density of cluster assignments (market regimes) and probabilities of different outcomes for changes in portfolio value. Covid related volatile period around the year 2020 is used to illustrate the performance of the proposed approach and in particular show how VaR and stress scenarios adapt quickly to changing market conditions. Another advantage of the proposed approach is that classification of market conditions into clusters can provide useful insights about portfolio performance under different market conditions.