Conventional automated decision-support systems often prioritize predictive accuracy, overlooking the complexities of real-world settings where stakeholders' preferences may diverge or conflict. This can lead to outcomes that disadvantage vulnerable groups and erode trust in algorithmic processes. Participatory AI approaches aim to address these issues but remain largely context-specific, limiting their broader applicability and scalability. To address these gaps, we propose a participatory framework that reframes decision-making as a multi-stakeholder learning and optimization problem. Our modular, model-agnostic approach builds on the standard machine learning training pipeline to fine-tune user-provided prediction models and evaluate decision strategies, including compromise functions that mediate stakeholder trade-offs. A synthetic scoring mechanism aggregates user-defined preferences across multiple metrics, ranking strategies and selecting an optimal decision-maker to generate actionable recommendations that jointly optimize performance, fairness, and domain-specific goals. Empirical validation on two high-stakes case studies demonstrates the versatility of the framework and its promise as a more accountable, context-aware alternative to prediction-centric pipelines for socially impactful deployments.