Four new Canadian research projects join the UK-led international AI Alignment Project where they will receive up to $165K per year

Four new research projects have been funded under the Canadian AI Safety Institute (CAISI) Research Program at CIFAR. As part of the AI Alignment Project, an international funding coalition led by the UK AI Security Institute, this initiative supports groundbreaking work in the field of AI alignment, ensuring advanced AI systems remain safe, secure and beneficial to society. 

The selected projects, leveraging fields like game theory, statistics and physics, will receive $165,000 for one year (with the possibility of an extension for a second year), alongside specialized compute resources and expert support to bridge the gap between AI’s rapid development and the safety frameworks needed to govern it. 

The addition of these new initiatives brings the total number of research projects at the CAISI Research Program at CIFAR up to sixteen, joining existing Catalyst Projects and Solution Networks in their efforts to drive high-impact AI safety research and implement practical solutions to the benefit of all. The impact of these projects and the overall program was recently highlighted in our 2025 Year in Review: Building Safe AI for Canadians report. 

“The AI Alignment Project is an important step in furthering Canada’s long history of leadership in driving AI research that is safe and trustworthy. As AI becomes increasingly present in our lives, it is more important than ever to ensure it is aligned with our values and serves the public good. By investing in the work of these Canadian researchers, we are building long-term economic resilience while cementing Canada’s position as a global leader in responsible AI development and deployment.”

— The Honourable Evan Solomon, Minister of Artificial Intelligence and Digital Innovation and Minister Responsible for the Federal Economic Development Agency for Southern Ontario

“The AI Alignment Project represents an opportunity to build on Canada’s strong ties with our international partners and work towards a common goal of furthering AI safety research. The selected projects cover a variety of key alignment issues that deserve our immediate attention and will contribute to ensuring AI systems are safe, trustworthy and interpretable, benefiting Canadians and beyond.” 

— Catherine Régis and Nicolas Papernot, co-directors of the CAISI Research Program at CIFAR.

Game-theoretic safety guarantees for advanced AI systems

Zhijing Jin (Canada CIFAR AI Chair, Vector Institute, University of Toronto)

As information systems become increasingly AI-centric and autonomous, traditional security frameworks no longer adequately address questions of safety, control and privacy, especially in situations where multiple AI agents collaborate autonomously. Canada CIFAR AI Chair Zhijing Jin proposes using game theory, a robust theoretical framework, to provide provable guarantees to mitigate misaligned behaviours and offer concrete tools for policymakers and AI developers to maintain control in multi-agent scenarios.

Sample-efficient online fine-tuning against resistant behaviors: statistical foundations for post-training alignment

Linglong Kong (Canada CIFAR AI Chair, Amii, University of Alberta)

Modern AI systems deployed in the real world often develop emergent misalignment (e.g., reward hacking, deceptive alignment) after deployment, an internal behavioural failure that causes them to deviate from their intended goals. Canada CIFAR AI Chair Linglong Kong proposes a statistical framework for sample-efficient online fine-tuning to establish whether corrective training can serve as a trustworthy safety mechanism or whether more fundamental safeguards are required.

Scaling laws, data distributions, and learning dynamics: simulated high-energy physics data as a benchmark for data in the wild

Yonatan Kahn (University of Toronto)

Current theoretical AI research often relies on overly simple data modeling, making it difficult to answer fundamental questions about scaling laws or whether models truly learn underlying latent parameters. University of Toronto Professor Yonatan Kahn proposes a method of physics-based data generation that will provide ‘ground-truth information’ to researchers, allowing them to predict and test some of the fundamental questions around how AI learns, generalizes and scales.

A unified statistical framework for quantifying rare event risks for language models

Bei Jiang (Canada CIFAR AI Chair, Amii, University of Alberta)

One of the central challenges of AI alignment is quantifying extremely small failure rates – probabilities so low that ordinary testing will never observe them. ‘Long-tail failures,’ like jailbreaks, policy evasion or subtle safety violations, are where oversight is weakest, making it impossible to compare models, set safety standards or measure model regression. Canada CIFAR AI Chair Bei Jiang will address this issue using standard statistical tools designed for these rare-event problems, which are currently underused in large language model evaluation.