Health AI must be designed with equity at its core
AI could transform African health systems, but without careful attention it may also amplify existing inequities. Equity must be built into how health AI is designed, validated, governed, and implemented.
An algorithm can assist health workers in systems with strong digital infrastructure, regulation, and workforce capacity. Photo: Soweto Graphics
AI is rapidly changing what is technically possible in health. Machine-learning models can detect patterns across complex datasets, support clinical and population-level decision making, improve diagnostic processes, and potentially extend scarce expertise to populations that health systems struggle to reach. Generative AI adds another dimension: the ability to synthesise and communicate health information at a scale we have not previously been able to manage.
For African health systems, these capabilities create significant opportunities. Yet my work across health research, implementation science, and data-driven decision making has made me equally interested in a more fundamental question: what happens when AI is introduced into health systems in which the underlying data, infrastructure, and distribution of power are already unequal?
This matters because AI does not eliminate the limitations of the systems from which it learns. It can reproduce them and, because algorithmic decisions can operate at enormous scale, potentially amplify them.
Inequity can begin with data
One of the most important lessons from my engagement with data science has been that algorithmic bias often begins before model development.
I encountered this through a health-financing initiative in East Africa that explored machine learning to estimate households' financial capacity. The policy objective was inherently equitable: to differentiate financial contributions while protecting households least able to pay.
But interrogation of the underlying data revealed a critical problem. Some of the most economically vulnerable populations were inadequately represented. A model trained on those data could therefore systematically overestimate the financial capacity of precisely the households the system was intended to protect.
This is an important distinction. A model may demonstrate good aggregate predictive performance and still distribute error inequitably across population subgroups. Overall accuracy therefore tells us relatively little about equity unless we also interrogate who is represented in the training data, where errors occur, and who bears the consequences of those errors.
The experience reinforced something increasingly recognised in AI governance: computational sophistication cannot compensate for information that is absent from the training data. The World Health Organization (WHO) has similarly cautioned that datasets that exclude rural communities and other disadvantaged groups can reproduce those exclusions in AI systems. For health systems, that is an equity problem, not merely a technical one.
Problem extends beyond algorithmic bias
Representation is only the first layer. Much of the discussion about responsible AI focuses on bias within algorithms. I am equally concerned about what surrounds them: data quality and provenance, the adequacy of validation datasets, weak governance structures, insufficient guardrails, and unclear accountability when algorithm-supported decisions cause harm.
Who decides whether a dataset is representative enough for a given use case? Are errors broken down by sex, geography, socioeconomic status and language, or only reported in aggregate? Somebody has to set acceptable thresholds for false positives and false negatives, and somebody has to watch for drift once a model is deployed. And when an algorithm shapes a consequential health or health-financing decision, accountability has to sit somewhere concrete.
These are not problems that data scientists can solve alone. They require governance. WHO's framework for AI in health similarly emphasises human autonomy, transparency, explainability, responsibility, accountability, and inclusiveness, including mechanisms through which people adversely affected by algorithmic decisions can question those decisions and seek redress.
These issues acquire another dimension in global health because AI is being introduced into a field already marked by significant asymmetries of technical, financial, and institutional power. African populations should not become primarily sources of data for technologies designed and owned elsewhere, while African health systems become downstream deployment environments with limited influence over how those technologies are designed, validated, and governed.
Data sovereignty is also about power
For this reason, we need a more ambitious understanding of data sovereignty. It cannot be reduced to the physical location of a server. It includes the authority to determine what data are collected, the purposes for which they can be used and reused, who has access to them, what health problems AI is prioritised to solve, and how the resulting scientific, social, and economic value is distributed.
This is why localisation cannot be an afterthought. AI systems developed predominantly using data, disease profiles, languages, and health-system assumptions from high-income settings cannot simply be presumed to generalise to African populations. External validation is important, but equitable AI requires more than validating imported models. It requires African researchers, health workers, communities, and policymakers to participate much earlier in defining the problem and shaping the technology.
Generative AI brings equity question closer to patients
These issues become particularly important as generative AI moves from analytical applications into direct interaction with patients and communities.
Through my work in adolescent and women's health, I see enormous potential for AI-enabled digital tools to expand access to sexual, reproductive, maternal, and mental health information, particularly for young women who may face stigma, distance, cost, or confidentiality barriers when seeking care.
But generative AI changes the nature of risk. If a young woman seeks information about contraception, pregnancy complications, HIV, or mental health, the quality of the output depends on the evidence represented in the model, its ability to interpret her context and language, and the safeguards governing what it communicates. Incorrect, biased, or culturally inappropriate information does more than produce an inaccurate output; it can shape a decision with real consequences for her health.
WHO's guidance on generative AI for health therefore stresses the need for well-defined tasks, appropriate evidence, stakeholder involvement, and post-deployment auditing of large multimodal models. Human-centred AI means ensuring that technology augments rather than displaces human judgement, while protecting privacy, autonomy, empathy, and accountability.
Implementation science must accompany data science
There is one further dimension of equity: who gets to benefit from AI? Health systems with interoperable digital records, reliable connectivity, computing infrastructure, strong regulatory institutions, and adequately resourced workforces will be better positioned to adopt AI than those without them. Without deliberate investment, AI could therefore widen the very disparities we hope it will close.
An algorithm can support an overstretched health worker, but it cannot repair a dysfunctional referral pathway. A diagnostic model is of limited use if the patient cannot get the test or treatment it recommends. And no matter how sophisticated a prediction tool is, it cannot compensate for fragmented records or a work environment where staff cannot fit it into routine care.
This is where implementation science has to accompany data science. Asking whether a model works is not enough. For whom does it work, and under what conditions? Can it be integrated into clinical and health-system workflows, and can staff act on what it recommends? And once it is deployed, who is watching its performance, and can it be governed and sustained at scale?
Equity must be part of the architecture
AI presents an extraordinary opportunity for African health systems. We should not approach that opportunity defensively. But neither should equity become an ethical assessment conducted after the technology has already been designed.
Equity has to be built into every stage of the AI lifecycle: how problems are defined, how data are structured, how models are developed and validated, how they are implemented and monitored, and how accountability and redress are handled when something goes wrong.
African communities cannot simply be sources of data, and African health systems cannot simply be deployment sites. African scientists have a role earlier than that: not just validating a model built somewhere else, but helping define the problem it is meant to solve.
For me, every health AI system needs to be able to answer three questions: who is represented in it, who holds the decision-making power, and who is accountable when it fails.
Get that right from the start, and AI could help us build more equitable health systems. Get it wrong, and we will have used remarkably powerful technology to lock our existing inequities into the infrastructure of tomorrow's healthcare.
The opinions expressed are those of the author and do not necessarily reflect the position of Re:solve Global Health.
Dr Nyawira Gitahi is a clinical researcher at the Kenya Medical Research Institute (KEMRI) and former Africa lead for DataDelta at IDinsight. She is faculty at KEMRI and holds an adjunct professorship at the University of Toronto. Nyawira currently heads the Africa-wide Leadership for Innovation and Excellence in Accelerating Research on Women’s Health (LEA-WH) fellowship implemented with the National Academy of Medicine. Her work spans implementation science, adolescent and women's health, health systems, data-driven decision making, and equitable approaches to AI and innovation in African health systems.
