Authors: Magdalena Gad-Nowak (senior associate at Timelex), Eleni Moraiti (associate at Timelex), Paraskevi Theofanous (associate at Timelex)
Artificial intelligence (AI) is increasingly becoming part of the European health research landscape. Across Horizon Europe and other research and innovation initiatives, AI-based tools are being explored for disease prediction, early detection, personalised medicine, clinical decision support, and healthcare optimisation.
At the same time, the growing use of AI in sensitive health-related contexts also raises important legal, ethical, and governance questions, particularly where projects involve the use of health data, including genetic information and biomarker-related data.
Importantly, many of these issues do not arise only once an AI system reaches the market or enters clinical practice. Indeed, key governance decisions are often made much earlier during the research and development phase. Choices relating to data quality, representativeness, system design, validation, oversight, or documentation may significantly influence whether an AI system can later be considered trustworthy, safe, and compliant with applicable regulatory requirements.
For this reason, health-AI projects should not wait until later stages of development to think about legal and ethical compliance. Even in research settings, certain governance priorities should be embedded into the project from day one. In practice, three areas are of particular importance to consider:
- data governance and bias prevention,
- meaningful human oversight and explainability, and
- documentation and accountability.
DISARM, a Horizon Europe project focused on the early detection and risk assessment of ovarian cancer, reflects this forward-looking approach. Although still at an early stage, DISARM has already completed several preparatory governance steps and the mapping of EU Artificial Intelligence Act (EU AI Act) requirements.
This blog post highlights three legal priorities that are central to responsible health-AI development and illustrates why they matter for projects such as DISARM. By addressing these priorities early, research initiatives can strengthen trust, reduce future regulatory barriers, and support the long-term uptake of AI-enabled solutions in healthcare.
Priority 1: Data governance and bias prevention
In health-AI projects, data governance is not simply a compliance exercise. It directly affects whether an AI system will ultimately be reliable, clinically useful, and trustworthy.
Data quality and representativeness matter
AI systems in healthcare depend heavily on the quality and representativeness of the data used to train, test, and validate them. This is particularly important in projects involving cancer prediction, early detection, clinical decision support, or personalised medicine. In such settings, weaknesses in the underlying data may lead to inaccurate outputs, misleading risk assessments, or unequal performance across different patient populations.
Research datasets are rarely perfect. Health data may originate from different hospitals, biobanks, laboratories, registries or previous research initiatives, often collected under different standards, methodologies, or technical conditions. Datasets may contain inconsistencies, missing values, demographic imbalances, or population gaps that affect how the AI system performs.
For example, if certain age groups, ethnic backgrounds, genetic profiles, or socio-economic groups are underrepresented in training data, the resulting AI system may perform less accurately for those groups. In healthcare, this is not merely a statistical issue: it may become a patient safety, discrimination and, increasingly, a regulatory issue.
Bias can emerge in different ways
Bias in health-AI is also not always intentional or obvious. It may emerge from historical inequalities in healthcare, differences in access to healthcare services, inconsistent diagnostic practices between institutions, or assumptions made during data selection and preparation. This may also require considering the specific clinical, demographic, or geographical contexts in which the system is intended to operate, as well as how sensitive personal data is processed, protected, and safeguarded, particularly where the use of such data may be necessary to identify or mitigate discriminatory outcomes. Even technically sophisticated models may reproduce or amplify these underlying imbalances if they are not identified and addressed during development.
This is one of the reasons why the EU AI Act places significant emphasis on data governance obligations for high-risk AI systems, including many healthcare-related applications. The regulation expects organisations to examine where datasets originate from, how they were prepared, what assumptions underlie them, whether important limitations or gaps exist, and whether the data is sufficiently relevant and representative for the intended purpose of the system.
In practice, it is important to not only look at the quantity of available data, but also at how the data was collected, labelled, filtered, harmonised, and prepared for training purposes. Governance risks arise long before the model is trained.
GDPR remains central
At the same time, projects operating in Europe must continue to navigate GDPR requirements when processing health data, genetic data, and/or biomarker-related data. GDPR compliance in health-AI projects goes well beyond obtaining consent forms or drafting privacy notices. It also requires careful consideration of issues such as lawful basis for processing, transparency obligations, data minimisation, purpose limitation, retention periods, security measures, and governance responsibilities between project partners.
Lawful processing in health-AI research therefore involves considerably more than collecting signatures on informed consent forms. It requires a structured assessment of governance responsibilities, access controls, safeguards for sensitive data, and the conditions under which data may be shared, reused, or further analysed throughout the project’s lifecycle.
This may become particularly complex in collaborative Horizon Europe projects, where multiple actors are involved in determining how data is collected, shared, analysed, and reused across different work packages, research activities, and technical infrastructures.
Questions around controllership, access rights, secondary use of data, and future reuse potential should therefore be addressed early on, before datasets, governance structures, and technical systems become difficult to modify.
Data governance is an ongoing process
Another important point to note is that data governance is not static. AI systems naturally evolve during research and development: datasets may expand, new project partners may join, functionalities may change, and additional data sources may be integrated over time. Governance mechanisms therefore need to be continuously revisited throughout the project lifecycle. A one-off legal or ethical assessment at the beginning of a project is never sufficient.
This does not mean that research datasets must be flawless before innovation can proceed. Research environments are inherently iterative; datasets evolve over time, and scientific uncertainty is unavoidable during early-stage development. However, projects should still be able to demonstrate that data-related risks are actively considered, monitored, and mitigated in a structured and responsible manner.
Priority 2: Human oversight and explainability
Human oversight as a core requirement under the EU AI Act
AI systems in healthcare should support – not replace – clinical judgement. This principle is reflected in the EU AI Act, particularly for high-risk AI systems used in medical and health-related contexts. Under Article 14 of the EU AI Act, human oversight is a core requirement for high-risk AI systems. The regulatory expectation is not merely that a clinician, researcher, or healthcare professional remains “in the loop” or is casually aware of the requirements, but that human oversight is both meaningful and effective in practice. It serves as a preventive safeguard aimed at minimizing the risks to health, safety and fundamental rights. In practical terms, AI-generated risk scores, classifications, or recommendations should be understood as decision-support outputs requiring professional review rather than autonomous clinical determinations. The EU AI Act explicitly states that human oversight shall correspond to the risks, level of autonomy, and context of use of the high-risk AI system.
Meaningful human oversight in healthcare contexts
In healthcare settings, AI outputs may directly influence diagnostic pathways, risk communication, treatment prioritisation, follow-up decisions, or broader professional assessments. Under the EU AI Act, high-risk AI systems must therefore be designed in a way that enables natural persons to properly oversee their operation, interpret outputs, identify anomalies, and intervene where necessary. Human oversight cannot function as a purely formal safeguard if users are unable to understand the system’s limitations, challenge its recommendations, or recognise situations where outputs may be unreliable.
Regarding the context, clinicians may require sufficient information to appropriately interpret AI-generated outputs and understand their confidence levels or limitations. Patients may need clear and accessible information regarding the role of AI within care or research processes. Developers and researchers require traceability mechanisms to support validation, debugging, performance monitoring, and continuous improvement. At the same time regulators and ethics bodies increasingly expect documented evidence demonstrating that risks are identified and controlled throughout the system lifecycle.
Explainability and the “Black Box” challenge
Human oversight is inextricably linked to the need for explainability. One of the much-discussed features of AI is its opacity, otherwise called the “black box” phenomenon, i.e., the inability even of technologically educated persons, including AI developers, to understand how certain outputs or decisions are generated. In healthcare, where each decision represents a high-stakes scenario due to its direct influence on human life, the inability of clinicians to explain and justify to their patients why a specific diagnosis or treatment prevails over another creates a gap of trust that is hard to overcome without concrete guarantees, such as transparency and explainability.
The highly personalised nature of healthcare also means that explainability must be adapted to different audiences and operational contexts. Understanding that explainability is a multi-layered requirement, from developers to healthcare professionals and, ultimately, to patients, demonstrates the importance of meaningful oversight with regards to how the AI system functions and being able to communicate its role and limitations with language that is clear, accessible, and trust-building.
The above considerations relate to the risk of automation bias, namely the tendency of users to over-rely on AI outputs, because systems appear technically sophisticated or authoritative. This risk is particularly acute when talking about decisions affecting human life and safety. Hence, proactive governance measures should include safeguards aimed at preserving critical human judgement, such as clear instructions for use, user training, escalation pathways, performance monitoring, and mechanisms allowing users to question or override outputs where appropriate.
Human oversight and data governance
Human oversight goes hand in hand with documenting and assessing the quality and representativeness of datasets used throughout development and validation activities, as analysed above. Such a holistic compliance approach contributes not only to scientific robustness, but also to accountability, trustworthiness, and compliance with the EU AI Act’s broader risk-management and data governance obligations.
Depending on the structure of the project, these safeguards may be reflected across Data Management Plans (DMPs), technical documentation, validation procedures, risk assessments, and broader governance frameworks.
Priority 3: Documentation and accountability
EU AI Act compliance must be evidenced
For high-risk AI systems, the EU AI Act requires providers to maintain technical documentation, enable record-keeping, implement risk management, provide information to deployers, ensure human oversight, address accuracy, robustness and cybersecurity, and follow conformity assessment procedures, where applicable. These obligations require a structured record of how the system was designed, developed, tested, governed, and updated.
For health-AI projects, this means that relevant decisions should be recorded progressively throughout the research and development lifecycle. The intended purpose of the system, data governance choices, risk assessments, validation activities, performance metrics, bias-related considerations, oversight arrangements, and security measures may all become relevant if an AI tool later moves toward clinical use, CE marking, or wider exploitation.
Data Management Plans support AI Act readiness
Data Management Plans (DMPs) are particularly important in this context. In health-AI projects, a DMP can help document how data is collected, accessed, shared, stored, protected, reused, and further processed. This is directly relevant to the EU AI Act readiness because data governance is one of the core requirements for high-risk AI systems, especially where training, validation, and testing datasets are used.
Documentation should develop with the AI system
Documentation should be created in parallel with the development of the EU AI-related work. If relevant records are only prepared at a late stage, it may become difficult to explain why particular datasets were used, how risks were assessed, how validation choices were made, or how human oversight measures were planned. This omission can prove to be a considerable obstacle when moving to market exploitation, where end users’ trust is essential for the success of a product or service.
Accountability requires clear roles
Accountability also requires clear allocation of responsibilities. While projects should have clarity regarding who is responsible for monitoring legal developments, maintaining documentation, reassessing the legal classification of tools and reviewing governance arrangements, these activities are often carried out collaboratively in large health research consortia, with contributions from legal, technical, clinical, and project management partners.
How DISARM Is Putting These Priorities into Practice
DISARM has already taken important steps to address the legal and governance challenges discussed above and to support the future development of trustworthy AI tools for ovarian cancer risk assessment and early detection.
From a data governance perspective, DISARM is designed to promote data quality, representativeness and responsible data management from the outset. The project’s clinical studies will involve up to seven clinical sites across five European countries, supporting the collection and validation of data across diverse healthcare settings and populations. In addition, patient associations are involved in the project and will participate in co-creation activities, helping ensure that patient perspectives are considered throughout the development process and supporting broader representation.
To minimise risks associated with bias, inconsistency and fragmented approaches to data collection, DISARM has already developed key governance and operational frameworks, including a Data Management Plan, study protocols, and standard operating procedures (SOPs). The project is also preparing training activities and establishing a common data collection and storage infrastructure to promote harmonised practices across participating clinical and technical partners.
GDPR considerations have likewise been integrated into the project’s preparatory work. Data protection requirements and safeguards for the processing of sensitive health-related data have been considered during the development of the study protocols and the co-designed informed consent forms, supporting a responsible approach to data use throughout the project lifecycle.
Beyond data governance, DISARM has also begun preparing for future AI regulatory requirements. An initial mapping of relevant obligations under the EU AI Act has already been conducted to help project partners understand which legal, technical and organisational measures may become relevant as AI-related activities progress. This forward-looking work supports future compliance planning, documentation efforts and evidence-building activities that may be required as the project’s AI tools mature and move closer to real-world deployment.
Taken together, these activities demonstrate how legal, ethical and governance considerations can be embedded early in a research project, helping create a stronger foundation for trustworthy, accountable and socially acceptable AI innovation in healthcare.
Looking Ahead: Legal Readiness as a Driver of Responsible Innovation
Early legal readiness supports responsible innovation, patient trust and future healthcare uptake of promising healthcare technologies and should therefore not be considered an obstacle to innovation. For DISARM, this means using the current mapping of AI Act requirements as a basis for future implementation, while the project’s scientific and technical work continues to develop.
As DISARM advances and results from the project begin to emerge, legal, ethical, and technical discussions will continue alongside scientific work. DISARM will closely monitor further guidance, harmonised standards, and implementation practices under the EU AI Act to strengthen the project’s long-term credibility and societal acceptance of its outcomes.
This work received funding from the European Union’s Horizon Europe Research and Innovation Programme under Grant Agreement No 101214318 (DISARM). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the Health and Digital Executive Agency (HaDEA). Neither the European Union nor HaDEA can be held responsible for them.
To stay up to date with the work, research, and results of the DISARM project, follow us on LinkedIn, X, and Bluesky.