Evaluation

You can download the ERN eUROGEN Evaluation Plan and ERN eUROGEN Evaluation Report by clicking the links.

The process of evaluating the ERNs and their Full Members every five years was officially launched in December 2022, with a deadline for healthcare providers (HCP) to submit their data by 16 February 2023.

The evaluation was an opportunity to learn and improve the existing system, identify lessons learnt and areas for improvement, and ultimately reveal if the ERNs are delivering the expected benefits for patients. This is part of the AMEQUIS framework, which aims to ensure that ERNs remain responsive to the needs of the rare disease patient community, deliver on their objectives, and improve their performance over time. AMEQUIS stands for Assessment, Monitoring, Evaluation and Quality Improvement System. Whilst the AMEQUIS project has since concluded, it produced a refined process and criteria to assess the applications of new HCPs and new ERNs (the ‘A’ part of AMEQUIS). Furthermore, it will allow the ERNs and their members to monitor, evaluate, and improve their activities systematically to achieve a higher level of performance, spotting areas of best practice and areas for improvement (the ‘MEQUIS’ part of AMEQUIS).

Full information on the process is available on the EC’s webpage on the Evaluation of the European Reference Networks, including the legal framework, timeline, evaluation process, and documentation.

The Consortium consisting of IDOM Consulting, Engineering, Architecture (IDOM) and the Andalusian Agency for Healthcare Quality (ACSA) was appointed by the EU Commission as the Independent Evaluation Body (IEB) to perform the technical evaluation of the European Reference Networks and their members, under the framework of the Implementing Decision 2014/287/EU which set out the criteria for establishing and evaluating European Reference Networks and their members at least every 5 years.

The evaluation was performed both at the Network level and the level of each individual Full Member of the ERN. The ERN eUROGEN Coordination Team handled the network-level evaluation. Regarding the Full Member evaluation, only those who joined the network in 2017 were included in the evaluation process this time. The evaluation aimed to assess the following:

• If the HCP continues to provide specialised and quality care,
• If the HCP team maintains the necessary levels of activity and experience,
• What the contribution of the HCP team has been to the Network,
• The value of the ERN for the HCP.

Full Members were required to do a self-evaluation based on seven thematic areas, each comprising several measurable elements for which they needed to provide evidence. This was returned to the IEB, who organised an online or on-site audit with a sample of full members from each ERN to follow up on those self-evaluations.

ERNs & HCPs Evaluation Webinar

The EC delivered a webinar on the ERNs & HCPs evaluation. You can download the presentation slides or watch the recording, which is also embedded below:

Endo-ERN Videos

Endo-ERN has also produced some very helpful videos about the process, which are embedded below.

Each year, every healthcare provider (HCP) member of ERN eUROGEN has a responsibility to provide information on the number of patients with rare diseases that it is treating, as well as other activities undertaken to support the network. This information is then collated by the Coordination Team and submitted to the European Commission. HCPs should gather their data as a matter of routine for each period as soon as it becomes available.

The current data collection exercise is due for submission to the Commission by the end of March. This is to collect data for  2021. Future collections will take place annually in the first quarter of a year, collecting data for the previous year.

A copy of the full definitions for all metrics can be found on the ECP here. However, the main ones of concern to individual HCPs are summarised below for your convenience, along with links to the information gathered for previous collection rounds.

2.1: Number of new patients referred to the Health Care Providers

4.1: Number of education/training activities not accruing higher education credits

4.2: Number of formal educational activities accruing higher educational credits

5.1: Number of Clinical Trials/Observational prospective studies/Observational cohort or cases-control studies/case-series studies

5.2: Number of accepted peer-reviewed publications in accredited scientific journals

6.1: Number of Clinical Practice Guidelines and other types of Clinical Decision Making Tools adopted

6.2: Number of new Clinical Practice Guidelines or other types of new Clinical Decision Making Tools (clinical consensus statements or consensus recommendations) written

7.1: Number of congresses/ conferences/ meetings at which the ERN activities and results were presented

Metric 2.1

Number of new patients referred to the Health Care Providers participating in the ERN with the diagnosis of
a disease or condition that falls within the scope of the ERN.

Definition: The total number of new patients attending the ERNs’ Health Care Providers for the first time during the reporting period, whose disease or condition falls within the scope of the ERN, whatever their age, including visits to outpatient’s clinics, hospital discharges and emergencies, coming from national and international referrals.

Note: Although the data collection only specifies numbers of new patients, as a network, we also collect numbers of both total patients and procedures.

Metric 4.1

Number of education/training activities not accruing higher education credits aimed at healthcare professionals delivered by the coordination teams or HCP members of the ERN.

Definition: The total number of unique education/training activities not accruing higher education credits aimed at healthcare professionals and/or patients, created by the ERN coordination team, HCP members, Affiliated Partners, ePAGs/Patient Organisations/Representatives of the ERN and delivered during the reporting period. Can be online or physical presentations, courses, educational webinars, and/or videos delivered by the ERN.

Metric 4.2

Number of formal educational activities that are accruing higher educational credits aimed at healthcare professionals delivered by the coordination teams or HCP members of the ERN.

Definition: The total number of unique education/training activities accruing higher education credits, certified by a formal educational body, aimed at healthcare professionals and/or patients, created by the ERN coordination team, HCP members, Affiliated Partners, ePAGs/Patient Organisations/Representatives of the ERN and delivered during the reporting period. Can be online or physical presentations, courses, educational webinars, and/or videos delivered by the ERN.

Metric 5.1

This indicator covers different activities involving ERN members in at least two Member States:
5.1.a – Number of Clinical Trials
5.1.b – Number of Observational prospective studies
5.1.c – Number of Observational cohort or cases-control studies
5.1.d – Number of case-series studies

Definition: The total number of unique clinical trials, observational prospective studies (including academic and Industry driven studies), observational cohort studies, case control studies or case-series studies that involve ERN members from two different Member States and acknowledge the ERN, either ongoing or finalized during the reporting period.

Metric 5.2

Number of accepted peer-reviewed publications in accredited scientific journals regarding disease-groups falling within the scope of the ERN and acknowledging the ERN.

Definition: The total number of unique peer-review publications that have been accepted in scientific journals during the reporting period regarding disease-groups falling within the scope of the ERN. Publications should be PubMed accredited scientific journals and involve as major contributors at least two Health Care Providers from two different Member States within the ERN, and which include an explicit acknowledgement of the ERN such as “This work is generated within the European Reference Network for…” or “This work is supported by the European Reference Network for…”

Metric 6.1

Number of Clinical Practice Guidelines and other types of Clinical Decision Making Tools, adopted for diseases within the scope of the ERN during the reporting period.

Definition: The number of Clinical Practice Guidelines and other types of Clinical Decision Making Tools, such as clinical consensus recommendations for disease areas within the scope of the ERN not developed by the ERN, that were formally endorsed and adopted by the ERN Board, and are publically available (e.g. on the ERN website).

Metric 6.2

Number of new Clinical Practice Guidelines or other types of new Clinical Decision Making Tools (clinical consensus statements or consensus recommendations) written by the ERN, in progress or finalized during the reporting period.

Definition: The number of Clinical Practice Guidelines (CPG) and Clinical Decision Support Tools (CDST: clinical consensus statements or consensus recommendations), developed by the ERN, involving at least two Health Care Providers from two different Member States within the ERN. They should acknowledge the ERN, for diseases within the scope of the ERN where no guidelines existed previously and be according to an evidence based recognized methodology. The new CPGs or CDST can be in progress or finalised during the reporting period.

Metric 7.1

Number of congresses/ conferences/ meetings at which the ERN activities and results were presented.

Definition: The total number of congresses/conferences/meetings at which the ERN activities and results were presented via a dedicated slot in the programme/agenda, acknowledging the ERN, during the reporting period.