Evaluation & Monitoring

Evaluation & Monitoring

AMEQUIS

The process to evaluate the ERNs and their Full Members will soon officially begin. This is part of the AMEQUIS framework, which aims to ensure that ERNs remain responsive to the needs of the rare disease patient community, deliver on their objectives, and improve their performance over time.

AMEQUIS stands for Assessment, Monitoring, Evaluation and Quality Improvement System. Whilst the AMEQUIS project has since concluded, it produced a refined process and criteria to assess the applications of new HCPs and new ERNs (the ‘A’ part of AMEQUIS). Furthermore, it will allow the ERNs and their members to systematically monitor, evaluate, and improve their activities to achieve a higher level of performance, spotting areas of best practice and areas for improvement (the ‘MEQUIS’ part of AMEQUIS).

An essential part of this process is evaluating the ERNs and their Full Members every five years. The evaluation will be performed both at the Network level and the level of each individual Full Member of the ERN.

The ERN eUROGEN Coordination Team will handle the network-level evaluation. Regarding the Full Member evaluation, only those who joined the network in 2017 are included in the evaluation process at this time. It will aim to assess the following:

  • If the HCP continues to provide specialised and quality care,
  • If the HCP team maintains the necessary levels of activity and experience,
  • What the contribution of the HCP team has been to the Network,
  • The value of the ERN for the HCP.

In practical terms, the ERN eUROGEN Coordination Team will send Full Members more detailed explanations of what is required in due course.

Full Members have been sent the link to the Evaluation Manual and Evaluation Toolbox previously but can click the link to download them again if necessary.

To summarise these, an Independent Evaluation Body (IEB) has been appointed by the European Commission to oversee the whole process. Full Members will be initially required to do a self-evaluation based on seven thematic areas, each comprising several measurable elements for which you will need to provide evidence. After that, this will be returned to the IEB, who will organise an online or on-site audit with a sample of full members from each ERN to follow up on those self-evaluations.

The ERN eUROGEN Coordination Team will follow this process closely and help Full Members as much as possible with the evaluation.

Data Collection

Each year, every healthcare provider (HCP) member of ERN eUROGEN has a responsibility to provide information on the number of patients with rare diseases that it is treating, as well as other activities undertaken to support the network. This information is then collated by the Coordination Team and submitted to the European Commission. HCPs should gather their data as a matter of routine for each period as soon as it becomes available.

The current data collection exercise is due for submission to the Commission by the end of March. This is to collect data for  2021. Future collections will take place annually in the first quarter of a year, collecting data for the previous year.

A copy of the full definitions for all metrics can be found on the ECP here. However, the main ones of concern to individual HCPs are summarised below for your convenience, along with links to the information gathered for previous collection rounds.

2.1: Number of new patients referred to the Health Care Providers

4.1: Number of education/training activities not accruing higher education credits

4.2: Number of formal educational activities accruing higher educational credits

5.1: Number of Clinical Trials/Observational prospective studies/Observational cohort or cases-control studies/case-series studies

5.2: Number of accepted peer-reviewed publications in accredited scientific journals

6.1: Number of Clinical Practice Guidelines and other types of Clinical Decision Making Tools adopted

6.2: Number of new Clinical Practice Guidelines or other types of new Clinical Decision Making Tools (clinical consensus statements or consensus recommendations) written

7.1: Number of congresses/ conferences/ meetings at which the ERN activities and results were presented

Metric 2.1

Number of new patients referred to the Health Care Providers participating in the ERN with the diagnosis of
a disease or condition that falls within the scope of the ERN.

Definition: The total number of new patients attending the ERNs’ Health Care Providers for the first time during the reporting period, whose disease or condition falls within the scope of the ERN, whatever their age, including visits to outpatient’s clinics, hospital discharges and emergencies, coming from national and international referrals.

Note: Although the data collection only specifies numbers of new patients, as a network, we also collect numbers of both total patients and procedures.

Metric 4.1

Number of education/training activities not accruing higher education credits aimed at healthcare professionals delivered by the coordination teams or HCP members of the ERN.

Definition: The total number of unique education/training activities not accruing higher education credits aimed at healthcare professionals and/or patients, created by the ERN coordination team, HCP members, Affiliated Partners, ePAGs/Patient Organisations/Representatives of the ERN and delivered during the reporting period. Can be online or physical presentations, courses, educational webinars, and/or videos delivered by the ERN.

Metric 4.2

Number of formal educational activities that are accruing higher educational credits aimed at healthcare professionals delivered by the coordination teams or HCP members of the ERN.

Definition: The total number of unique education/training activities accruing higher education credits, certified by a formal educational body, aimed at healthcare professionals and/or patients, created by the ERN coordination team, HCP members, Affiliated Partners, ePAGs/Patient Organisations/Representatives of the ERN and delivered during the reporting period. Can be online or physical presentations, courses, educational webinars, and/or videos delivered by the ERN.

Metric 5.1

This indicator covers different activities involving ERN members in at least two Member States:
5.1.a – Number of Clinical Trials
5.1.b – Number of Observational prospective studies
5.1.c – Number of Observational cohort or cases-control studies
5.1.d – Number of case-series studies

Definition: The total number of unique clinical trials, observational prospective studies (including academic and Industry driven studies), observational cohort studies, case control studies or case-series studies that involve ERN members from two different Member States and acknowledge the ERN, either ongoing or finalized during the reporting period.

Metric 5.2

Number of accepted peer-reviewed publications in accredited scientific journals regarding disease-groups falling within the scope of the ERN and acknowledging the ERN.

Definition: The total number of unique peer-review publications that have been accepted in scientific journals during the reporting period regarding disease-groups falling within the scope of the ERN. Publications should be PubMed accredited scientific journals and involve as major contributors at least two Health Care Providers from two different Member States within the ERN, and which include an explicit acknowledgement of the ERN such as “This work is generated within the European Reference Network for…” or “This work is supported by the European Reference Network for…”

Metric 6.1

Number of Clinical Practice Guidelines and other types of Clinical Decision Making Tools, adopted for diseases within the scope of the ERN during the reporting period.

Definition: The number of Clinical Practice Guidelines and other types of Clinical Decision Making Tools, such as clinical consensus recommendations for disease areas within the scope of the ERN not developed by the ERN, that were formally endorsed and adopted by the ERN Board, and are publically available (e.g. on the ERN website).

Metric 6.2

Number of new Clinical Practice Guidelines or other types of new Clinical Decision Making Tools (clinical consensus statements or consensus recommendations) written by the ERN, in progress or finalized during the reporting period.

Definition: The number of Clinical Practice Guidelines (CPG) and Clinical Decision Support Tools (CDST: clinical consensus statements or consensus recommendations), developed by the ERN, involving at least two Health Care Providers from two different Member States within the ERN. They should acknowledge the ERN, for diseases within the scope of the ERN where no guidelines existed previously and be according to an evidence based recognized methodology. The new CPGs or CDST can be in progress or finalised during the reporting period.

Metric 7.1

Number of congresses/ conferences/ meetings at which the ERN activities and results were presented.

Definition: The total number of congresses/conferences/meetings at which the ERN activities and results were presented via a dedicated slot in the programme/agenda, acknowledging the ERN, during the reporting period.