AMC 21.B.100(a) and 21.A.15(b)(6) Level of involvement (LoI) in a certification project for a type certificate (TC), a major change to a TC, a supplemental type certificate (STC), a major repair design or UK technical standard order (UKTSO) authorisation for an auxiliary power unit (APU)

CAA ORS9 Decision No. 1

1. Definitions

Risk: the combination of the likelihood and the potential impact of a non-compliance with part of the certification basis.

Likelihood: a prediction of how likely an occurrence of non-compliance with part of the certification basis is, based on a combination of the novelty and complexity of the proposed design and its related compliance demonstration activities, as well as on the performance of the design organisation.

Criticality: a measure of the potential impact of a non-compliance with part of the certification basis on product safety or on the environment.

Compliance demonstration item (CDI): a meaningful group of compliance demonstration activities and data of the certification programme, which can be considered in isolation for the purpose of performing a risk assessment.

The CAA panel: a CAA panel is composed of one or more experts who are responsible for a particular technical area. Each technical area addressed during product certification is covered by a CAA panel.

The CAA discipline: a CAA discipline is a technical subarea of a CAA panel.

CAA’s level of involvement (LoI): the compliance demonstration activities and data that the CAA retains for verification during the certification process, as well as the depth of the verification.

2. Background

The applicant has to submit a certification programme for their compliance demonstrations in accordance with point 21.A.15(b). The applicant has to break down the certification programme into meaningful groups of compliance demonstration activities and data, hereinafter referred as ‘CDIs’, and provide their proposal for the CAA’s LoI.

The applicant should also indicate the CAA panel(s) that is (are) affected by each CDI. This AMC explains:

(a) how to propose the CAA’s LoI for each CDI as per points 21.A.15(b)(6), 21.A.93(b)(3)(iii), 21.A.432C(b)(6) as well as 21.A.113(b); and

(b) how the CAA will determine its LoI on the basis of the criteria established in point 21.B.100.

The CAA will review the proposal and determine its LoI. Both parties, in mutual trust, should ensure that the certification project is not delayed through the LoI proposal and determination.

Additionally, in accordance with point 21.A.20, the applicant has the obligation to update the certification programme, as necessary, during the certification process, and report to the CAA any difficulty or event encountered during the compliance demonstration process which may require a change to the LoI that was previously notified to the applicant.

In such a case, or when the CAA has other information that affects the assumptions on which the LoI was based, the CAA will revisit its LoI determination.

In accordance with points 21.A.33, 21.A.447 and 21.A.615, irrespective of the LoI, the CAA has the right to review any data and information related to compliance demonstration.

Note: This AMC should not be considered to be interpretative material for the classification of changes or repairs.

3. Principles and generic criteria for the LoI determination

The CAA determines its LoI based on the applicant’s proposal in view of the risk (the combination of the likelihood of an unidentified non-compliance and its potential impact). This is performed after proper familiarisation with the certification project in three steps:

— Step 1: identification of the likelihood of an unidentified non-compliance,

— Step 2: identification of the risk class, and

— Step 3: determination of the CAA’s LoI.

This AMC contains criteria, common to all the CAA panels, for the determination of:

— any novel or unusual features of the certification project, including operational, organisational and knowledge management aspects;

— the complexity of the design and/or compliance demonstration;

— the performance and experience of the design organisation of the applicant in the domain concerned;

— the criticality of the design or technology and the related safety and environmental risks, including those identified on similar designs; and

— the data and activities to be retained by the CAA.

Note: Additional panel-specific criteria are available in further informative material published by the CAA. This material should not be considered to be AMC.

For CS-23 commuter (or CS-23 level 4 airplanes as defined in CS-23 Amdt 5), CS-25, CS-27 and CS-29 aircraft, all the panel-specific additional criteria should be considered. For the other products, the panel-specific criteria should only be considered for CDIs that affect noise, propulsion, development assurance and safety assessment (DASA), operational suitability data (OSD) and software and airborne electronic hardware.

The criteria used to determine the likelihood and the potential impact of an unidentified non- compliance generally allow a proportionate approach to be applied, in particular in order to differentiate between CS-25 and general aviation (GA) aircraft projects.

3.1. LoI determination at CDI level

The determination of the CAA’s LoI is performed at the level of the CDI (please refer to AMC 21.A.15(b)(5)).

The applicant should demonstrate that all the affected elements of the type-certification basis as specified in point 21.B.80, of the OSD certification basis as specified in point 21.B.82, and of the environmental protection requirements as specified in 21.B.85, the corresponding means and methods of compliance, as well as the corresponding certification activities and data, are fully covered by the proposed CDIs. If the provided data does not clearly show that this is the case, the applicant should clearly state to CAA that all the above-mentioned elements are fully covered.

Note: There could be different ways to ‘clearly show’ that all the elements of the certification basis are included in at least one CDI. For instance, this could be achieved by means of a ‘CDI reference’ column added in the table that lists all the elements of the certification basis.

3.2. Method for determining the likelihood of an unidentified non-compliance

3.2.1. Principle The likelihood of an unidentified non-compliance is assessed on the basis of the following criteria:

— novelty,

— complexity, and

— the performance of the design organisation.

3.2.2. Novelty

For the purpose of risk class determination, the following simplification has been made: a CDI may be either novel or non-novel.

Whether or not a CDI is novel is based on the extent to which the respective elements of the certification project, as well as the related requirement or means of compliance, are new/novel to either the industry as a whole, or to the applicant, including their subcontractors, or from a CAA panel perspective.

The determination that a CDI is novel may be driven by the use of new technology, new operations, new kind of installations, the use of new requirements or the use of new means of compliance.

When an applicant utilises a type of technology for the first time, or when that applicant is relatively unfamiliar with the technology, this technology is considered to be ‘novel’, even if other applicants may be already familiar with it. This also means that a type of technology may no longer be novel for one applicant, while it may still be novel for other applicants.

The following list includes some examples:

— new materials or combinations of materials;

— a new application of materials or combinations of materials;

— new manufacturing processes;

— a new or unusual aircraft configuration and/or system architecture;

— a novel reconfiguration of systems;

— a new interface or interaction with other parts or systems;

— the unusual location of a part or a system, or an unusual construction;

— a new or unusual use;

— new functions;

— new kinds of operations;

— the potential for new failure modes;

— the introduction of a new threat (e.g. new threats regarding fire, fuel, hydrogen, energy storage devices, etc.) or a new prevention/detection/mitigation method;

— new maintenance techniques;

— novel operating conditions or limitations;

— a new human-machine interface (HMI); or

— new flight or cabin crew tasks.

Another consideration is the extent to which the requirements, means of compliance or guidance have changed or need to be adapted due to particular novel features of the design. The following list includes some examples:

— recently issued or amended CSs with which the applicant has little or no experience;

— new or adapted special conditions;

— new or adapted equivalent safety findings;

— new or adapted deviations;

— new or adapted guidance or interpretative material;

— new or adapted means of compliance (i.e. other than those previously applied by the applicant) or unusual means of compliance (different from the existing guidance material and/or different from industry standard practices), e.g. the replacing of tests by simulation, numerical models or analytical methods;

— the use of new or adapted industry standards or in-house methods, as well as the CAA’s familiarity with these standards and methods;

— a change in methodology, tools or assumptions (compared with those previously applied by the applicant), including changes in software tools/programs; or

— novelty in the interpretation of the results of the compliance demonstration, e.g. due to in-service occurrences (compliance demonstration results are interpreted differently from the past).

— Additional new guidance/interpretative material in the form of new certification memoranda (CM) may be considered for the determination of novelty if its incorrect application/use may lead to an unidentified non-compliance. In the context of novelty, the time between the last similar project and the current project of the applicant should also be considered.

Regardless of the extent of an organisation’s previous experience in similar projects, a CDI may be classified as novel if there are specific discontinuities in the process for transferring information and know-how within the organisation.

3.2.3. Complexity For the purpose of risk class determination, the following simplification has been made: a CDI may be either complex or non-complex. For each CDI, the determination of whether it is complex or not may vary based on factors such as the design, technology, associated manufacturing process, compliance demonstration (including test set-ups or analysis), interpretation of the results of the compliance demonstration, interfaces with other technical disciplines/CDIs, and the requirements. The compliance demonstration may be considered to be ‘complex’ for a complex (or highly integrated) system, which typically requires more effort from the applicant. The following list includes some examples:

— Compliance demonstration in which challenging assessments are required, e.g.:

— for requirements of a subjective nature, i.e. they require a qualitative assessment, and do not have an explicit description of the means of compliance with that requirement, or the means of compliance are not a common and accepted practice; this is typically the case where the requirement uses terms such as ‘subjective’, ‘qualitative’, ‘assessment’ or ‘suitable’/‘unsuitable’

— in contrast, engineering judgement for a very simple compliance demonstration should not be classified as ‘complex’;

— a test for which extensive interpretation of the results may be anticipated;

— an analysis that is sensitive to assumptions and could potentially result in a small margin of safety;

— the classification of structures, depending on the conservatism of the method;

— an advanced analysis of dynamic behaviour;

— a multidisciplinary compliance demonstration in which several panels are involved and interface areas need to be managed (e.g. sustained engine imbalance, extended-range twin-engine operation performance standards (ETOPS), 2X.1309 assessment, flight in known icing conditions, full authority digital engine control (FADEC)- controlled engines, etc.);

— when the representativeness of a test specimen is questionable, e.g. due to its complexity;

— the introduction of complex work-sharing scheme with system or equipment suppliers.

For major changes, the complexity of the change should be taken into account, rather than the complexity of the original system.

Whether or not a CDI is complex should be determined in a conservative manner if this cannot be determined at an early stage of the certification project. When greater clarity has been achieved, the complexity may be re-evaluated and the LoI adapted accordingly.

3.2.4. Performance of the design organisation

The assessment of the level of performance of the design organisation takes into account the applicant’s experience with the applicable certification processes, including their performance on previous projects and their degree of familiarity with the applicable certification requirements.

For approved design organisations, the CAA uses relevant data to consider the design organisation’s expected performance at an organisational, panel or discipline level, depending on the availability of data.

This data stems from design organisation audits, the applicant’s measured level of performance on previous projects, and their performance during the familiarisation phase. The CAA shares this data with the respective design organisations (in the form of the design organisation approval (DOA) dashboard).

For each CDI proposed by the applicant, the DOA holder’s performance associated with the affected disciplines or panels is to be considered.

If one CDI affects more panels or disciplines than the others, a conservative approach should be followed in selecting the lower performance level. As an alternative, that CDI may be assessed separately for each affected the CAA panel or discipline.

If, for a well-established organisation, there is no shared performance data available at the panel level, it may be acceptable to propose the overall DOA holder’s performance. If the organisation or its scope are fundamentally new, the ‘unknown’ level of performance should be conservatively proposed by the applicant.

The determination of the performance of the design organisation may also take into consideration information that is more specific or more recent than the information on the DOA holder’s dashboard, e.g. experience gained during technical familiarisation with the current certification project, the performance of compliance verification engineers and of the affected technical areas, as well as the performance of the design organisation in overseeing subcontractors and suppliers.

The performance of some applicants’ organisations is not known if:

— The CAA has agreed in accordance with point 21.A.14(b) that the applicants may use procedures that set out specific design practices, as an alternative means to demonstrate their capability (excluding UK technical standard order (UKTSO) applicants for other than APU, covered by point 21.B.100(b)); or

— the applicants demonstrate their capability by providing the CAA with the certification programme in accordance with point 21.A.14(c).

In these cases, the assumed level of performance is ‘unknown’.

Exceptionally, the CAA may consider a higher level of performance for a specific CDI if that is proposed and properly justified by the applicant.

The following list includes some examples:

— a CDI with which the CAA is fully familiar and satisfied (from previous similar projects) regarding the demonstration of compliance proposed by the applicant;

— if the applicant fully delegates the demonstration of compliance to a supplier that holds a DOA, the performance level of the supplier may be proposed.

3.2.5. Likelihood of an unidentified non-compliance

Assessing the likelihood of an unidentified non-compliance is the first step that is necessary to determine the risk class.

The likelihood of an unidentified non-compliance should not be confused with the likelihood of occurrence of an unsafe condition as per AMC 21.A.3B(b). In fact, that AMC provides the CAA’s confidence level that the design organisation addresses all the details of the certification basis for the CDI concerned, and that a non- compliance will not occur.

The likelihood of an unidentified non-compliance is established as being in one of four categories (very low, low, medium, high), depending on the level of performance of the design organisation as assessed by the CAA, and on whether the CDI is novel or complex, as follows:

Step 1 — Likelihood of an unidentified non-compliance
CDI Performance level of the DOAH	No novel aspects, no complex aspects	No novel aspects, but complex ones; Novel aspects, but no complex ones	Novel and complex aspects
High	Very low	Low	Medium
Medium	Low	Medium	High
Low or unknown	Medium	High	High

3.3. Criticality

The second step that is necessary to determine the risk class is the assessment of the potential impact of a non-compliance on part of the certification basis regarding the airworthiness or the environmental protection of the product. For the purpose of risk class determination, the following simplification has been made: the impact of a non- compliance can be either critical or non-critical.

Some of the guidance below has been derived from GM 21.A.91, not due to a major/minor change classification, but because the same considerations may be applied to determine the effect of a non-compliance on the airworthiness or environmental protection at the CDI level. It is therefore normal that some of the CDIs of a major change that consists of several CDIs may be critical, and others may be non-critical.

The potential impact of a non-compliance within a CDI should be classified as critical if, for example:

— a function, component or system is introduced or affected where the failure of that function, component or system may contribute to a failure condition that is classified as hazardous or catastrophic at the aircraft level, for instance for ‘equipment, systems and installations’, e.g. where applicable as defined in 2X.1309;

— a CDI has an appreciable effect on the human–machine interface (HMI) (displays, approved procedures, controls or alerts);

— airworthiness limitations or operating limitations are established or potentially affected;

— a CDI is affected by an existing airworthiness directive (AD), or affected by an occurrence (or occurrences) potentially subject to an AD, a known in-service issue or by a safety information bulletin (SIB); or

— a CDI affects parts that are classified as critical as per CS 27.602/29.602, CS-E 515, or that have a hazardous or catastrophic failure consequence (e.g. a principal structural element as per CS 25.571).

If the classification of the potential impact of a non-compliance within a CDI as critical is based on the criterion that the CDI is affected by an AD, then the impact of a non- compliance within that CDI may be reclassified by the CAA as non-critical due to the involvement of the CAA in the continued-airworthiness process.

During the early stages of a project, the criticality in terms of the potential safety consequence of a failure may not always be known, but should be conservatively estimated and the LoI should be subsequently re-evaluated, if appropriate. 3.4. Method for the determination of risk classes The risk is determined as a combination of the potential impact of an unidentified non-compliance with part of the certification basis (vertical axis) and of the likelihood of the unidentified non-compliance (horizontal axis) using the following matrix. As a consequence, four qualitative risk classes are established at the CDI level.

Step 2 — Risk classes
Likelihood (see Section 3.2.5) Criticality (see Section 3.3)	Very low	Low	Medium	High
Non-critical	Class 1	Class 1	Class 2	Class 3
Critical	Class 1	Class 2	Class 3	Class 4

The various inputs and the resulting risk class determination are of a continuous nature, rather than consisting of discrete steps. The selected risk class provides the order of magnitude of the CAA’s involvement and is used as a qualitative indicator for the determination of the CAA’s involvement described in Section 3.5 below.

Under specific circumstances, the risk class that is determined on the basis of the above criteria may be reduced or increased on the basis of justified and recorded arguments. For a reused and well-proven item of compliance demonstration for which:

— the CDI is independent of the affected product type or model; and

— the design, operation, qualification, and installation of the product are basically the same; and

— the certification process is identical to one that was used in a modification already approved by the CAA,

— the CDI may be accepted as being similar, resulting in reduced LoI, as the likelihood of an unidentified non-compliance is low. Furthermore, when an identical CDI is reused for the compliance demonstration in a new project, there is no involvement in the compliance demonstration verification, as the likelihood of an unidentified non- compliance is very low.

3.5. Determination of the CAA’s LoI

The CAA’s LoI in the verification of compliance demonstration is proposed by the applicant and determined by the CAA in Step 3 on the basis of the qualitative risk class identified per CDI in Step 2, as well as by applying sound engineering judgement.

The CAA’s LoI is reflected in a list of activities and data, in which CAA retains the verification of compliance demonstration (e.g. review and acceptance of compliance data, witnessing of tests, etc.), as well as the depth of the verification. The depth of the verification for individual compliance reports, data, test witnessing, etc., may range from spot checks to extensive reviews. The CAA always responds to those retained compliance demonstration activities and data with corresponding comments or a ‘statement of no objection’.

In addition, some data that is not retained for verification may be requested for information. In this case, no ‘statement of no objection’ will be provided.

It is recommended that an LoI should be proposed for each of the the CAA disciplines involved. Depending on the risk classes determined in Section 3.4 above, the CAA’s LoI in:

(a) compliance demonstration verification data; and

(b) compliance demonstration activities (witnessing of tests, audits, etc.), may be as follows:

— risk Class 1: there is no the CAA involvement in verifying the compliance data/activities performed by the applicant to demonstrate compliance at the CDI level;

— risk Class 2: the CAA’s LoI is typically limited to the review of a small portion of the compliance data; there is either no participation in the compliance activities, or the CAA participates in a small number of compliance activities (witnessing of tests, audits, etc.);

— risk Class 3: in addition to the LoI defined for Class 2, the CAA’s LoI typically comprises the review of a large amount of compliance data, as well as the participation in some compliance activities (witnessing of tests, audits, etc.); and

— risk Class 4: in addition to the LoI defined for Class 3, the CAA’s LoI typically comprises the review of a large amount of compliance data, the detailed interpretation of test results, and the participation in a large number of compliance activities (witnessing of tests, audits, etc.).

By default, the following activities require the CAA’s involvement in all cases:

— initial issues of, and changes to, a flight manual (for those parts that require the CAA approval and that do not fall under the DOA holder’s privilege);

— classification of failure cases that affect the handling qualities and performance, when:

— performed through test (in flight or in a simulator); and

— initial issues of, and non-editorial changes to, airworthiness limitations.

If the risk assessment (Steps 1 and 2 above) is made on the level of a compliance demonstration activity or on the level of a document, the risk class provides an indication for the depth of the involvement, i.e. the verification may take place only for certain compliance data within a compliance document.

4. Documentation of the LoI

The LoI proposal in the certification programme should include the applicant’s proposal regarding the compliance demonstration verification activities and data that would be retained by the CAA, as well as the data on which the LoI proposal has been based. For this purpose, the applicant should appropriately document the analysis per CDI, considering the above criteria. In cases where the rationale for the assessment is obvious, it is considered to be sufficient for the applicant to indicate whether or not a CDI is novel or complex, and whether or not the impact is critical.

The CAA documents the LoI determination by accepting the certification programme or, if it deviates from the proposal, by recording its analysis regarding the deviations from the proposal, and notifies the applicant accordingly.

5. Sampling during surveillance of the DOA holder

It should be noted that all the previously defined risk classes may be complemented by the sampling of project files during surveillance of the DOA holder, independently from the ongoing certification project. This is necessary in order to maintain confidence in the DOA system and to constantly monitor its performance.