GM1 Article 11 Annex E. Functional test based (FTB) methoology

CAA ORS9 Decision No. 46

GM.FTB

(a) The FTB methodology is used in the following situations:

(1) For the UAS Designer to conduct an FTB design appraisal, which demonstrates the UAS operational reliability.

(2) For the UAS Operator to take credit from the FTB design appraisal conducted by the UAS Designer to show compliance with the relevant OSOs. This has the benefit for the UAS Operator going through the OA process to provide automatic compliance with a number of OSOs, in particular when the Operator does not have a fully established relationship with the Designer or does not have access to the UAS design data.

(3) For the UAS Operator to demonstrate safe and successful operations over time in order to expand their operational approval, based on the concept of “reliability growth model”.

(4) The FTB methodology is not considered feasible for UAS operations with a SAIL above V.

These three approaches are detailed in the following sections.

(b) The UAS Designer may use the FTB methodology to conduct an FTB design appraisal, which demonstrates the UAS operational reliability. The following aspects should be considered in applying the FTB methodology:

(1) Functional testing should be conducted, which may be divided into two types:

(i) ‘Functional tests’ are operational test cycles that are fully representative of end-state operations, with test points that verify safe operation at the operational limits and corners of the UA envelope.

(ii) ‘Induced failure tests’, which typically address demand-based systems, i.e. systems that are not continuously active and are triggered only under certain failure conditions. These tests are required where functional tests alone are not sufficient to demonstrate operational reliability, e.g. to cover likely failures.

(c) Although ASTM F3478-20 is not an officially accepted standard, it provides useful guidance for the development and deployment of an FTB campaign. Topics discussed in ASTM F3478-20 include:

(1) Development of operational flight tests, as well as specific (ground) testing to verify underlying system parameters statistically, e.g. component and UA MTBF, operational hazard rates, parachute reliability. Both the UAS Designer and the competent authority need to understand the assumptions made when attributing a distribution type to a system parameter (e.g. exponential, normal, Weibull, gamma distributions).

(i) Any infringement or loss of control occurring during the test campaign will require a root cause analysis. If design modifications are necessary following the investigation, an analysis is performed to assess whether the FTB flying hours performed prior to the modification may still be considered valid. Some tests or the entire FTB campaign might have to be reconducted.

(ii) UAS Designers and competent authorities should be cognisant of the systems, such as software or airborne electronic hardware-based systems that do not allow accurate analysis under operational time or demand-based testing. These systems should use system-specific analyses (e.g. multiple condition/decision coverage, model checking, development assurance, design and analysis) appropriate to the SAIL level.

(d) The CAA may grant a specific flight test authorisation to conduct the functional and induced failure tests needed to complete the FTB campaign.

(e) The UAS Operator may take credit from the FTB design appraisal conducted by the UAS Designer to show compliance with the relevant OSOs. To do so, the following conditions need to be met:

(1) The functional tests performed by the Designer cover the full operational scope/envelope intended by the Operator.

(2) The functional tests performed by the Designer have been executed following the operational procedures and the remote crew training referred to in the operational authorisation, which meet the integrity assurance of the associated OSOs.

(3) The Operator’s maintenance instructions are established based on the Designer’s instructions and requirements which were used for maintenance, repair, or replacement of UAS sub-systems during the functional tests performed by the Designer.

(4) Any deviation in the UAS configuration from the configuration used by the Designer during the FTB campaign are confirmed by the Designer to not impair the validity of the FTB design appraisal.

(5) The minimum number of test cycles has been achieved for the corresponding SAIL, with no failure occurrence:

(i) 30 hours for SAIL 1;

(ii) 300 hours for SAIL 2;

(iii) 3000 hours for SAIL 3; and

(iv) 30000 hours for SAIL 4

Note: this allows achieving a factor of 95% confidence in the reliability of the operation per a binomial/Poisson distribution.

(6) The functional tests performed by the Designer have been executed by the Designer according to principles or standards considered adequate by the CAA, including the following:

(i) The functional tests have been executed using an acceptable sample size of UAS.

(ii) Safe life limits for UAS sub-systems sensitive to wear-out conditions based on the maximum cycles and hours demonstrated by one or more fleet leader UAS (i.e. the UAS with the longest time and/or cycles compared to other UAS used during the FTB campaign) have been derived by the Designer and captured in the FTB design appraisal limitations.

Note: induced failure tests may also help demonstrate compliance with the following OSOs:

(iii) OSO 5 and Containment requirements: safety and reliability / safe design (e.g. induced failure tests with no loss of control or containment as pass-fail criteria).

(iv) OSO 6: C3 link performance appropriate for the operation (e.g. if the distance from a C2 radio transmitter/receiver is a critical factor, then the demonstration of the maximum allowable range from the transmitter/receiver in the most likely worst-case conditions is needed).

(v) OSO 18: Automatic protection of the flight envelope from human errors.

(f) The UAS Operator may use the FTB methodology to demonstrate safe and successful operations over time in order to expand their operational approval, based on the concept of “reliability growth model”, as follows:

(1) The UAS Operator should operate with a low SAIL approval and then, through operational experience, gather sufficient operational data to justify an increase in the SAIL based upon the increase in operational reliability demonstrated. This approach is only valid under representative operating conditions, without requesting additional strategic or tactical mitigations.

(i) The CAA may accept accumulation of FTB hours between Operators if the UAS configuration, operational procedures, training, etc. are demonstrated to be equivalent.

(ii) This method does not cover expanded operating conditions, which would require additional testing and/or analysis to be performed by the UAS Designer.

(iii) As an example, the Operator may start operating with a SAIL 2 operational approval to fly over a population density of up to 500 people per km². As they demonstrate 3,000 hours of operation with no loss of control, they may be approved by the CAA to operate at SAIL 3 under the exact same operating conditions, with an allowable maximum population density increased to 5,000 people per km².

(iv) The UAS Operator should demonstrate that:

(A) the next population band does not introduce new hazards. If new hazards are introduced, they should be mitigated through test or analysis.

(B) The conditions listed in (e) have been met, in particular the minimum number of test cycles required for the desired SAIL per (e)(5).