AMC4 ORO.FC.231(d)(1) Evidence-based training
CAA ORS9 Decision No. 32
RECOMMENDED GRADING SYSTEM METHODOLOGY — VENN MODEL
(a) To grade a competency, the instructor should assess the associated OBs of each competency against the following dimensions by determining:
(1) what was the outcome of the threat management, error management and undesired aircraft state management relating specifically to the competency being assessed;
(2) how well the flight crew member demonstrated the OB(s) when they were required. This includes:
(i) how many OBs the flight crew member demonstrated over the EBT phase (e.g. EVAL, MT, SBT) when they were required; and
(ii) how often the flight crew member demonstrated the OB(s) when they were required;
| Abbreviated word picture VENN model | ||||
|---|---|---|---|---|
| Grading | OUTCOME (1) | HOW WELL (2) = | HOW MANY (i)+ | HOW OFTEN (ii) |
| 1 | unsafe situation | ineffectively | few, hardly any | rarely |
| 2 | not an unsafe situation | minimally acceptable | some | occasionally |
| 3 | safe situation | adequately | many | regularly |
| 4 | safe situation | effectively | most | regularly |
| 5 | enhanced safety, effectiveness and efficiency | in an exemplary manner | all, almost all | always |
(b) Grades should be determined during each EBT module as follows:
(1) EVAL — overall performance of the phase for each competency at level 1 grading metrics.
(2) MT — overall performance of the phase at level 0 grading metrics. When the phase is graded ‘not competent’, it requires level 2 grading metrics.
Note: Only a limited number of competencies may be observed and graded in this phase (e.g. PRO, FPA, FPM); the others are ‘to be left in blank’.
(3) SBT — overall performance of the phase for each competency at level 1 grading metrics. Unless just culture and the necessary non-jeopardy environment during training may be compromised. In that case, level 0 grading metrics.
Note: In-seat instruction (ISI) should not be included in any assessment.
(c) Where any competency is graded below the minimum acceptable level of performance (grade 1 on a 5-point scale), an outcome of additional FSTD training is required.
(1) Additional level 2 grading metrics must be recorded.
(2) The flight crew member should not be released to unsupervised line operations until each competency is demonstrated at or above the minimum acceptable level of performance.
(d) Where all competencies are determined at or above the minimum acceptable level of performance (grade 2 on a 5-point scale), the outcome should be COMPETENT. Consistent grading below the average (2 on a 5-point scale) may indicate a need for training to elevate the performance to the average (grade 3 on a 5-point scale). As a minimum, the following conditions apply:
(1) Any competency graded with 2 requires level 2 grading metrics.
(2) Any competency graded with 2 in any simulator session of the 1st module followed by a grade 2 in the same competency in the EVAL of the 2nd module requires individual tailored training in the SBT of the 2nd module. (First example: 1st Module SBT graded with 2, 2nd Module EVAL graded with 2 in the same competency, thus the 2nd SBT should be an individual tailored training on that competency. Second example: 1s module EVAL graded 2, 2nd module EVAL graded 2 on the same competency, thus the 2nd module SBT should be individual tailored training on that competency).
(3) Any competency graded with 2 in three consecutive modules requires individual tailored training. If at the end of the tailored training (3rd SBT) the competency continues being graded with 2, additional FSTD training is required within the next 3 months. For instance, following the example above, the SBT in the 2nd Module was an individual tailored training. In the 3rd Module during the EVAL the same competency is graded with 2 and individual tailored training is applied. The SBT is graded with 2 again. The pilot may continue line operations but should receive additional FSTD training within the next 3 months.
(4) The operator should not release a flight crew member to unsupervised line operations when more than four competencies (the majority of the competencies — five competencies or above) are graded with 2 in any single simulator session of the module.
(5) Any EVAL graded with 2 in more than three competencies requires individual tailored training in the SBT. If at the end of the module more than three competencies continue being graded with 2, the pilot may continue line operations but should receive additional FSTD training within the next 3 months.
(e) ‘Individual tailored training’ refers to a simulator session tailored to the pilot’s individual training needs, which may require a different programme or syllabus. Normally, it may be done during the SBT and normally there is not an increase of FSTD volume (no extra simulator session). It may require an increased volume of training such as CBT, additional briefings, etc. Any individual tailored training may be substituted by additional FSTD training before the start of the next module.
(f) ‘Additional FSTD training’ refers to the fact that in addition to the requirements of tailored training, there is an increase of FSTD volume (extra simulator session). It normally happens after individual tailored training.