GM2 ORO.FC.231(d)(1) Evidence-based training
CAA ORS9 Decision No. 32
RECOMMENDED GRADING SYSTEM METHODOLOGY — VENN MODEL
(a) Grades may be determined during each EBT module as follows:
(1) For each assigned grade:
(i) the observed performance should be identified with one or more OBs; and
(ii) the OB(s) should simply link the observed performance to the competency; they are not to be used as a checklist.
(2) At the completion of the EVAL, the grade should be the overall assessment of the performance of each competency during the EVAL. Although it is not recommended, if the instructor performs an overall grade (additional to level 1), it should be at level 0 grading metric (competent or not).
(3) The underlying philosophy of the individual tailored training and additional FSTD training is the identification of the pilot’s individual training needs during the EVAL or EVALs. However, there may be cases in which such an identification may be complemented using other phases or combination of phases along the EBT programme. Nevertheless, when this happens consistently to a large number of pilots, it may indicate a problem of instructor standardisation.
(4) At the completion of the MT, only a limited number of competencies can be graded. The others are to be left in blank. Note: The grade of a competency as ‘not observed’ is a relevant set of data to be used in the EBT programme (e.g. may be used for instructor concordance assurance programme, programme design, etc.), while ‘competency left in blank’ is stating the obvious, which is that MT is a skill retention phase and therefore it focuses on only some of the competencies which may provide NO opportunity to observe all the competencies.
(5) At the completion of the module, grades should be assigned for each competency, based on the overall assessment of training during the SBT.
(6) In exceptional occasions, the instructor may have been unable to assess one or two competencies in the EVAL or SBT. A ‘not observed’ may be graded. The training system performance and concordance assurance system may use these metrics to improve instructors’ standardisation and the EBT programme design. When the operator grades the MT alone (instead of grading the MT and EVAL together), a ‘not observed’ grading may be frequent. It also occurs when the instructor grades each one of the manoeuvres.
(b) The word pictures are standardised according to the VENN model but may be simplified once instructors become familiar with the system.
| Word picture VENN model | |
|---|---|
| Application of procedures (PRO) | |
| 5 | The pilot applied procedures in an exemplary manner, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot applied procedures effectively, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot applied procedures adequately, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot applied procedures at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot applied procedures ineffectively, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |
| Communication (COM) | |
|---|---|
| 5 | The pilot communicated in an exemplary manner, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot communicated effectively, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot communicated adequately, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot communicated at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot communicated ineffectively, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |
| Flight path management — automation (FPA) | |
|---|---|
| 5 | The pilot managed the automation in an exemplary manner, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot managed the automation effectively, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot managed the automation adequately, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot managed the automation at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot managed the automation ineffectively, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |
| Flight path management — manual control (FPM) | |
|---|---|
| 5 | The pilot controlled the aircraft in an exemplary manner, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot controlled the aircraft effectively, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot controlled the aircraft adequately, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot controlled the aircraft at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot controlled the aircraft ineffectively, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |
| Application of knowledge (KNO) | |
|---|---|
| 5 | The pilot showed exemplary knowledge, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot showed adequate knowledge, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot showed adequate knowledge, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot showed knowledge at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot showed inadequate knowledge, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |
| Leadership & teamwork (LTW) | |
|---|---|
| 5 | The pilot led and worked as a team member in an exemplary manner, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot led and worked as a team member effectively, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot led and worked as a team member adequately, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot led and worked as a team member at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot led or worked as a team member ineffectively, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |
| Problem-solving & decision-making (PSD) | |
|---|---|
| 5 | The pilot solved problems and made decisions in an exemplary manner, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot solved problems and made decisions effectively, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot solved problems and made decisions adequately, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot solved problems and made decisions at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot solved problems or made decisions ineffectively, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |
| Situation awareness (SAW) | |
|---|---|
| 5 | The pilot’s situation awareness was exemplary, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot’s situation awareness was good, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot’s situation awareness was adequate, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot’s situation awareness was at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot’s situation awareness was inadequate, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |
| Workload management (WLM) | |
|---|---|
| 5 | The pilot managed the workload in an exemplary manner, by always demonstrating almost all of the observable behaviours to a high standard when required, which enhanced safety, effectiveness and efficiency |
| 4 | The pilot managed the workload effectively, by regularly demonstrating most of the observable behaviours when required, which resulted in a safe operation |
| 3 | The pilot managed the workload adequately, by regularly demonstrating many of the observable behaviours when required, which resulted in a safe operation |
| 2 | The pilot managed the workload at the minimum acceptable level, by only occasionally demonstrating some of the observable behaviours when required, but which did not result in an unsafe situation |
| 1 | The pilot managed the workload ineffectively, by rarely demonstrating any of the observable behaviours when required, which resulted in an unsafe situation |