CAA Exam Format Info

11th September 2017   
CAA Exam Format Info

From the CAA – New Exam Questions Format Info

 

The Information Notice in respect of the new examination system, Quadrant, has now been published. It is available at:

 

http://publicapps.caa.co.uk/modalapplication.aspx?catid=1&pagetype=65&appid=11&mode=detail&id=7971

 

There have been two changes to the dates previously indicated. These changes affect Radio Navigation and Flight Planning.  Both subjects have been brought forward as a result of necessity: Radio Navigation - due to the increasing demand for PBN assessment; Flight Planning - to ensure that the second-SME overview of all new questions in all other subjects is possible.

 

The question concentration of new style to old style is very low. It remains our immediate intention to phase in new style questions only where deemed appropriate and in conjunction with new questions recently supplied to the ECQB. Over time the UK CAA SMEs will review older questions and convert them where possible to the new style – and leave untouched those questions only fitting the multiple-choice mould.

 

We have completed checking and adjusting the batch of ‘converted’ Air Law questions. A very low number will be going live in the new style of answer formats. It is expected that similar low numbers will be involved with the other subject areas too. Please remember, our intention is a gentle transition.

 

Multiple choice questions

Delivered in the same way, but using the new system, Quadrant.

 

In-line questions

Students make one or more choices from drop-down answer options. To obtain the mark(s) for this type of question, all of the drop-down options must be completed with the correct answers. There is no partial scoring or negative scoring.

Example of stem wording:

Use the drop-down option(s) to complete the following statement; marks are only awarded for a fully correct answer.

Question…..

 

Multi-select questions

These questions require candidates to select the answers that apply to the question. Marks are only awarded for a fully complete answer, where all the correct selections have been made. There are no marks for a partially correct response and no negative marking.

There are no plans to leave these questions undefined – that is to say each question will clearly state the number of selections to be made.

Example of stem wording:

Question…..

(You are required to select the three correct answers to obtain the mark(s) for this question.)

 

Text (type-in-the-answer) questions

These questions require the input of data. While there are many applications for these types of questions our philosophy is to keep their use simple. In general, these question types will be used for numerical answers. These questions will specify clear instructions to the candidate.

Where a question is testing knowledge of a legal value (e.g. a final reserve time) there will be no tolerance (answer range) applied for an incorrect input.

Those questions requiring mathematical calculation will have a tolerance applied in keeping with the style of the question under the guidelines set by EASA. The tolerance value will not be supplied to the candidate.

Questions are being checked for clarity. For example, new style questions requiring the calculation of height / altitude / pressure altitude etc., using a lapse rate, will have the examiner’s lapse rate value quoted (27 ft / 1000 versus 30 ft / 1000 etc).

Marks will only be awarded for an answer within tolerance (range). Where there is no range, exact answers only receive marks.

Example of stem wording (for Air Law/Operational Procedures/Legal values):

You must type in the exact answer to obtain the mark(s) available for this question. Use only whole numbers. 

Question…..

 

Monitoring

Our assessment partner has provided IT tools for performance monitoring, which will allow questions to be identified that are not performing well for candidates. These questions will be re-checked for fairness / accuracy and appropriate action will be taken to ensure robust and fair assessment and marking.

Candidates will still be able to leave comments in the same way as they do now. I would urge you to advise your students to leave comments on questions that, in their opinion, are unfair. These comments are used to make improvements and are gratefully received.

 

Just for clarification in the meantime:

 

·         The ‘type-in-the-answer’ questions will be numerical and the tolerance, or range, stipulated by EASA in the ECQB release will negate the need for the required accuracy to be in question for the majority of questions. However, SMEs will consider this during their review to ensure that candidates receive fair examinations. 

 

·         Multi-select questions will provide the number of options to be selected in the stem.

 

To allay concerns, please be aware that the initial migration of questions to Quadrant has been in the original multiple-choice format. Initially, only a few questions will be converted, where deemed appropriate, to the new answer styles. The initial dilution with new style questions will be low – it is not our wish to de-harmonise the examination system.

 

The video demonstrates the functionality of the entire e-Examination platform – Quadrant, which has been developed to be used in many applications including Part FCL and AML.

 

Part FCL assessment will involve only the four styles described previously, these are:

 

  • Multiple-choice – the current question style where the candidate selects a single answer from four options.
  • Multi-select – candidates must select options from a list to correctly answer the question. This question style will replace some of the current questions where a list of numbered options is provided in the question stem - and the candidate must select the correct group of options in the answer.
  • Inline – involves the correct completion of sentences using a dropdown menu containing words, numbers, or a combination of both.
  • Text – type-in-the-answer questions requiring the input of keystrokes. This style may be used to answer calculation-type questions, or questions requiring the extraction of data from a chart or table.

 

The essay style question referred to in the video is not applicable for Flight Crew examinations.