Atlassian Certifications - A Peek Behind the Curtains

[#1]

I have had the great honour of being a part of Atlassian’s Certification SME program for a while now, which has given me a greater insight and understanding of the entire process that is required to produce a single certification to the public.

With approval from the Atlassian Certification team I thought this would be a great opportunity to share my knowledge of this high-level exam development process with everyone. The goal of this article is to provide some insight into everything that goes into making sure that certification exams remain challenging but fair to potential exam takers.

Join me, as we walk through the process together from start to finish:

Defining the Target Exam Candidate & Topic List

The certification team is working to build out a robust catalog that covers knowledge of all the Atlassian solutions, such as the recently released “Confluence Essentials” certification.

Once the solution to be covered is identified, a profile for the target exam candidate will be constructed. For professional certifications, APC’s, the candidate is in an administrative product role. Typically, they have some real-world experience administering the products.. For the associate level certifications, ACAs, the target candidate is a current or future user, as these exams cover both concepts and use of the solutions.

Next, a list of exam topics is generated with a bit more detail of the solution features that will should be included and excluded from testing coverage. This list is then reviewed with subject matter experts, also known as SMEs, to confirm or debate what makes sense to be included and excluded based on the solution area(s) to be tested and why the knowledge level of the target exam candidate.

Creating Questions to Fulfill the Topic List

Once the target candidate and topic list has been decided, the certification team will get to work creating a pool of exam questions that provide an equal coverage of all the items listed in the topic list. Each exam question will include a correct answer as well as a number of incorrect answers with reasoning as to why the correct answer is correct and the incorrect answers are wrong.

All questions are reviewed internally by other members of the certification team to validate technical accuracy of the questions and provided answers and to make the language used in the questions and answers makes sense.

Subject Matter Expert - Peer Review of Questions

After the certification team is comfortable with the questions, a larger peer review will take place with SMEs that are made up of both external users and Atlassian employees from other parts of the organization.

The SMEs will review the exam questions, performing the same level of scrutiny on the language of the questions and answers as well as technical accuracy of the correct and incorrect answers. The main objective of both a peer review and the internal review is to make sure that the questions and the options are easy to read and understand.

The objective of this exercise is to ensure that exam takers are being tested on their knowledge of the solution and it’s features, instead of deciphering what is being asked about in the question.

Subject Matter Expert - Blind and Open Mock Exams

Next, a group of SMEs (usually from the same pool of either Atlassian employees or external users) will do two rounds of standard setting to determine the baseline difficulty for each question on the exam.

In the first “blind“ round, SMEs will review all questions and select what they believe to be the correct answer for each question. Next, they will act as judges to rate how difficult they believe each question will be for the profile of the target exam candidate to answer. During this round, SMEs will not be able to view the correct answer. This ensures that their initial rating is based only on their own interpretation of the difficulty for the target candidate.

In the second round, SMEs are able to view the answers for each question as well as the justification why the correct answer correct and why the incorrect answers are not correct. They will also be able to see if they were able to successfully answer the question correctly. The purpose of this second round is to given SMEs the opportunity to change their difficulty score now that they can see if they were able to get all the answers correct, and how challenging it will be for the target exam candidate to deduce the correct answer from the incorrect ones.

The final product will be a pool of exam questions of varied difficulty and proper coverage of the exam topics, which will adequately test the target exam candidate’s knowledge of the solution.

Publish & Monitor

After all of this, a detailed description of the certification will be finalized and made available to the public. The last thing that will happen is the certification team will monitor the new exam data to compare how exam takers are performing with the questions compared to the original baseline to ensure that the intention is matching the results.

Finally, the certification team does a rinse and repeat of this process which might be based on a few product changes, invalid questions, or an exam to get a refresh due to significant product changes since the exam’s initial release.

I hope this insight provides a better understanding of the process that goes into producing a certification exam. There are many people involved in the creation and review of every single question to ensure the exam taker can focus on the intent of the exam questions and not spend time interpreting complex scenarios. The process also aims to produce robust and fair exams that demonstrate the exam candidate's mastery of the covered solution.

I hope this has inspired you to consider taking a certification exam in the future and good luck!

P.S. Do you consider yourself an SME in any Atlassian products? Has reading this made you want to get involved? Then check out what's coming up and fill out the recruitment form if you’d like to participate in future certification projects: https://atlascert-sme.atlassian.net/wiki/external/OWQxNmZhOTRlMTk2NDMxNWIwODQyNjI0MDdlOWUxOWM

[Andy Gladstone]

Incredibly insightful article @Jimmy Seddon! Thanks for sharing your experience with us. I know the next time I take an exam I'll have a greater appreciation for the process that took place prior to my logging in to the certification platform.

[Susan Waldrip]

What a great article, @Jimmy Seddon ! This is not only good info as background, but for me at least, it's helpful in my preparation for an exam. I especially like the Blind and Open Mock Exams stage, great in-depth cross-check of each question. Thank you for the description, good to know Atlassian applies so many best practices in training development!