The diagram depicts the specific areas where the HireVue platform was affected by this project's implementation.
The HireVue Platform
The HireVue platform enables users to create, schedule, and assess asynchronous and in-person job interviews. The primary interview format is on-demand video interviewing. Interviewing panel members can record their questions, and candidates can record their answers for later evaluation. On average, candidates complete 18,000 interviews a day. Candidates have the flexibility to participate in these interviews at their convenience.
Project Context
CodeVue is an add-on feature set for the HireVue On Demand asynchronous video interviewing package. It grants users access to skill-based assessments tailored for recruiting engineers and developers through coding challenges. Users have the option to craft their own custom programming challenges or choose from a library of pre-built and verified challenges offered by HireVue's services team. Additionally, the Live Interviewing experience includes a coding whiteboard feature, allowing hiring managers and interview panelists to collaborate with candidates on coding tasks.
Recognizing the opportunity for growth and increased profitability, HireVue set out to expand the initial feature sets of CodeVue. The primary goal was to introduce additional functionalities that would not only enhance CodeVue's appeal but also allow them to enter a larger market segment.
.
The features of CodeVue had a significant impact on nearly all user personas I was designing for. Foremost among them were the candidates and their interview experience, which received the highest priority in the conceptualization of solutions. It was vital to create highly intuitive experiences for candidates, eliminating any distractions that might arise from navigating the interface, thus enabling them to focus entirely on completing the challenges.
Also, non-technical talent acquisition specialists were regularly responsible for crafting programming challenges or interviews that involved coding tasks. They collaborated closely with hiring managers to identify the most effective programming challenges to include in the interviews, while some hiring managers even created challenges themselves.
Hiring managers were also concerned about the potential for candidates to cheat on the code challenges, fearing that answers could be easily found on the Internet and subsequently copied and pasted.
Team Members
Adam Hennings - Product Management
Bradley Trinnaman - Product Design
Thatcher Risom - Engineering Lead
Chris Thatcher - Engineering
Scott Keller - Customer Success
Design Responsibilities
Partner with a product manager to assist in discovery efforts
Usability research and data analysis
Contribute to release strategy
Assist in competitive analysis
Handoff prototypes and static mockups for development inspection
Review the development builds before launching to production
Problem Space
The integrated development environment (IDE) provided for candidates to complete the challenges was out of date and not user-friendly. Candidates expressed difficulties using it during job interviews, which added stress and extra time to their overall experience of answering programming challenge questions. While the challenge types were primarily focused on computer programming, customer feedback indicated a desire to assess other essential developmental skills, particularly in the area of database query for recruiting analysts familiar with SQL and database querying.
Additionally, the question library experience within the HireVue platform was also found to be non-intuitive and required enhancements to accommodate programming challenges effectively.
Team Objectives:
Update the IDE to a more user-friendly version and make it compatible with database challenges.
Expand the range of challenge types to include other vital developmental skills, like adding a SQL challenge type for database queries.
Review the anti-cheating features to ensure the integrity of the assessment process.
Improve the challenge creation workflow and incorporate the option to create database query challenges.
Who I am designing for:
Technical Candidates
Graduates in software development have foundational skills in computer programming, coding, testing, and debugging. They approach challenges logically and analytically, using software development tools, frameworks, agile methodologies, and current platforms.
Goals and Behaviors:
They look for jobs that meet their requirements in terms of compensation, schedule flexibility, commute time, career growth, and skill development.
Pain Points:
Candidates receive minimal feedback, making it difficult to assess their performance and areas for improvement.
Interview Panelist
Interview panelists assist hiring managers in conducting candidate interviews, and evaluating them according to criteria and job requirements. Their responsibilities include conducting structured interviews, posing relevant questions, and actively listening to candidates' responses.
Goals and Behaviors:
Evaluate job applicants thoroughly to find qualified individuals aligned with the company's values and requirements.
Pain Points:
Training on potential biases and mitigating them through structured interview techniques and objective evaluation criteria.
Hiring Manager
Responsible for overseeing the recruitment and selection process for various positions within the organization. They align candidates with company values and goals. Hiring manager responsibilities are additional to regular duties.
Goals and Behaviors:
Identify and hire candidates with the required skills and qualifications.
Pain Points:
Struggle with learning and adapting to new technologies. Responsibilities are additional to regular duties.
Discovery and Research Overview
Upon the completion of a HireVue interview, every candidate is requested to fill out a customer satisfaction survey. This survey collects feedback on their interview experience and highlights areas that could be improved. I had access to the survey responses of candidates who undertook interviews with coding challenges. From this pool, I selected a small sample and began analyzing the data to identify patterns and themes that could enhance the user experience of the IDE.
Conducting competitive audits was another aspect of our research. We explored how candidates experienced challenges in other platforms and examined the design of IDEs used for database challenge completion. Additionally, we audited Learning Management System (LMS) platforms utilized for teaching SQL and database queries.
.
For usability testing, I created mock-ups with incremental improvements to the IDE. We programmed these changes into a development environment and tested prototypes to observe their impact. This allowed for a unique approach to usability testing. During “coding challenge parties” at HireVue, where challenges were created to be added to the question library, about ten staff engineers would complete the challenges and rate their difficulty level (easy, medium, or difficult). I closely observed these engineers while they tackled the challenges with the new IDE changes. Afterward, I sought their feedback and suggestions for further enhancements.
Throughout the IDE's development, I conducted three sessions in this manner with our internal engineers before releasing an updated IDE to candidates, incorporating the valuable insights gathered from the testing and feedback process.
The data represents a single sample from a selected population comprising candidates who responded to the C-SAT survey after participating in a CodeVue challenge interview.
At first, I was surprised by the abundance of comments in the "Highly Satisfied" category. I presumed that candidates might have been content with the IDE. However, upon further examination, I discovered numerous negative comments about the IDE. Despite this, the candidates expressed high satisfaction with the overall interview experience.
The same data sample was clustered into emergent themes
Themes
After clustering the data, the following themes emerged:
Interview onboarding
Coding assistance (Auto-complete)
Requesting more time to solve the challenge
Writing custom test cases
Interview experience
More practice questions
Using other systems than CodeVue
IDE improvement
IDE performance
During an on-site study with customers, we discussed the design of the question bank where CodeVue Challenges are managed. The customers expressed difficulties in selecting and previewing challenges from this bank.
CodeVue: On-site Observation Study
As part of our research, we utilized another method: conducting on-site visits to customers who had talent acquisition specialists collaborating with hiring managers to create their interviews. Over a period of three days, we closely observed the methods and coordination efforts employed by these teams during the interview creation process. Additionally, we conducted interviews with the hiring managers to gather their insights on improving the creation process of coding challenges. We also sought their input on developing a database query version of the challenge.
Furthermore, we took the opportunity to gather feedback on the HireVue question library or question bank, aiming to identify areas where improvements could be made. Our focus during these visits was to understand the information that hiring managers and talent acquisition specialists needed to make informed decisions when selecting challenge types.
Original IDE - The data collected from the C-SAT survey primarily revolved around the original design of the IDE.
(Right) The primary focus of complaints was on the switching between the Challenge Tab and Test Case Output tab, as well as the lack of ability to resize panels.
Design Solutions
Improved IDE
The common themes expressed by candidates in the customer satisfaction survey revolved around feedback regarding the IDE. They expressed the desire to have the ability to write their own test cases, and they also sought coding assistance, such as features like type-ahead suggestions and keyboard shortcuts. Additionally, candidates requested a dedicated space for debugging their code and an opportunity to explain their thought process while answering the challenge.
Furthermore, candidates expressed interest in the IDE's performance, seeking faster processing of their outputs. Checking their answers became problematic due to the limited visual real estate offered by the IDE.
.
The output window was hidden behind a tab, making it challenging for candidates to switch between the code challenge and the output window to understand any issues with their code or why they weren't answering the question correctly. To seamlessly blend the typically dark mode interfaces of IDEs and coding environments with HireVue's predominantly light platform, I developed a dark inner face color palette that harmonized with the lighter interface.
With a focus on ensuring the IDE's intuitiveness for candidates, I started implementing changes that incorporated layout patterns commonly found in professional IDEs, particularly those used in Microsoft Visual Studio, which was the most widely utilized IDE among engineers. Additionally, I needed to create space and provisions for the upcoming database challenge type to be seamlessly integrated into the IDE.
The first change to the designs involved creating a new Dark UI color palette.
Here are some initial sketches of the new IDE design.
I built a UI component library for the new IDE.
The final IDE design mockup features:
(Left) A challenge description panel with an Attachments tab.
(Top Right) An editor panel that allows candidates to select Key Bindings and Languages.
(Bottom Right) The Test Case Output panel is now intuitively placed and resizable, enabling candidates to inspect their output effectively.
An image from the production release demonstrates the dynamic panel interactions.
Both Candidates and Hiring Managers expressed a desire for auto-complete functionality, and I was delighted to implement this feature.
In the responsive design, the panels are resized to remain visible and functional. Feedback regarding moving the timer to the top header indicated that it provided a strong affordance, being easily seen while working, without becoming overly distracting.
Database Challenge Additions
By implementing dynamic sliders that allow candidates to resize windows and choose their preferred layout, I successfully transformed the IDE into a more adaptable tool, especially for database challenges.
Instead of having the output area solely for displaying programming challenge outputs, I modified it to accommodate data tables. These data tables were designed to be responsive and resizable, catering to the users' preferences for table size.
.
For SQL or database challenges, it is common to provide an example of a correct answer to guide candidates in generating the correct response independently. The resizable panels offered candidates more visual real estate, enabling them to better analyze the tables in the output section and improve their problem-solving approach. These changes not only enhanced the user experience but also made the IDE more versatile for different types of challenges, particularly in the context of database-related tasks.
Regarding the database challenges, the Output panel is substituted with the Query Results panel, which is capable of displaying data tables.
Question Bank Redesign
In the older version of the question bank, users were presented with a small modal window offering limited information when selecting coding challenges. However, in the new design, I enlarged the UI to cover the entire viewport window, providing a more comprehensive view. Additionally, we introduced tags indicating the difficulty level of each challenge, allowing users to quickly identify easy, medium, or difficult challenges.
.
Concise challenge descriptions and code snippets were made available for users to review. The redesigned question bank now follows an intuitive shopping pattern, enabling users to add questions to a "shopping cart area." This enhancement allows them to rearrange questions and select the coding languages they want candidates to use when tackling the challenges. The new design streamlines the process, offering users more control and efficiency in managing coding challenges.
The previous Question Bank was implemented as a model that overlaid the Position Creation Page in HireVue.
The final design of the Question Bank incorporated the reuse of the filtering component I designed for the Analytics Dashboards, enabling users to search for Coding Challenges efficiently. Additionally, I implemented color-coded tags to enhance visibility and user experience.
When challenges are added to the right, users have the option to choose which coding languages to assign to each challenge.
Video Playback Enhancements:
In the HireVue platform, hiring managers and interview panelists use the Evaluator interface for evaluations. When a CodeVue challenge is included in an asynchronous on-demand job interview, users see a dedicated section with valuable features, including video playback to observe the candidate's code-writing process.
.
It also highlights instances where the candidate may have left the browser window to search for external information or copied and pasted text. These features help identify potential cheating based on the hiring manager's preferences and expectations. I ensured a consistent experience by incorporating the new dark interface and re-designing interactions within the challenge review panel.
Video Playback shows hiring managers candidate actions such as pasting text or losing browser focus during the challenge. It also includes a Code Similarity score, comparing responses to detect potential cheating.
*The Evaluator interface was designed by Eleanor Kramer. I integrated the CodeVue reviewing panel into the interface, which presented a challenge as my Dark UI needed to blend with HireVue's Light UI.
In HireVue's Evaluator section, Hiring Managers can rate and review candidates' responses to CodeVue challenges after interviews.
The current UI, utilized by Hiring Managers and Technical Recruiters, has received negative feedback about its usability. Users find it challenging to discern where to begin and how the inputted information is utilized within the system.
Creating a Database Challenge: Prototype
The final enhancement I worked on during my time with the CodeVue team involved rebuilding the self-service challenge creation workflow. Our goal was to incorporate the ability to create both programming challenges and the new database query challenges. The existing interface had received feedback from users, particularly talent acquisition specialists unfamiliar with writing programming challenges, stating that it was difficult to understand. This was especially true when it came to creating database challenges, where a space for adding a database schema was needed.
Story Mapping Workshop
During the development process of this new workflow, the team adopted the story mapping methodology, which was a transformative experience for me. I have since started using story maps instead of drawing user flow charts in my projects. To create this workflow, we held a workshop where two engineers and one customer service director participated in crafting their own story map for how they envisioned the workflow should progress.
.
We gathered everyone's ideas from the story maps and identified common elements, which we then aggregated to create a master story map. This master story map served as the foundation for building the prototype of the new challenge creation workflow.
Data Base Challenge Creation - Prototype
The redesign of this workflow was conceptualized as a prototype; unfortunately, I moved to another team before it could be implemented in our production environment. My idea was to create a wizard-like experience, guiding users step-by-step through the completion process, so they only had to focus on one task at a time.
Static mockups from the new challenge creation prototype
Project Outcomes
Working on products in the CodeVue space proved to be a unique challenge for me, considering that the users I tested my solutions with were engineers, whereas I lacked a strong background in coding. Despite this, the project provided me with valuable insights, especially in building interactions tailored for edge-cases and general usage.
The successful addition of database query challenges significantly expanded Codevue's market reach. Additionally, feedback from candidates who utilized the new IDE has been overwhelmingly positive. They find the revamped interface to be more intuitive and helpful in completing challenges. Features like auto-complete and the timer, conveniently accessible in the header, have notably improved their efficiency and reduced stress levels during the challenges.