Project Overview
Military fighter pilots need access to vital information quickly – traditionally this information comes from a literal stack of documents on their kneeboard, a clipboard strapped to their leg.
We worked with the US Navy to conduct UX research into the human-factor challenges of digitizing a paper kneeboard and partnered with a military contractor, SoarTech, to create a proof-of-concept demo.
My Contributions
As the design lead for Phase 2 of this project, I took the human factors research gathered from Phase 1 and worked closely with stakeholders to create a native prototype on the iPad. I also led design discovery workshops to validate our assumptions and understand the unique working environment of fighter pilots.
Impact
After 4 months of rapid development, the final prototype was successfully demoed at a nationwide conference for military technology in Florida in late 2019. The project gained additional stakeholder buy-in for continued research and development.
Could the Navy pilot kneeboard be more intuitive?
The US Navy is currently experimenting with electronic flight bags and digitized kneeboard applications to lighten both the physical and mental load of dealing with a paper kneeboard.There are many apps on the market that address a singular aspect of a kneeboard, like navigation or weather or airport documents. Yet, only a few have successfully consolidated those features together, and none have tailored these features to Navy F-18 fighter pilots.
Questions We Tackled
Overall, how do we make it easier for Navy pilots to get the information they need while engaged in critical tasks of flying an F-18 jet?
- How do we elegantly consolidate the multiple functions of a pilot kneeboard into one digital product?
- How can we help Navy fighter pilots anticipate problems while in the air and quickly find a solution?
- How can we use AI to assist in the pilot’s decision-making?
Our Approach
Design-Thinking Discovery
Created a narrative for a stakeholder demo to focus the scope and effort for greatest value.
User Centered Design with Native iOS
Design which focused on the pilot, combined with expertise in designing for an optimal iPad tablet experience.
Our North Star
In Phase 2, we partnered with SoarTech, a military contractor that specializes in artificial intelligence. In the discovery portion of the project, I began by identifying and understanding the problem. By clearly outlining the problem, I could then use that as our North Star to guide our efforts to solving that problem.
Sticky notes from the problem statement exercise
Final Problem Statement:
The current state of pilot kneeboards has focused primarily on digitizing flight data (formerly on paper) via multiple apps/resources.
Project Patella will address this gap by making kneeboard docs easily accessible and integrated for fighter pilots.
Our initial focus will be making a demo-able proof of concept of flight-driven AI.
Journey Mapping
Because this application will be built for a highly specific and nuanced environment, it was vital that we had an in-depth understanding of the user's (in this case, an f-18 fighter pilot's) experience throughout the entire process. To do so, I sat down with a former navy fighter pilot in front of a whiteboard and asked him to walk me through every step of a typical mission, from pre-flight to debrief. I grouped the steps into phases and highlighted key events with storyboard illustrations. The goal of this was to figure out at what points our AI-driven application could assist the pilot during a mission without interrupting or delaying the pilot's actions.
Feature Prioritization
We then extracted pain points and opportunities from the journey map and converted them into potential features for our kneeboard app. To align on the priority of these features, I led a 2x2 feature prioritization exercise with our stakeholders and product team in which we mapped each feature on a matrix based on its potential and the degree to which it could relieve the pain points of a pilot.
Initial Prototypes and Design Tradeoffs
The prototypes below were created based on user research from Phase 1. The initial prototypes accounted for user needs and pain points, but the designs were not platform specific. For instance, we prototyped a night mode setting for the kneeboard UI, since many pilots stated that they usually wore night vision googles for night missions, and bright screens would be difficult to read with the googles on.
These prototypes were eventually scrapped because a lot of the components were not iOS standard, and a requirement for Phase 2 was that the app had to run on iOS. Since Dark Mode for iOS was still a new feature at the time, and based on the scope of the project, we did not prioritize a night mode for the final proof of concept. However, some aspects of the UI were kept for the final product, such as the sidebar navigation, a format that is familiar to pilots as it mimics the controls in a fighter jet with the buttons located next to the screens in the cockpit.
Style Guide
A style guide was create in conjunction with the interactive prototype to ensure that the interface would adhere to the iOS Human Interface Guidelines. Components contained specs and details of their use cases.
Challenges
- R&D vs Agile: Remaining flexible despite a difference between our (Agile) development process and SoarTech's research and development process. We gave our partners at SoarTech a taste of our Agile process through integrating them into our sprint ceremonies and design/development cadence
- AI-driven design became Design-driven AI: We had to design and build the iOS architecture/app first without having proper backend dependencies from the client. This required an in-depth understanding of the user's experience and full engagement and testing with our fighter pilot users every step of the way to ensure that the AI would fit seamlessly into their workflow.
- Focus: With limited time to execute a proof-of-concept app, we had to narrow our scope. The goal was to create a demo-able prototype to showcase to stakeholders, so I worked with our team to make sure that every effort we made was toward executing that goal. I also developed a demo script based on the storyboard that would highlight the most useful aspects of the AI-integrated app.
- From Paper to Digital: Fighter pilots had access to all kinds of complex charts and manuals in case of emergencies. There are charts for every kind of emergency out there and it can be frustrating having to leaf through a stack of papers to get the directions they need to handle a specific contingency. When we digitized those charts for the app, we had to ensure that the user interact and inputs were at a minimum in order to maximize the efficiency of the UX.
(Far left) a bingo chart helps the pilot calculate fuel, altitude, and speed require for descent.
The digitized version made the chart much easier to read by only showing the data that was relevant to the pilot's current flight conditions.
Final Prototype
Here is a video demo of the final prototype showcasing the AI assisting with two types of emergencies - a landing gear error and a weather emergency resulting in a reroute to a divert field.
The final concept also included a Unity flight simulator (built by SoarTech) to mimic the conditions inside a cockpit and sent mocked environmental and aircraft data to the application.
.png)
