Behind the Scenes: Insights from User Testing the new DMP Tool Designs

TL;DR

• The rebuild of the technology behind the DMP Tool offered a chance to refresh the user interface
• We conducted 12 user testing sessions to have real users walk through wireframes of our new tool designs to offer feedback and find issues
• People liked the new designs but had a lot of small areas of confusion around some aspects like sharing and visibility settings
• We made hundreds of changes to our designs based on feedback and continue to make updates for better usability in the new tool
• Fill out this short form to have the option to join future feedback sessions

Why we needed new designs

As mentioned in our last bog post, the team behind the DMP Tool has been working on an rebuild of the application to improve usability, add new features, and provide additional machine-actionable features.  To provide all of this advanced functionality, we need to do a pretty big overhaul of the technology behind the DMP Tool, and it was a good time to give the design a more modern upgrade as well, adding new functionality while hopefully making existing features easier to use.

How we made the first drafts and tested them

Over the past few months, we’ve worked closely with a team of designers to create wireframes—interactive prototypes that allow us to test potential updates to the user interface without fully developing them. These wireframes are crucial for gathering feedback from real users early, ensuring that our vision for a better tool meets their expectations.  While a lot of thought and planning went into these initial designs, we wanted to make sure people were finding the new site as easy and intuitive as possible, while still offering new, more intricate features.

To do this, we recruited three groups of people, 12 total, who work on different parts of the tool to test out these designs:

• 5 researchers, who would be writing DMPs in the tool
• 4 organizational administrators, who would be adding guidance to template in the tool
• 3 members of the editorial board or funder representatives, who would be creating templates in the tool

We recruited volunteers from the pilot project members, from our editorial board, from social media, and from asking those we recruited to share the invitation with others. We conducted virtual interviews with each person individually, where we let them explore the wireframe for their section, gave them tasks to complete (e.g. “Share this DMP with someone else”), and asked questions about their experience.  For the most part we let people walk through the wireframes as if they were using it for real, thinking out loud about what they were experiencing and expecting.

What we found from testing

It was illuminating for the team to see live user reactions from these sessions, and watch them use this new tool we’re excited to continue work on. 

We loved to hear users say how excited they were for a particular new feature or how much they liked a new page style.  At times it could be disheartening, watching a user not find something that we thought was accessible, but those findings are even more important because it means we have an area to improve.  We made a report about the findings after each group of users and worked with the designers on how to address the pain points.  Sometimes the solution was straightforward, while other times we wrestled with different options for weeks after testing.

Overall, we found that people liked the new designs and layout and could get through most tasks successfully.  They appreciated the more modern layout and additional options. But there were many areas that the testers identified as confusing or unclear.  There are many specific examples, with before-and-after screenshots, in the Appendix.  Some of the top changes made revolved around the following areas:

• Decreasing some text in areas that felt overwhelming, moving less important information to other pages or collapsed by default
• Adding some text to areas that were particularly unclear, such as what selecting “Tags” for a template question would do
• Connecting pages if people consistently went somewhere else, such as adding a link to sharing settings on the Project Members page since that’s where people looked for it first
• Moving some features to not show until they’re needed, such as having Visibility settings as an option in the publishing step and not the drafting step
• Clarifying language throughout when things were unclear, such as distinguishing whether “Question Requirements” was about what the plan writer was required to write when creating their DMP or whether that was about the template creator marking whether a question is required or had display logic
• Having additional preview options when creating a template or adding guidance to understand what a question or section would look like to a user writing a DMP
• Making certain buttons more prominent if they were the primary action on a page, like downloading a completed DMP that originally was hard to find

Even though the main structure worked well for people, these small issues would have added up to a lot more confusion and obstacles for users if we hadn’t identified them before releasing.  

Wrapping up and moving forward

The whole team learned a ton from these sessions, and we’re grateful to all the participants who signed up and gave their time to help us improve the tool.  This sort of testing was invaluable to find areas to improve – we made dozens, if not hundreds, of small and large changes to the wireframes based on this testing, and we hope it’s now much better than it was originally. We’re still working on updates as we build our designs for more areas of the site, but feel better now about our core functionality.

If you’d like to be invited to participate in surveys, interviews, or other feedback opportunities like this for the DMP Tool, please fill out this brief form here: Feedback Panel Sign-Up. For anyone that signed up but wasn’t selected for this round, we may reach out in the future! 

We loved seeing how excited people are about this update, and we can’t wait to share more.  The most common question we get is – when is it releasing!  That’s going to be quite some time, and we don’t have more to share yet, as we’re still too early in the development process.  But stay tuned here for more updates as we do! 

We want to thank Chan Zuckerberg Initiative (CZI) for their generous support for rearchitecting our platform. We wouldn’t be able to make all of these helpful updates along with our back-end transformations without it.

Appendix: Specific Examples

Important note: Even the “updated wireframes” shown here are not final designs. We have not yet completed a design pass for things like fonts, colors, spacing, and accessibility; this is just a quick functionality prototype so we could get feedback early. Even the functionality shown here may change as we develop based on additional feedback, technical challenges, or other issues identified.

Sharing settings

For those who want some more details and specific examples, here are a few of the top areas of confusion we found:

There was sometimes confusion in how to share a plan with others, and what the distinction is between a Project collaborator (e.g. another researcher on the grant who may not be involved in the DMP) and a DMP collaborator (e.g. a peer who is giving feedback on writing the DMP but not on the project).  The current live tool has both “Project Members” and “DMP Collaborators” on the same page which we thought contributed to this confusion, so we wanted to separate those who can edit the DMP into a separate Sharing section.  However, testers had a hard time finding these sharing settings, and often went to the Project Members page to grant DMP access.  So, we added a link to these settings where people were looking (the new section in the green box), and added more detail to the sharing page about whether they were invited or had access due to being a collaborator, changed some language within this like “Collaborator” to “Project Member,” with the option to change access.

Current tool:

On the current tool, these two types of collaborators are on one page

Initial wireframes:

The Collaborators page in the new wireframes, which was part of the overall project details.A separate page on the plan itself had sharing settings, and was completely distinct from collaborators

Updated wireframes:

Since people often looked for sharing settings on this page, we added a link from here, and changed the name to the more clear Project MembersThis page was updated to give more information and control on invitations, and to make clear if people were added on because of an invite or because they were a project collaborator

Card layout

Many parts of the tool used a new, more modern Card format for displaying lists of items to choose from.  This allowed us to show more information than in a list, and adapt to smaller screens. However, we saw in some areas that people had trouble scanning these cards to find what they were looking for, like a plan or template, when they expected to search in alphabetical order.

For example, picking a template in the first draft used a boxier card format. People found it harder to find the template they were looking for, since they wanted to quickly scan the titles vertically.  So we changed it to a different format that should be easier to scan, even if it doesn’t show as many on one page.  Note we also now have the option to pick a template other than from your funder, a common request in the current tool.  

Current tool:

Currently, selecting your funder brings up a list of templates with no other information, and you can’t select a different template

Initial wireframe:

This format allows more information if we want to add details that might help people pick the right template

Updated wireframe:

This update still allows us to show more information, but the vertical layout means a person’s eyes can move in the same spot down the list to scan titles more easily if they know what they want

Flow through the tool

People appreciated that they could move around more freely in the new design, as compared to the more linear format of the current tool. However, that also occasionally made people feel “lost” as to where they were in the process of writing a DMP.  So we added more guidance, breadcrumps, and navigation while still allowing the freedom of movement throughout the process.

For example, while writing a plan, users will now be able to see the other sections available and understand where they are in the Project tree.  We also reduced some of the text on screen due to people feeling overwhelmed with information, putting some best practices behind links that people can visit if they wish to, and moved the Sample Answer people were most interested in to above the text box for better visibility.

Current tool:

The current tool has more distinct phases from writing a plan to publishing. In this view, a person is answering a single question and then would move on to the next.

Initial wireframe:

In our first draft, people clicked into each question rather than having all one one expandable page. But people weren’t always sure where they were in the process or how to get back.

Updated wireframe:

We added the navigation seen on the left and top here to allow people to see what else is in the plan and more easily get to other sections. We are also still working on how to reduce how much text is on the screen at once, for example by minimizing the guidance, but this is not final. We also moved the sample text above the question and removed the answer library for now.

Layout changes

In addition, there were tons of small changes throughout, changing layouts, wordings, and ordering of options in response to areas of confusion.  Some places we scaled back a bit of functionality since the number of new options were overwhelming, while other places we added a bit more that people needed.

In the first draft of the wireframes, the visibility settings of the plan were on the main overview page of the plan.  This was concerning to users since they were still drafting at this stage, and even if they may want it public once they published it, the setting in this location made it seem like it was public now.  Instead we added a status and setting on the overview page, but the visibility setting does come up until a person gets to the Publish step, somewhat like the current tool that has those options later than in the plan writing stage.

Current tool:

Currently, setting visibility is later in the “Finalize” stage

Initial wireframe:

In the first draft, this visibility settings were on the main plan page, which made people think it was public already as opposed to that it would be public once published.

Updated wireframe:

The updated main page, with many changes based on feedback, including visibility as a status on the right, which isn’t set until it is publishedNow, visibility is set only once a person goes to publish their DMP.

We made similar change to creating a template, moving the visibility settings to be selected in the publishing stage instead of being in a Template Options menu people didn’t always see right away.  They expected to set that visibility at the time they published it, so that’s where we moved that option to be, consistent with how the plan creation flow works.