Stuart Silverstein Experience Strategist/Designer

back to all articles

Starting a New Research Program

Recently, I was faced with a bit of a new challenge: establish new research procedures and set up a research department. While I had done projects for companies like this with my agency, what made this unique was the fact that this was not a single, unique project… this was a big hairy problem that required several phases, and a lot of prioritization. The company had very little in terms of analytics, and had done almost no user research, save a few unmoderated usability tests. However, the product team knew that they had to begin to focus on User Centered Design. The product team consisted seasoned veterans who knew the benefit of doing user research, and they needed my help in procedural expertise combined with my experience in doing user research to help them make the transition from Executive centered design to user centered design.

Add to this the fact that the platform now was on multiple devices, and an undefined experience strategy and you now have a tangled web to untangle. There were so many issues in so many directions to uncover, and the company needed answers at an impossible speed. Every area possible of research needed help: Usability, user strategy, analytics, industry research…. and it needed to be done six months ago. The business was a sort of start up in the fact that it didn’t know exactly who the users were, and more than a startup because they had some very sophisticated people who had created 2 versions of the application…. without much user feedback…. so the design team and the product team desperately wanted some answers. It was very hard for them to effectively design without knowing who they were designing for and what their needs were, so they hired me to help them start; first by setting up an infrastructure, then creating a program for testing.

Starting a new research program

While the above situation is but one we encounter, there are many other situations you may find that you need to set up a research program. You may be a UX team of one, where the company just found the need or the funds to hire a UX designer, and you will be expected to validate, as well as design, the products you create. You may have inherited or started working with a team of UX designers that once reported to a creative director, and because there was no UX leadership their was neither the desire or the need to create a UX research program. You may also be working in a creative focused environment, such as an ad agency, that simply focused on creating a great concept first, then the UX designers are expected to do the interaction design with minimal research because the project team didn’t build in the time or the budget for any type of research.

In all of the above situations, it is a challenge to create truly meaningful products as a UX designer without the benefit of some sort of research. You can just continue to crank out wireframes based on the ideas of the creative team or executives. You can just put your ideas out into the world and not know anything about how effective what you did was. You can just accept the fact that, in some organizations, UX design is just not valued (even organizations that are digital products!)

You have the choice… but you also have the choice to start to do something about it. If you can start to show the value of the research, prove that the work it yields is valuable and worth the investment, you can get more resources allotted to research, which in turn will improve the process, and yield more meaningful products.

Even if you do not work on a “digital product,” starting some research into your process on marketing or informational sites will help your team be more effective in making design decisions. It will help you fix easy problems (low hanging fruit), and gain momentum for those projects that you know will impact the overall experience, and thus improve performance of the site. Also, you will discover new areas of exploration that can lead to new products or marketing campaigns. Research can be as much inspiration as it is verification.

So where do you start?

If you are starting in a company with no history of research or User-Centered design, you will have a ton of questions, and little concrete confirmed answers. You’ll want to start with the basics – infrastructure, current status, and user definition. This will most likely not be in one study/project, so you will need to break all of this into bite size pieces and assemble once you have the chance, The below list are good places to start your audit when first start to develop your research process. You can do any or some of the below, however you will need to do all of the following to really start to flesh out your strategy.

  1. Audit usability of current applications: You can do an audit of current applications or website(s). I find a heuristic evaluation of the apps can be done relatively quickly and will give you a good sense of where the apps are at. You can make notes of any questions that you have that pertain to the core functionality of the site. I usually mark down each app, and the questionable items.In addition, you can also do a run-through of the core tasks of the site. Collect all the data necessary to create a list of the main tasks on the list, and go through them one by one. Keep notes of the experience, and any issues you encounter during your run through.
  2. Audit of projects in the pipeline. Talk with the product managers and see what is currently going on, and where each of the projects are at. You may find that a product or new piece of functionality may still be early enough to get a quick round of testing in. to get feedback back and incorporated. You may also find a product that is just about to be released that you can do Consumer Acceptance Testing on. Even if you cannot do anything about these current projects due to release timing, see when they are going to be at a point that you can do some kind of testing with them. This will allow you to see if there are any major usability issues and/technical bugs that might need to be addressed in the first revision. You might find some mission critical issues that need fixing, and you can get those on the backlog.
  3. Audit of software, tools and data available. Find out any of the tools the company already has at its disposal. Some of the tools might be usable, some might not, and some you will have to make do with even though they are a less than optimal for providing insight. You should also talk to all departments if possible – especially Business Intelligence and Marketing as they might have tools they are using for surveys and analytics that are already in place. The more you can piggyback your efforts on the heels of the other departments the more complete your data set will be, and the better the insight. You may also want to ask them about any industry data they have available, such as individual reports or any subscriptions the company has to reports by Nielsen or Forrester, etc.
  4. Get a list of login credentials you might need. Once you have a list of the apps, create a document or wiki entry that has a list of all the tools, and their logins. You’ll need this handy when you need to start doing tests. You’ll want to distribute credentials to other UX and product team members if applicable. A few tools to look for are:
    • Survey distribution – such as SurveyMonkey, Foresee, Qualtrics Research suite, etc
    • User recruitment/recruiters – such as ethn.io or Qualtrics Site Intercept. You may want to supplement that by having a local market research recruiter. Some survey or remote usability testing software company will provide users for an additional cost, so keep that in your back pocket as well. I prefer to have a mix of site intercept, recruiter and email list so you don’t exhaust any resource, but you will want to make sure you have at lease one of those three available at a moments notice.
    • A/B or Multivariate Testing – Such as Google Website Optimizer (included in Google Analytics), Omniture Test and Target, or Optimize.ly.
    • Prototype/Flat artwork click testing– such as VerifyApp/SolidifyApp, Usabilla, Silverback, Loop11, or Chalkmark. These will be good for quick prototype and concept testing.
    • Unmoderated remote usability testing – such as UserTesting.com or UserZoom. Some will provide users, and some will require you send users from your own list, or a recruiter.
    • Moderated online usability testing – Online meeting/screen sharing applications for online moderated usability testing, such as join.me, Skype or Google hangouts. Don’t forget, you’ll need a way to record the screen session and audio, so that will need to be part of your plan.
    • Video cameras and recording software —  desktop, mobile and any additional devices you’re going to test such as TV’s, GPS units, etc.-I like to use Morae, or Telestream. For cameras, I like the Logitech C920 and the Elmo Visual presenter for mobile. I have seen the MOD 1000, which I think looks pretty cool as well.
    • Analytics packages – including things like scroll analytics and heat mapping – A few of my favorites (in addition to the old standbys Google Analytics and Adobe Analytics) are KISSMetrics and ClickTale.
    • Online card sorting/Taxonomy testing – such as Treejack
    • Voice of the customer – Customer feedback apps are great place to find unsolicited feedback. Apps like Get Satisfaction, Opinion Lab and Kampyle are great for gathering open ended feedback. Some even give metrics and word cloud analysis to better understand the open ended responses.
  5. Review analytics – Results usually speak the loudest in organizations that do not have much UX research… especially business results. Make it your job to become intimately knowledgeable on all engagement metrics (pages per visit, time on site, return visitors, visitors with time on site over 10 min), and conversion metrics (sales, acquisitions, retention and any other funnel metrics). Before making any case for research, start here, and articulate to stakeholders how the data can be interpreted multiple ways, but, without the UX research to back it up, you are just making supposition. For instance, you may see low conversion rates, but a very high Net promoter score, which proves contradictory. That means the people who purchase are happy once they convert, but why are they not converting? Is it a technical issue? Is it a pricing or value issue? Is it a usability issue? The qualitative will help uncover this but the analytics has to lead the conversation.
  6. Rooms available/space for in person usability testing – At some point you’ll need to do some in-person usability testing, Inquire with an office manager or secretary who manages the meeting rooms. Do an inquiry as to what rooms you can block out for a day and keep things set up. Also, find out how far in advance do you need to book those if you were able to run a test.You want to also assess the feasibility of using that room. If it is dark, and cold, and has terrible lighting, you may have a hard time making users comfortable. If the room is less than desirable, figure out what it might need to make it more comfortable. It may be a lamp and a sofa, which will cost all of a few hundred bucks, and its all set up for you to use whenever. Plus, You will need to find a place for viewers. It may be that you broadcast sessions to stakeholder’s cubes and then reconvene in the session room afterwards, but a best case scenario is to broadcast to a room for all to view. The more you can make it a spectacle, the more people will come. The more people who come, the better the company will know the users, and the better decisions the company will make. It will start to become part of the culture. This starts with finding a place for everybody to gather during sessions, and promoting that people sit in and listen.
  7. Interview customer service. The single best source for any user information in a company not currently doing research is customer service. They are the eyes and ears of the company, and are the one who has the best representation of the voice of the user. In the CSR reports is a goldmine of User research. Usability problems? Check? Product Features? Check? Value Prop? Check? All of those can be seen in CSR reports as, well as interviews with CSRs. You may need to create a custom dashboard within the tools used for customer service reporting, or you may need to get your hands on the reports that are issued to the company by the business intelligence team. In any case, you want to look for the classification of tickets, and the details on those tickets, and try to group them into issues to investigate.You will most likely see a lot of technical issues – such as a lot of refunds because a product failed to transact or deliver the merchandise—such as a download. These affect UX metrics, and are all a part of improving the experience. You will also see marketing failures—whereby users did not know what they were getting, ,how the service works or that they thought they were getting something different than they received. You will also see usability issues, such as users calling to complete a transaction because they didn’t know how to do it online.Another thing to do is sit in on several sessions of customer service phone calls. That will help you really hear the voice of the customer, and hear real stories of how the product is not meeting its objectives. It is also great for helping to develop empathy with consumers who are using your product.
  8. Benchmark study: A benchmark study, no matter how small/big, should always be included as a part of the start of any program. You have to know where the product is in order to make effective improvements. Some of your initial research you will need will be collected during the investigation phase (such as customer service reports and analytics). Some will need to be added on – such as a benchmark usability test. A Benchmark can be done in pieces – so you may start by adding a few extra tasks to your remote studies, or add an ethnographic interview to your in person studies. If you start collecting a little bit of data in your other studies, you will find yourself with the data to conduct a benchmark study at the end of a few months. A benchmark should include:
    • Usability assessment of current platforms – heuristic and user testing
    • Customer service and analytics reports
    • Demographic and ethnographic study information
    • Quantitative research on attitudes and behavior
    • User Journey map (optional)
    • Information on industry trends (optional)

Creating a strategy:

Once the audit is done, you will start to see gaps in knowledge that need to be addressed in order for Product and leadership teams to make better more informed and less risky decisions. Once you have compiled a list of questions or gaps in knowledge that the company has as a whole, it is time to collect those into a spreadsheet, and categorize them into one of the five areas.

Five areas of concentration:

  • Improve Current products – How do we make our current products better and easier to use?
  • Assess work in the pipeline – How do we know if the ideas we are currently working on are going to work tactically and if these pieces of functionality are useful to users?
  • Understand the user/business opportunity – Who is our user? How do they behave? What are their needs, and how are we meeting their needs for a profit?
  • Put the right features on the backlog – Once we know who the user is, does the backlog help us to create a complete experience to engage them? Have we put the right stuff on the backlog that will matter to the company bottom line.
  • Look for new ideas and business opportunities– Once we know our user, are there other opportunities just outside of our field of vision that we can create to engage the user?
  • Infrastructure and procedural implementation- Once we have gotten our research established, how do we make it easy for us to continue researching by having the right tools and procedures at our fingertips? How do we evangelize this throughout the company?

Before we get into the prioritization of the research program, I am first going to assume that you have done research and are familiar with most standard methods of research (usability testing, focus groups, card sorts, ethnographic research, etc). There are many great books on the various methods, so I will assume you have done at least a few of the above, or at least know how to start to do any of the above methods, and which method is best for the information you need.

Prioritization party:

In any sort of predicament where you have way too much stuff to get done in the amount of time you have to get it done, the first thing is to prioritize. We needed to make sure current products were usable, we had to test work in progress and we hadn’t defined exactly who the user was. This created a lot of chaos, so the product team needed to have some sort of information on which to base its decisions. I was working on some interaction design projects in addition to the research and found it difficult to design when the target was not clearly defined. In my experience, in order to hit a target I met first have it defined. You must know who your user is, so that you can make all the micro interactions work, and make the interface intuitive. However, when you have a broad audience it is all to easy to justify any decision because you can come up with a multitude of reasonable use cases that make your solution work. Without a target then, you are just chasing after every shiny object because any shiny object could be a diamond. The product team was also frustrated that they couldn’t get any answers on if their design was getting them closer or further away from who the audience was.

So, once we identified the questions, and the categories, we held a prioritization party. We got the design team in a room and gave them 3 points per question on the board, and they had to force rank everything on a scale of 1 to 5. That means if you have 10 questions, you have 30 points to distribute. Do this same exercise with the product management and leadership (if you can). Make it a party! Buy some beer and pizza and have a good time with it. Plant a seed with those teams that with a little bit of extra time and money, you can all make more informed decisions, coming to better results in the end.

Priority strategy:

Put together a focus plan based on the results of the exercises — i.e. 20% improve current products, 50% Assess products in the pipeline, 20% Understand the user. Once you have this use this as a guide for planning subsequent studies. Remind the leadership that this fulfills the strategy agreed upon by the team and will help the company make better UX decisions. To that effect, start to see if you can incorporate tests to find some of that information on new projects.

Execution of the plan

After the audit and prioritization is done, it’s now time to start getting some projects in order to gather your data. The first place to start is with senior management to get the questions that they want answered first. This will make getting research approved easier. It will also get you credibility the fastest. They will also be your allies in distributing information to the other departments in the company. The best way I have found to start the conversation with them is to ask them, “If you could ask a genie to come and tell you anything about your customers, or the website(s) or application(s), what would that be?” Take very careful notes about these questions and try to answer as many as you can explicitly in the first presentation.

Infrastructure first:

As you are starting to add tests and research to your work, the first thing you must prioritize is the establishment of key infrastructure items. You will need to have tools in place before you can execute any test, so you will need to make sure to work on that before executing any new tests. Your remote testing tools, survey tools and recruiting are the most important, so get those up first before working on in house usability equipment. It’s a lot easier to get some quick wins remotely before asking for several thousand bucks, and a room blocked out for a week.

Another item at this point is to start working on getting any tools you might need integrated into to site on the backlog. This may be analytics packages, or recruitment packages. These can take a while to get implemented, so get the info on the software, talk it over with tech/finance, and get it on the log so you have it available when you need it. Interrupter software (like ethn.io) are crucial, as they will be the future lifeblood of the research program, providing the most accurate user, so make sure to get that into the queue as soon as you can.

You will also need a recruitment email, so start writing copy, gathering a design template, and talking with the email marketing crew about how you can harvest users from that list. Get it ready. You will need it to execute the plan, so do it in advance when you have some down time,.

Execute in small increments

So first thing when starting out, it is important to put together a list of possible projects that you feel will start to improve the quality of the product. To start, pick the top 2 categories that arose from your prioritization party. and execute one test from each category and efficiently as possible. You may be able to piggyback a few infrastructure items on these first tests, which will be good. The idea is to set up a few things little by little, and after a while you will have an entire infrastructure built. Things will go very slowly at first, so allow extra time for you to create plans, test, integrate and iterate on the internal process. Prepare for your plans to change and you will be derailed on to another task. Put aside a small amount of time per week to work on it. As soon as you have some down time, pick it up again. It can take a long time to get to where you want to be if you have a lot of design work to get done.

Start with easy wins:

In order to get some momentum going, you will want to start with the lowest hanging fruit, these can be things like demographics or usability studies, or analytics reviews to see which content is the most viewed, or a flat artwork test on a comp of a current prototype. Since most people believe in quantitative vs qualitative, the first test might be quantitative – such as a survey or flat artwork – recruited by email. While not the best representation of users, most companies can support an email blast to users with a link to a quantitative test. Use the initial quantitative studies to start the conversation, explaining that it is only one side of the equation, and some qualitative will help you better understand your results.

Another easy quick win can be remote unmoderated user testing – such as user testing – on a new feature to be released. This can help you start to see the value of feedback, and you can start to get some qualitative feedback at the same time. Furthermore, you can use this to see how well your screener behaves and start to fine tune it. If you have a good screener, (and your company is not too specialized in its audience) you will get some good users and some good feedback. Usertesting.com also has really easy exports, so you can add some video to your presentations.

Next Steps

Once you have completed a few studies, the below is a list of suggestions for you to implement to make sure the research gets traction, and can make a lasting impact on the business.

Publicize results as much as possible: The research is of little use if it doesn’t make it past the UX team. The place to get buy in is from Product and design leaders in the organization, as they will see the most value, and be able to help you figure out how to publicize the results. Also, it is a great idea to invite the entire product and/or dev team to the review of the results. That way other team members can learn as well, and make better decisions.

Follow up with product teams :

After you have finished a round of research, go out and discuses the implications. The is best done casually over a beer or two where you don’t have the stress of the clock, and a conference room reservation that ends in an hour. Discuss the problems users are having, and what the current road map looks like. Are you addressing any major problems? What can you do together to make sure that the bigger UX problems get addressed as soon as possible.

Find out what else keeps your boss/client up at night

Take the boss of the product owner out to lunch and listen to what keeps him up at night. All C-level/Directors have burning questions that they want answered before moving confidently in a direction. Most are willing to take calculated risks, but if you can inform them how a user views the product, you might find that issues get fixed faster. I’ve found that if you find out what the biggest question is, and design a study to answer that question, that you can get approval for more research projects (provided you do a good job of proving your conclusions). Also, be aware, that if you are going to take on a project that keeps your bosses up, quantitative will always win over qualitative with C-level, so make sure you have the quantitative tools in place first to initiate those conversations. Also, speed + accuracy = respect, so if you can do these studies in the fastest yet accurate ways possible, you will be better positioned in the future to do more research.

Discuss some hypotheses

Once you have a few studies, you can start to create hypotheses on why you are getting the results you are. You hopefully will have some user interviews, some quantitative, industry research, etc, so start to create some hypotheses to explain the data. Unless you have enough data to back it up, this can be something as simple as a Word doc with some notes explaining who the users are, and why they do what they do. Once you have these, bring them up and the next UX/design team meeting to see what others observations are, and see what tests you can do to prove the hypotheses, or collect data to prove or disprove your theories.

Create straw-man personas, scenarios, etc and then start to refine over time:

As part of your hypotheses, you should definitely take a stab at the “who” in the UX equation. With some basic research, you can take a stab at some straw man personas (personas that are rough, and maybe lack some detail). Over the next few months, flush those out by adding contextual inquiry to your usability tests, or adding questions to unmoderated tests, etc. Here are a few exercises:

– Put together a list of questions that you need answered in order to confidently find out “who” is using your product. Then see if some of the other more important questions (especially the ones that keep your bosses up at night) can be combined with your persona questions to create a study. Most C-level people don’t care about personas and UX strategy, so you may need to create these for the UX team at first.

– Have a persona party with the product team, where you discuss the audience segments, and tasks. Try to develop a UX team view on the personas, and create a straw man for circulation amongst the team.

A final thought

Research is key in gathering consensus and collecting information to create truly meaningful design. However it is the first thing to get cut when time and resources are short. A lack of research can lead you to make incorrect assumptions that have minor impact or it can lead you to create products that are entirely wrong, or lead you to chase bad business opportunities. Creation of a research process can help the organization stay in touch with users, and stay on track and focused building meaningful products and websites. It is also the “proof” you need to help guide the product in the right direction, and informing team members and product owners.

Resources:

Below is a list of my go-to software for creating a UX research program:

  • Natasha Terpstra

    This is such a great article – thank you!