Bad Research Got You Down?

Good research does not guarantee good decisions, but it certainly helps.
And bad research, barring getting lucky and guessing right, almost inevitably leads to bad decisions. 

There’s a lot of bad research out there! Throw in the fact that most of us lack formal training in research methods, and you’re behind even before you get started.

In Caveat Emptor: Becoming a Responsible Consumer of Research, co-authors Elizabeth Weaver Engel, M.A., CAE, Spark Consulting, LLC, and Polly Karpowicz, CAE, set out to help association executives become better consumers and sponsors of research. The whitepaper gives us a primer on key research concepts and methods (including the flaws and fallacies) while offering solid advice on how to find the best path forward when conducting a research project. We like the connection they make with the important role association execs have in serving as a trusted source of information for their components, as well as helping their volunteers and leaders develop their own research literacy skills.

Since we’re all about components (chapters), we asked Elizabeth and Polly how association execs can use the information to help their components make better data-driven decisions.

Note: Q&A was edited for length. However, since there is such a treasure-trove of information that the authors shared, we’re including a pdf of the full Q&A. Download here if you’d like to read. More importantly … download the full white paper.

MARINER: What drew you to create this whitepaper?

ELIZABETH: Polly and I took on this project because we wanted to help our fellow association execs be more informed about what constitutes sound research so you can better evaluate the research you’re relying on and make better decisions in service to your members and your association’s mission.

Our goal was to create a concise, accessible, practical resource for association execs, volunteers, component leaders, and other stakeholders from associations of all sizes and types.

MARINER: On the list of must-have skills for volunteer leaders, where does research acumen fit in? Perhaps another way of asking is – do volunteer leaders need to have research competency and if so, what might look like?

ELIZABETH:  In some ways, this gets directly to the goal Polly and I had when we set out to write this whitepaper. We’re not arguing that association staff or volunteer leaders need to all become professional researchers or data analysts or the like. Rather, we’re trying to encourage association leaders – paid and volunteer – to become educated consumers, conversant in the processes and methodologies that constitute quality research, and to provide a primer on what they need to know and the skills they need to develop to do that.

MARINER: How can association execs use your paper to guide their component leaders in a) using the research the association provides, and b) conducting their own research specific to their communities?

POLLY: Let’s start with the second part. Association execs should not require our components to do their own original research, given the importance of and challenges inherent in conducting research responsibly. In the paper, we outline some of the ways associations can get into trouble conducting original research, and it’s complex, with significant potential for serious consequences and missed opportunities.

That said, there are significant opportunities for collaborating with component leaders in your research activities. This kind of collaboration can result in deeper insight when you leverage the core assets of the association and the component – the lead association’s capacity, reach, and resources to execute reliable research and the component’s direct link to people with specialized perspectives and knowledge related to the research questions.

We absolutely should encourage component leaders to use research conducted by the association. Component leaders may encounter barriers related to awareness, access, or ability to use the association’s research. You may need to reach out to component leaders several times to build awareness and provide specific training on how to access and put the association’s research to use in their affiliates.  Association execs should ask these questions and address any issues that may be discovered:

  • Are component leaders aware of the association’s research?
  • Can they access the research? Are there any barriers to access we can reduce or eliminate?
  • Do they need technical or conceptual help applying the research to the component?
  • Does the research provide useful insight for component leaders?

MARINER: National associations preach to their components to make data-driven decisions and to ask members before making decisions. Yet research is often outside the resource scope for components – what options might you offer a small component? How might they tap into informal research or passive data gathering options?

ELIZABETH: You’re exactly right. Not always, of course, but in many cases, our regional, state, or local components are run by small teams of volunteers who also, you know, have day jobs in the profession or industry the association serves. They lack the time to conduct original research or to even acquire the skills they would need in order to do so.

This is a perfect example of a service the national office should provide for your components, both offering on-demand data analytics (where you can also easily help them benchmark against themselves, against other chapters, and/or against your national averages) and incorporating their perspectives in your national-level research projects.

I also want to highlight that last bit – incorporating their perspectives into your research projects – as it is critical and often overlooked. Your chapter leaders are closer to your members than your national staff. They also have a unique perspective, being both staff-adjacent and “end users” of the association’s programs, products, and services. If you work with them up front, you will start from a place of deeper knowledge and as a result will ask better questions that will provide more insight into what’s really going on with your members and other stakeholders.

MARINER: What would you say is the number one aspect (positive or negative) of data collection that readers should consider when in the planning stages of a research project?

POLLY: The most important thing you should consider is bias that can slip into your research. Whether intentional or unintentional, bias can come from anyone involved in the research and creep in any point in the research cycle. The problem with bias is that lowers our confidence in the validity and reliability of our research, making it difficult (or impossible) for any clear or valid conclusions to be drawn from the results.  No one wants this!

You should become familiar with the kinds of bias that can happen and the techniques and practices that researchers use to minimize research bias before you start any data collection (note: it’s impossible to avoid bias entirely) and understand the extent and effect of any bias that may have inevitably slipped into your research. In the whitepaper, we review the various kinds of bias, where it crops up, and what you can do about it.

MARINER: Any final advice you’d like to share?

ELIZABETH: Be skeptical. Seek out opportunities to improve your information literacy skills. Read the methods section. If you have the opportunity, ask questions directly of the study’s authors. Researchers, including outside vendors your association might choose to work with, who are doing quality work will be happy to answer them and will be honest with you about the limits of their program or project design.

POLLY: Know the basics and be aware of specific concerns for associations. Take a look at our interviews with Jeff Tenenbaum on avoiding antitrust liability in association research projects and with Dr. Sharon Moss on responsible conduct of association research.

Following on Elizabeth’s last tip, when you ask for clarity, be persistent if your question about research findings isn’t answered the first time. Get help from the research experts in your organization and in the association sector. Find out what research sources your members, stakeholders, and component leaders use. Where do they go for answers to their research questions?

Download the paper today!

*Check out these additional resources: