Can ChatGPT Make B2B Screening Sexy Again?
Business Development Director, Clare McHatton has been exploring how ChatGPT can make the life of a market researcher simpler.
As a market researcher, designing effective screening questions for B2B surveys can be challenging. You must target the right individuals, with the necessary knowledge to provide accurate and insightful responses for your customers. The below are just some reasons this can be easier said than done:
- Job titles and the responsibilities they represent vary widely and change rapidly*
- People change jobs, responsibilities, company size and even industry, a lot.
- People who are genuine and valuable respondents can be, shall we say ‘optimistic’ with their title, depth of knowledge and scope of responsibility.
These issues are not exclusive to online panels either, with specialist B2B panels, or networks recruited via LinkedIn or Dun & Bradstreet, encountering the same difficulties.
This brings me to ChatGPT. This is something that has been occupying the news for what feels like the entirety of 2023 so far! However, in the context of designing screening questions, it has some really useful applications. And anything that can make this part of the research process simpler and more effective is worth talking about, right?
How Can ChatGPT Be Applied to Designing Screening Questions?
ChatGPT goes a step further than we humans have the time – or resources – to do. It has the ability to identify the job titles we should target by analyzing role descriptions and interpreting the degree of knowledge and responsibility.
While it can’t stop humans changing their mind, job, or industry, it can help researchers safely cast a wider net, to screen very specifically and objectively for the level of relevant expertise qualifying respondents need.
Before I wax lyrical about the applications of ChatGPT, I should caveat my thoughts with the following – the free version I used is not quite there with crafting non-leading questions. It kept suggesting yes/no questions, which are definitely a screening no-no.
However, as researchers we can design effective, non-leading screening questions and use ChatGPT to illicit the industry-specific knowledge needed to make these really specific and engaging.
So, what are some quick, simple ways you can use ChatGPT today, to help screen for particular B2B respondents more effectively?
1. Test Knowledge Using Industry-Specific Terminology
For example, looking for Web3 game developers, you could ask, “Which answer most closely matches the definition of Web3?” early on in the screening, to root out both bots and unqualified respondents.
As Web3 is particularly difficult to define, within a few prompts and requests for further brevity (ChatGPT is very verbose), it came up with the makings of a fairly decent knowledge test:
Real definition: Web3 gaming is a new generation of online games that leverage decentralized technologies to offer players greater control over their in-game assets and experiences.
False definition 1: Web3 gaming allows players to physically experience the game using AR and VR technologies, is run by advanced AI, and provides a direct link to the IoT.
False definition 2: Web3 gaming lets players control their in-game characters with their thoughts, uses quantum computing to provide hyper-realistic graphics and physics, and is hosted on a global network of interconnected satellites.
2. Red Herring Questions
Adding made-up brands to an industry-relevant brand recognition test will root out unengaged or ill-informed respondents. Survey takers should not be selecting a brand that doesn’t exist.
For cloud computing experts, ChatGPT came up with the following list of tools:
8 brand names for tools that a cloud computing expert might use:
- Terraform (IaC tool)
- AWS Management Console (CMP)
- Google Cloud Logging (monitoring and logging tool)
- Kubernetes (container orchestration tool)
- AWS Lambda (serverless computing framework)
- Azure Functions (serverless computing framework)
- Ansible (configuration management tool)
- Chef (configuration management tool)
To identify if respondents were genuine, I chose a red herring that was not similar to any of given tools as to be mistaken for one: Kombucha.
I then put this list, with the red herring word, into ChatGPT and asked it to assess the relevance. This was the response:
The following items from the list are relevant to cloud computing:
- Terraform (IaC tool)
- AWS Management Console (CMP)
- Google Cloud Logging (monitoring and logging tool)
- Kubernetes (container orchestration tool)
- AWS Lambda (serverless computing framework)
- Azure Functions (serverless computing framework)
- Ansible (configuration management tool)
- Chef (configuration management tool)
Kombucha, on the other hand, is not relevant to cloud computing. Kombucha is a fermented tea beverage and is not related to any of the technologies or tools listed that are used in cloud computing.
3. Open-ended Questions
In B2B surveys, open-ended questions can offer valuable insights into how knowledgeable people really are, above and beyond whatever job title they’ve been given.
ChatGPT allows us to quickly craft a specific question for the subject matter and then review responses to assess the relevance of what is said, and the language used. As a side note, as AI bots become more sophisticated, this is something we will all need to be doing to make sure our question design ensures authentic responses.
Two easy open-ends to use are, 1) asking a question that requires an opinion:
‘As an AI language model, I don’t have personal opinions or biases, but I can provide some general information’
Or 2) asking for profanities!
‘I’m sorry, but I cannot provide you with a list of swear words or any other inappropriate content. As an AI language model, my purpose is to provide useful and informative responses while maintaining a respectful and professional tone.’
Working with ChatGPT, Not Against It
Ultimately, whether you believe ChatGPT is just hype, or the best thing since sliced bread – it is here to stay. Therefore, we should be identifying ways we can work with it not against it, to incorporate it into our everyday and make certain tasks that bit easier.
As experts in B2B research, we understand the challenges that our customers face. We use our own proprietary technology to help our clients design robust screening to ensure we reach the right respondents and create the best data. Our team are always looking at new tools and technologies we can use to continually deliver delight.
*Is it Personnel, HR, People Opps or Talent Team in your company this month?
**When testing open-ended questions, Richard Bowman author of ‘Prompt for Brands’ found that humans average 7 words when completing an open-end, whereas ChatGPT averages a whopping 221!
Send your quotetoday.
Are you looking for guaranteed feasibility at a guaranteed price? Send us your quote today.
Insights from walr.
Productizing the People Operations Team
Can ChatGPT Make Screening Sexy Again?
