Getting Insights

Surfacing detailed audience insights through analysis is SUPER valuable for improve marketing campaigns. As we gain understanding on their pain points, concerns and opinions we can explain our value in a way that really clicks.

Reddit discussions are a useful source of information. However gems of insights are hidden in the muck of a discussion threads are interlaced with jokes, memes and all sort of crud - and reading through the whole thread

With multiple LLMs at our disposal, this is quite possible in 3mins.

Our Example:

Reddit Thread: ChatGPT vs. Claude. What do you prefer more?

(Yes- it's quite meta. But Educational as it reaches the same conclusion as this post)

⠀ ⠀

The Analyst - Claude

⠀

Artifacts

Claude can generate Artifacts - interactive analyses, charts, and even web apps—right in the sidebar. Here's one I created for fun!

This is super useful as you could do exploratory data analysis and get insights on the fly!

⠀

Getting data into Claude

However to build an artifacts we need data - and structured data.

💡 At the moment Claude can't search so here is a neat trick to get structured data from reddit: just add .json behind the thread you want to analyze like so
...chatgpt_vs_claude_what_do_you_prefer_more/.json

and out comes this fairly gnarly looking Json - ugly but the same content but structured for programs and artifacts.

Now - we could just paste this into Claude and start the analysis:

⠀

The Analysis

Here we try out the following prompt:

"I am doing social listening for marketing for chatgpt vs claude. what are some insights from the comments chatgpt_vs_claude_what_do_you_prefer_more "

The prompt is deliberately kept simple - as I found at the exploratory stage, if I over-engineer prompts it would shoehorn the findings to what I want instead of finding patterns)

⠀

and below is the inital output:

⠀

Key Insights:

Usage Patterns & Preferences

• Many users utilize both tools for different purposes rather than exclusively using one
• Some users preserve ChatGPT tokens/credits for minor tasks while saving Claude for more complex work
• Several users praise Claude's technical capabilities but still default to ChatGPT due to familiarity and ease of use

Claude's Perceived Strengths

• Consistently praised for superior coding abilities, especially for complex technical tasks
• Noted for better performance in nuanced critical thinking
• Better at troubleshooting its own code
• More precise and conservative in responses when uncertain
• Handles context and project organization better through its Projects feature
• Some users, including a physician, find it "game-changing" for professional work

ChatGPT's Perceived Strengths

• More versatile with additional features like image generation and web browsing
• Voice capabilities (frequently mentioned as a key differentiator)
• More conversational and natural in interaction
• Better at creative tasks and casual use
• More engaging personality with "wit & humor"
• Web search functionality is valued by users

Pain Points

• Claude users complain about message/prompt limits
• Some users find Claude overly cautious or restrictive
• Claude sometimes seen as too formal or rigid ("like an elementary school nun")
• Pricing and token management is a consideration for users of both platforms

Market Opportunities

• Strong demand for voice features in Claude (multiple users mention this)
• Users express willingness to pay more for unlimited access
• Interest in combined/integrated solutions (some users mention using platforms that allow access to both)

⠀

The Artifacts

With the data in, creating an artifact is as simple as asking it -

"cluster into categories for each topic and points, and use artifacts to analyze"

⠀

And Claude replies:

" I'll analyze the comments and create a visualization to cluster the topics and key points.

Let me also create a visualization specifically for the relative popularity of different features and aspects: I've created two interactive visualizations to help analyze the data"

⠀

Is it so easy? Or is it?

Here is where working with LLM gets hairy.

My own rule is that I would ALWAYS CHECK LLM's WORK

⠀

A quick click into the code and I could see that the input are just hard values. 😠 😡

However - just asking it - "is your data from the Json or fabicated?" set it straight.

⠀

I spent the next minute scanning through its script, making sure it wasn’t hallucinating and checking the assumptions baked in. (all passed the sniff test)

I then refined the chart further—breaking down each bar and segment into positive vs. negative sentiment. Now, the insights are even clearer!

Finally I got it to generate samples below-

I'll list out the actual comments from the data for each category:

Technical/Coding

ChatGPT:

• Positive: "Better at troubleshooting its own code than ChatGPT" (score: 5)
• Negative: No negative mentions found

Claude:

• Positive: "Claude tends to follow my prompts better and produce code that is more likely to work as requested the first time" (score: 6)
• Negative: No negative mentions found

Features:

ChatGPT:

• Positive: "ChatGPT's ability to make images and search the web are very helpful for my use cases" (score: 7)
• Negative: No explicit negative feature mentions

Claude:

• Positive: "I love their projects feature and sonnet 3.5 is great" (score: 18)
• Negative: "won't let me, or it's a very strict limit"

The Baseline - The ChatGPT

(The Fast and Convenient)

For comparison, here’s what ChatGPT can do with it;s search capability:

"I am doing social listening for marketing for chatgpt vs claude. what are some insights from the comments chatgpt_vs_claude_what_do_you_prefer_more/ "

Here are some pretty good initial findings:

1 - Product Features and Performance:

• Claude's Advancements: Users recognize Claude for its rapid progress, particularly in coding, language processing, and overall performance, especially with updates like Sonnet 3.5 and agentic computer use.
  
• ChatGPT's Versatility: ChatGPT is valued for its versatility, excelling in creative writing and coding tasks, with praise for its adaptability.

2 - User Preferences and Usage Patterns:

• Task Allocation: Users tend to reserve Claude for more complex tasks, leveraging ChatGPT’s free version for simpler jobs to optimize token usage.
• Perceived Superiority: Some users feel that Claude outperforms ChatGPT-4 Pro, prompting long-time ChatGPT Pro subscribers to consider canceling their subscriptions.

3- Integration and Ecosystem:

• Tool Compatibility: ChatGPT's integration with tools like DALL-E enhances its appeal, especially for users looking for a multifaceted AI solution.

While these are really useful, Claude's analysis provide so much depth!

And it is up to your imagination how you want your Analyst Assistant (Claude) to slice and dice

the data.

Insights - LLMs As Different Focused Assistants

From this exercise and the output we could see:

Claude Is leading in the coding / data analytics space
With Artifacts there is also the possibility of interactivity. Which is useful for exploratory analysis.

However to wield it effectively some coding knowledge is paramount at the moment to fix hallucinations.

ChatGPT while fully capable in its own right seems to take on the less specialist role, favoring better UX/UI and muticapabilities (which was how ChatGPT won the world over to LLMs in the first place)