MIT Study: Do Readers Prefer Human, AI, or Human+AI Content? Answered in 3 Bar Charts

MIT Sloan’s Business School recently conducted an insightful study evaluating various content creation methods. This unique research pitted ten professional content creators from a leading consulting firm against ChatGPT-4. The participants, including these creators and the AI, generated advertising content for products and persuasive content for campaigns.

The findings offer valuable insights for professionals in digital marketing and SEO. Let’s explore the study’s details and its key takeaways.

AI vs Human Content Study Highlights:

So, to understand this study and the charts below, you’ll first want to understand the basics of this study.

• Researchers:
  • Yunhao Zhang, MIT Sloan
  • Renee Gosline, MIT Sloan

• Date Written: May 20, 2023

• Study Size: 1,212 participants

• Content Creation Methods:

  • Human: 10 professional content creators from a leading consulting firm

  • AI: Using ChatGPT-4

  • Augmented Human: Human editing AI-generated content

  • Augmented AI: AI refining human-generated content

• Writer Prompts:

  • Create persuasive campaign content in under 100 words. Encourage actions like stopping racism or starting recycling.

  • Write advertising content for a product in under 100 words. Use provided product pictures and feature descriptions.

• Participant Asks:

  • Willingness-to-Pay: State the largest amount they would pay for the content.

  • Interest or Persuasion: Rate their interest in the product from the campaign.

• Conditions:

  • Baseline: Participants evaluated content without knowing its creation method (human or AI).

  • Partially Informed: Participants were aware of the four content creation methods but not which was used for each piece.

  • Informed: Participants knew the exact content creation method used for each piece.

Study Findings

The first chart shows the satisfaction ratings given by the 1,212 participants. In the baseline condition, AI-generated content was the most satisfying, closely followed by Augmented AI. Human-only content scored the lowest.

The second chart reveals the maximum amount participants would pay for the advertised product. AI and Augmented AI content were more compelling in creating perceived value.

The third chart illustrates interest and persuasion of the content, separated by campaign and product writing. Here, AI was greatly preferred for campaign copy, while Augmented AI was more favored for product copywriting.

Key Takeaways

Participants Prefer AI or Augmented AI Content

This really surprised me.  Given that AI content is relatively new to the mainstream, considerably faster to produce, and significantly less expensive to create, I would expect professional writers to create more compelling writing.  According to the study, participants preferred AI and AI-augmented content due to three reasons:

AI and Augmented AI Perceived as Higher Quality

Participants often viewed content created by AI and augmented AI as superior in quality compared to that created by humans or augmented humans.

Effectiveness of ChatGPT-4

The use of ChatGPT-4, a sophisticated AI model, was effective in generating content that resonated well with the participants, indicating the potential of AI in content creation.

Human Bias in Content Evaluation

There was a noticeable bias towards human-generated content. When participants knew the content was created by humans, they rated it higher, suggesting a preference for human authorship.

No Significant AI Aversion

Contrary to some expectations, the study found no substantial negative bias against AI-generated content among the participants.

Objective Evaluation in Baseline Condition

In the baseline condition, where participants were unaware of the content’s origin, their evaluation was based solely on the content itself, leading to an impartial assessment of AI-generated content.

Collaboration Between Humans and AI

The study highlighted the effectiveness of human-AI collaboration in content creation, suggesting that a combined approach could yield high-quality results.

In Conclusion

This study suggests that AI is a powerful tool in content creation, but it doesn’t signal the end for human content creators. The study’s scope was limited to short prompts ideal for AI, and human writers excel in understanding audiences, storytelling, and adding personality in long-form copy.

Interestingly, Augmented AI scored similarly to human writing. Common practice involves AI drafting first and then human editing, but the study suggests the reverse workflow could be beneficial. In my content creation, I plan to use ChatGPT-4 for final revisions and note that aversion to AI content is rapidly diminishing.

The full study is available online if you’d like to dig deeper and extract even more leanings.  And here’s another idea: download the PDF and upload it to ChatGPT-4, then you can ask it questions for direct answers!

Human vs AI Content FAQs