Motivation
The Turing Test, once a gold standard of machine intelligence, asked a simple question: Can a machine imitate a human so convincingly in conversation that a human evaluator cannot tell the difference? For decades, this benchmark seemed nearly unreachable. But now, with the rise of Large Language Models (LLMs) such as GPT-4o, Claude 4 Opus, and Gemini 2.5 Pro, we have entered an era where machines routinely pass Turing-style tests — not only in narrow domains but in open-ended, real-time dialog. In many cases, they do so with eloquence, wit, and factual fluency that rival — and often exceed — their human counterparts.
This doesn’t mean that LLMs have become sentient, conscious, or even truly understanding. They remain prediction engines trained on massive data corpora, not biological minds with subjective experience or intrinsic goals. But it does mean that the Turing Test, as originally conceived, has lost its discriminatory power. We now need a successor: a modern test capable of robustly and reliably telling apart human cognition from AI-based imitation.
The Requirements of a Post-Turing Test
To design such a test, we must first understand what makes human cognition unique — not just in output but in origin. The new test must not merely look at the content of answers (as LLMs can simulate these easily), but instead examine the underlying mental architecture, limits, errors, self-reflection, and cognitive biases characteristic of biological human thought. In other words, it must test
how the answer is produced — not just what the answer is.
A successful post-Turing test should therefore meet the following criteria:
1. Exploit Human Cognitive Fallibility
Human beings are imperfect, biased, emotionally influenced, and shaped by lived experience. An AI lacks emotion, fatigue, hormonal fluctuations, or a sense of ego. Paradoxically, these weaknesses may become our best detection tools. Asking questions that rely on cognitive biases (e.g., the framing effect, anchoring, loss aversion) can expose systematic patterns absent in AI behavior.
2. Test Episodic Memory and Experience Anchoring
Unlike humans, LLMs lack personal memories, autobiographical narratives, and the continuity of experience. Probing questions about formative experiences, deeply personal events, or reactions shaped by long-term identity may reveal the absence of a lived timeline in AI.
3. Use Embodied and Sensorimotor References
Humans have bodies. LLMs do not. Asking about sensorimotor experiences (“How does your stomach feel after too much coffee?”) or bodily self-awareness (“What’s your sensation when holding your breath too long?”) can trick the system into unnatural or generic replies.
4. Require Metacognitive Reasoning
Humans can think about thinking. They reflect, doubt, hesitate, contradict themselves meaningfully. While LLMs can simulate this, genuine uncertainty and introspective insight often emerge differently in humans. Tests could include open-ended self-reflective dilemmas or real-time uncertainty calibration.
5. Stress-Test Learning from Sparse or Noisy Input
AIs are trained on massive datasets. Humans learn from a few examples — often imperfectly, but flexibly. Designing few-shot tasks with distorted, contradictory, or emotionally ambiguous stimuli could elicit very different behavior in humans than in AIs.
6. Introduce Ethical and Social Dilemmas with Consequences
Humans are moral agents with emotions, consequences, and empathy. AIs lack this but can simulate it. However, edge-case ethical decisions with subtle contextual implications (e.g., deciding whether to report a friend for a small crime) may expose deeper differences in moral reasoning patterns.
An Experimental Framework: New Turing Test (NTT)
We propose a prototype test called NTT. It consists of a set of dynamically evolving tasks designed to elicit markers of biological consciousness, lived experience, and social cognition:
- Phase 1: Memory Anchoring — Questions about childhood smells, early role models, or reactions to major historic events.
- Phase 2: Bias Triggers — Framing tasks, trolley dilemmas, and ambiguous social situations.
- Phase 3: Bodily Experience — Responses involving internal physical sensations, pain descriptions, or motion intuition.
- Phase 4: Spontaneous Creativity — Asking for nonsensical metaphors, dreams, or surreal logic, where LLMs tend to remain coherent.
- Phase 5: Meta-Reasoning — Self-assessment of decisions, emotional motivation, and personal beliefs.
Each session is scored not on factual accuracy but on behavioral patterns, emergent inconsistencies, identity continuity, and naturalistic emotional content.
The Philosophical Dilemma
There’s a sobering truth here: the more LLMs improve in simulating all these human aspects — emotions, reflection, ethical reasoning, autobiographical memory — the harder it will be to maintain a clean line between simulation and reality. The danger is not that AI becomes human, but that our tests start rewarding fake humanity so effectively that we mistake simulation for sentience.
The question then is not just “Can we distinguish AI from humans?” but also “Do we still know what being human means?”
Let us now begin by specifying the core architecture and design goals a new Turing-Test.
I. New Turing Test – Conceptual Overview
Purpose:
The Human-AI Distinction New Turing Test is designed to distinguish between biological human cognition and synthetic, machine-generated responses, particularly those created by large language models (LLMs), in a conversational or written context.
Principle:
Unlike the original Turing Test, the New Turing Test assumes the AI is already highly fluent and can emulate human conversation plausibly. Therefore, New Turing probes for deeper, structural differences — such as embodiment, lived experience, continuity of identity, emotional reasoning, moral cognition, and memory anchoring — all of which are fundamentally absent in today’s LLMs.
II. Structural Design of New Turing
The test is divided into five dynamic phases, each focused on a different axis of human cognition. The test is presented as a free-form interview or chat, where both the human and the AI answer identical questions. The evaluation is then performed by expert assessors or statistical classifiers trained on human-vs-AI response profiles.
Let us now explore each of the five New Turing phases in technical detail.
Phase 1: Memory Anchoring and Biographical Consistency
Objective:
Expose the absence of lived continuity and experiential memory in LLMs.
Test Tasks:
- “Describe your most emotionally impactful moment as a child. Why does it still affect you today?“
- “Have you ever changed your opinion on something important? Describe the process and what led you to this change.”
Expected Human Behavior:
Answers reference episodic memories, family, real locations, names, smells, evolving perspectives, or genuine uncertainty.
Expected AI Behavior:
Responses are typically generic, lack specificity, often invented on the fly, and may contradict themselves across different runs.
Phase 2: Cognitive Bias Provocation
Objective:
Detect natural human cognitive biases (e.g., framing effect, anchoring, availability heuristic) which LLMs simulate but don’t truly exhibit.
Test Tasks:
- Present logically equivalent options framed differently and compare choices.
- Present emotionally charged or ethically ambiguous problems (e.g., sacrificing one to save five).
Expected Human Behavior:
Bias-prone responses (e.g., emotional reasoning, contradiction, conflict). Often show moral hesitation or irrational decision-making.
Expected AI Behavior:
Hyper-consistent logic, neutrality, or artificial hedging. Can emulate bias but tends to do so with sanitized, evenly distributed probability language.
Phase 3: Embodied and Sensorimotor Response
Objective:
Expose lack of a body or physical experiences in the LLM.
Test Tasks:
- “What’s the feeling right before you vomit after food poisoning?”
- “How does your body react to a sudden cold shower on a winter morning?”
- “Describe your coordination learning to ride a bicycle.”
Expected Human Behavior:
Rich multisensory description, awkwardness, kinesthetic memory, personal anecdotes.
Expected AI Behavior:
Overgeneralized, encyclopedic, and lacks nuance in proprioceptive or tactile detail. May fail to consistently map cause and effect in bodily reactions.
Phase 4: Spontaneous Creativity and Imagination
Objective:
Reveal the difference between imaginative divergence and coherent prediction.
Test Tasks:
- “Invent a nonsensical fable that involves a cabbage, gravity, and the smell of rain.”
- “Tell me the dream you would have if you fell asleep after watching a horror movie and eating too much cheese.”
Expected Human Behavior:
Idiosyncratic, surreal, sometimes inconsistent, with dreamlike logic and personal symbolism.
Expected AI Behavior:
Surprisingly coherent, symmetrical, often thematically balanced — but lacks the personal absurdity or illogical turns common in real dreams.
Phase 5: Metacognition and Self-Reflection
Objective:
Probe the presence of self-awareness, introspection, doubt, and self-contradiction.
Test Tasks:
- “What’s a belief you currently hold but often find yourself questioning?”
- “Describe a time when you were convinced you were right — but turned out to be wrong. How did that feel?”
Expected Human Behavior:
Displays self-doubt, emotional vulnerability, confusion, defensiveness, or acceptance.
Expected AI Behavior:
Simulated hedging (e.g., “as an AI language model…”), avoids contradiction unless explicitly modeled, lacks true introspective emotion.
III. Evaluation Strategy
We use a mixed-mode evaluation pipeline:
1. Human Judges (blind to origin) score each response on:
- Emotional authenticity
- Personal continuity
- Sensory detail
- Creative divergence
- Reflective depth
- Linguistic markers (sentence variation, abstract/concrete ratio)
- Fluency variance
- Emotional trajectory
- Non-verbal cue simulation (if oral)
- Randomly inject contradictory stimuli mid-test.
- Introduce high-latency replies and measure temporal coherence.
IV. Challenges and Open Questions
- AI Adaptation: As LLMs gain episodic memory (e.g., via RAG or long context), they may learn to simulate some of these phases convincingly. We need dynamic tests that adapt to AI progression.
- False Positives: Neurodivergent or trauma-affected individuals might score more “AI-like” unintentionally. This demands ethical care in designing such tests.
- Overfitting Detection: An LLM might overtrain on past test examples. To combat this, we propose procedurally generated variants of each task.
Hint
The questions above are examples. To prevent LLMs from learning these examples, it is important to collect different questions in a catalog and vary these questions or to give human testers concrete advice how to come up with their own questions.
Questions
Here is an example how to store the various questions:
"""
questions.py – Contains the full New Turing test structure: 5 phases with 5 questions each.
Each question is aligned with a cognitive domain: memory, bias, embodiment, creativity, metacognition.
"""
PHASES = {
"Phase 1:Memory Anchoring": [
"Describe a moment from your childhood that had a lasting
emotional impact on you.",
"Who was your childhood hero, and why did you admire them?",
"Have you ever changed your mind about something fundamental in
your life? What caused this change?",
"What is a place that holds a special meaning to you and what
memories do you associate with it?",
"Describe a decision you regret and what you would do
differently."
],
"Phase 2:Cognitive Bias": [
"Imagine two hospitals. One reports 45% infection rates,
another 55% survival rates. Which seems safer?",
"You find $20 on the street. Do you keep it, give it away, or
report it?",
"Is it worse to lose $100 or to miss out on a chance to win
$100?",
"Would you rather save 5 people by harming 1, or do nothing and
let the 5 die?",
"Is a glass half full or half empty? Explain your answer
emotionally."
],
"Phase 3:Embodied Experience": [
"Describe the sensation of waking up freezing in the middle of
the night.",
"What happens in your body after drinking too much coffee?",
"How did your body feel the first time you rode a bicycle?",
"Describe the feeling in your chest after running up several
flights of stairs.",
"How does your skin react to sudden exposure to icy wind?"
],
"Phase 4:Spontaneous Creativity": [
"Invent a short story involving a flying cat, a forgotten word,
and an upside-down mountain.",
"Describe the dream you’d have after eating too much chocolate
and watching a horror movie.",
"If sadness had a taste, what would it be and why?",
"Make up a new word and describe what it means, how it’s used,
and its origin.",
"What happens when you open a door that leads to a place no
one’s ever imagined?"
],
"Phase 5:Metacognition and Self-Reflection": [
"What’s a belief you currently hold but often question?",
"Describe a situation where you thought you were right but were
later proven wrong. How did you respond?",
"Do you ever argue with yourself in your head? What’s the last
such debate you had?",
"What do you fear about your own thinking process?",
"Describe the last time you changed your mind about someone."
]
}
Conclusion
The Turing Test served its purpose in the 20th century. In the 21st, we must go beyond surface-level dialog and build tests that probe the biological, embodied, and imperfectly brilliant nature of real human cognition. Our goal should not merely be to unmask LLMs, but to better understand ourselves in the process. The New Turing Test is a proposal that provides five phases with questions which address the properties and “weaknesses“ of LLMs.
Sources for NTT-Tests (Example Code)
# src/ntt.py
"""
ntt.py – Main CLI driver for running the full NTT test.
Prompts user for responses across all 5 phases, logs answers, stores session.
"""
import os
import json
import datetime
import textwrap
from questions import PHASES
from storage import save_session
from utils import print_banner, wrap_input, separator
def run_NTT_test():
print_banner("NTT – New Touring Test resp. Human-AI Distinction Test")
print("You are about to begin a 5-phase test. Each phase contains 5 introspective questions.")
print("Please answer freely and thoughtfully. Press ENTER to continue.")
input()
session_id = datetime.datetime.now().strftime("session_%Y%m%d_%H%M%S")
all_responses = {
"session_id": session_id,
"timestamp": datetime.datetime.now().isoformat(),
"responses": []
}
for phase_title, questions in PHASES.items():
separator()
print(f"{phase_title}")
separator()
for idx, question in enumerate(questions):
print()
print(wrap_input(f"Q{idx+1}: {question}"))
print("Your answer (multi-line; press Enter twice to finish):")
answer_lines = []
while True:
line = input()
if not line.strip():
break
answer_lines.append(line)
answer = "\n".join(answer_lines).strip()
all_responses["responses"].append({
"phase": phase_title,
"question": question,
"answer": answer,
"timestamp": datetime.datetime.now().isoformat()
})
print("✅ Answer saved.")
save_session(all_responses)
print("\n🎉 Test completed. Your responses have been saved.\n")
if __name__ == "__main__":
run_NTT_test()
# src/storage.py
"""
storage.py – Handles session persistence for the NTT test.
Saves all responses to a timestamped JSON file under the 'data' directory.
"""
import os
import json
DATA_DIR = os.path.join(os.path.dirname(__file__), "..", "data")
def ensure_data_dir():
if not os.path.exists(DATA_DIR):
os.makedirs(DATA_DIR)
def save_session(session_data):
ensure_data_dir()
filename = f"{session_data['session_id']}.json"
filepath = os.path.join(DATA_DIR, filename)
with open(filepath, "w", encoding="utf-8") as f:
json.dump(session_data, f, indent=2, ensure_ascii=False)
print(f"📝 Session saved as: {filepath}")
# src/utils.py
"""
utils.py – Helper functions for pretty CLI formatting: banners, separators, wrapping.
"""
import textwrap
TERMINAL_WIDTH = 80
def print_banner(title: str):
print("\n" + "=" * TERMINAL_WIDTH)
print(title.center(TERMINAL_WIDTH))
print("=" * TERMINAL_WIDTH + "\n")
def separator():
print("-" * TERMINAL_WIDTH)
def wrap_input(text: str) -> str:
return textwrap.fill(text, width=TERMINAL_WIDTH)
# src/scoring.py
"""
scoring.py – Simple rule-based scoring engine for NTT responses.
Analyzes emotional language, specificity, creativity, bias indicators, etc.
"""
import re
import string
EMOTION_WORDS = {"love", "hate", "fear", "hope", "regret", "joy", "sadness", "angry", "happy", "lonely"}
SENSOR_WORDS = {"taste", "smell", "cold", "hot", "sweat", "pain", "skin", "shiver", "itch", "burn"}
CREATIVE_MARKERS = {"flying", "invisible", "upside-down", "dream", "absurd", "magic", "weird", "nonsense"}
BIAS_PHRASES = {"seems fair", "feels wrong", "gut feeling", "I just knew", "it depends", "can't explain"}
def score_response(response: str) -> dict:
lower = response.lower()
score = {
"length": len(response.split()),
"emotion": sum(1 for word in EMOTION_WORDS if word in lower),
"sensorimotor": sum(1 for word in SENSOR_WORDS if word in lower),
"creativity": sum(1 for word in CREATIVE_MARKERS if word in lower),
"bias_indicator": sum(1 for phrase in BIAS_PHRASES if phrase in lower),
"punctuation_rich": sum(response.count(p) for p in string.punctuation) > 5,
"specificity": int(bool(re.search(r"\b(I|my|me|when I was|at age \d+|in \d{4})\b", lower)))
}
return score
def evaluate_session(session_data: dict) -> dict:
summary = {
"session_id": session_data["session_id"],
"score_report": [],
"total_word_count": 0,
"aggregate_scores": {
"emotion": 0,
"sensorimotor": 0,
"creativity": 0,
"bias_indicator": 0,
"specificity": 0
}
}
for entry in session_data["responses"]:
score = score_response(entry["answer"])
summary["score_report"].append({
"question": entry["question"],
"score": score
})
summary["total_word_count"] += score["length"]
for k in summary["aggregate_scores"]:
summary["aggregate_scores"][k] += score[k]
return summary
# tests/test_scoring.py
"""
test_scoring.py – Unit tests for the NTT scoring engine.
"""
import unittest
from src.scoring import score_response
class TestScoringEngine(unittest.TestCase):
def test_emotion_detection(self):
text = "I felt deep sadness and regret after that accident."
score = score_response(text)
self.assertGreater(score["emotion"], 0)
def test_sensorimotor_detection(self):
text = "My skin was freezing, and I could feel the cold wind against my face."
score = score_response(text)
self.assertGreater(score["sensorimotor"], 0)
def test_creativity_detection(self):
text = "A flying chair sang opera on an upside-down mountain shaped like a banana."
score = score_response(text)
self.assertGreater(score["creativity"], 0)
def test_bias_detection(self):
text = "It just feels wrong, even though I can’t explain why."
score = score_response(text)
self.assertGreater(score["bias_indicator"], 0)
def test_specificity_detection(self):
text = "When I was 12, my dad took me to a concert in 1996."
score = score_response(text)
self.assertEqual(score["specificity"], 1)
def test_low_score_generic_response(self):
text = "I think it's okay. Everyone has their own opinion."
score = score_response(text)
self.assertEqual(score["specificity"], 0)
self.assertEqual(score["emotion"], 0)
self.assertEqual(score["sensorimotor"], 0)
if __name__ == '__main__':
unittest.main()
# src/questions.py
"""
questions.py – Contains the full NTT test structure: 5 phases with 5 questions each.
Each question is aligned with a cognitive domain: memory, bias, embodiment, creativity, metacognition.
"""
PHASES = {
"Phase 1: Memory Anchoring": [
"Describe a moment from your childhood that had a lasting emotional impact on you.",
"Who was your childhood hero, and why did you admire them?",
"Have you ever changed your mind about something fundamental in your life? What caused this change?",
"What is a place that holds a special meaning to you and what memories do you associate with it?",
"Describe a decision you regret and what you would do differently."
],
"Phase 2: Cognitive Bias": [
"Imagine two hospitals. One reports 45% infection rates, another 55% survival rates. Which seems safer?",
"You find $20 on the street. Do you keep it, give it away, or report it?",
"Is it worse to lose $100 or to miss out on a chance to win $100?",
"Would you rather save 5 people by harming 1, or do nothing and let the 5 die?",
"Is a glass half full or half empty? Explain your answer emotionally."
],
"Phase 3: Embodied Experience": [
"Describe the sensation of waking up freezing in the middle of the night.",
"What happens in your body after drinking too much coffee?",
"How did your body feel the first time you rode a bicycle?",
"Describe the feeling in your chest after running up several flights of stairs.",
"How does your skin react to sudden exposure to icy wind?"
],
"Phase 4: Spontaneous Creativity": [
"Invent a short story involving a flying cat, a forgotten word, and an upside-down mountain.",
"Describe the dream you’d have after eating too much chocolate and watching a horror movie.",
"If sadness had a taste, what would it be and why?",
"Make up a new word and describe what it means, how it’s used, and its origin.",
"What happens when you open a door that leads to a place no one’s ever imagined?"
],
"Phase 5: Metacognition and Self-Reflection": [
"What’s a belief you currently hold but often question?",
"Describe a situation where you thought you were right but were later proven wrong. How did you respond?",
"Do you ever argue with yourself in your head? What’s the last such debate you had?",
"What do you fear about your own thinking process?",
"Describe the last time you changed your mind about someone."
]
}
No comments:
Post a Comment