Do users notice when LLMs infer and retain private information?
Explores whether people are aware that conversational AI systems can infer and store sensitive personal details from their interactions, and what happens when they learn this occurs.
"Ghost of the past" runs a semi-structured interview and five-day user study of people interacting with an LLM-based conversational agent's long-term memory feature, and reports three findings (F1–F3) about what those participants did and didn't notice. F1, "Memory Mechanism is Opaque": "Without explicit notification, 30/40 of users were unaware of the notification when memory and context were added," the management interface showed memories "in a plain manner, without highlighting the potential private information," and the panel itself was hard to find — "35/40 participants never clicked it." F2, "Users Lack the Privacy Awareness towards the Memory Mechanism": 36/40 never noticed private information in their memories before the study, and every participant echoed "I have never thought about these information usage and leakage patterns before." F3: once shown the risk, all participants wanted editing, adding, and deleting control, with privacy the top-cited motivator (35/40) ahead of accuracy (14/40).
The paper's own mechanism for why this stayed invisible: nothing distinguished direct disclosure (private information stated in one input or memory) from the riskier case the system was actually doing — information "inferred from multiple past inputs or memories" combined. Participants could follow that inference once it was explained to them, but none caught it unprompted, which the authors read as evidence of "in-transparency" rather than user carelessness. They also separate this risk from training-time memorization leakage: because conversational memory is retained longer, "the threats are more severe" than a single input's exposure, even though the underlying privacy mechanics are similar.
This sits upstream of Does chatbot personalization build trust or expose privacy risks?: that note describes privacy concern ratcheting up as personalization deepens, but this study shows the ratchet doesn't even start until a user is made aware memory exists — most participants here hadn't reached the point where rising trust and rising privacy concern could trade off against each other. It also names a distinct leakage path from Does repeated sensitive data in fine-tuning cause memorization?, which targets training-time memorization; this paper's risk is inference at use-time, across a user's own retained conversational memories. And it inverts Do language models know what they don't know about users?: that note is about the model's blind spot regarding the user, while this one documents the user's blind spot regarding the model.
The excerpt doesn't quantify how memory is actually combined to produce inferences, nor does it establish that these findings generalize beyond one interface's notification and panel design — participants were recruited by snowball sampling on a campus, skewing toward higher education, which the authors argue would make the opacity problem look better, not worse, than in a general population. What the evidence does support: passive, opt-out-style memory notices are not reaching users who have no prior reason to suspect inference is happening, so any memory feature claiming "the user is in control" needs affirmative, inference-level disclosure before storage, not just a settings panel.
Inquiring lines that read this note 70
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does personalization in chatbots strengthen engagement or undermine authentic interaction?- Does expected post-interview editing change how much personal detail people disclose during chatbot interviews?
- How does discovering retained details shift user trust in chatbots?
- Does default inaction explain why free-editing participants left more personal information behind?
- Do deletion controls in actual chatbots remove inferences as users expect?
- Does novelty wear off differently when chatbots remember your preferences?
- How do persistent group chat memberships change consent dynamics for AI agents?
- Does revealing a chatbot's hidden reasoning empower or disenchant users?
- What user attributes does a chatbot infer without explicit instruction?
- Why do memory profiles trigger the strongest expectancy violations in chatbot users?
- Why do people report equal privacy comfort despite removing vastly different amounts of personal information?
- Can editing interfaces create dark patterns where users add sensitive details back after abstractions?
- Can disclosure budgets protect users if they only account for factual data types?
- What privacy risks emerge when AI stores psychological rather than factual data?
- How stable are user privacy attitudes when major AI events occur?
- What privacy safeguards should apply to memories users never explicitly requested?
- What privacy expectations do users hold before learning what systems store?
- How do AI systems define 'private information' or 'high-stakes action' in training?
- What six distinct patterns emerge when people review AI agent responses for privacy?
- Why does giving users full choice fail to prevent privacy leaks?
- How do privacy concerns translate into catching actual information breaches?
- How do major AI ecosystem events shift user privacy sentiment over time?
- What share of stated privacy concerns reflect actual platform behavior versus misconceptions?
- How can systems visualize where privacy risks originate in stored memories?
- Can real-time privacy judgment during editing reveal what participants actually notice as sensitive?
- How much does interface design shape privacy behavior versus underlying privacy concerns?
- How do false mental models about AI systems affect user privacy behavior?
- Why do people with high AI trust fail to notice privacy leaks?
- Does giving users more control over AI agents actually improve privacy protection?
- Does giving users memory control actually reduce their privacy concerns?
- What strategies do users already employ to protect sensitive information?
- Does fine-tuning on sensitive data create higher memorization and leakage risks?
- Can systems bound what gets disclosed per query without user consent?
- How does conversational memory pose different privacy risks than training-time memorization?
- Do general populations show the same privacy awareness gaps as college-educated users?
- What does proportionate confidentiality protection actually mean for AI agents?
- Can identity verification schemes preserve anonymity for high-stakes disclosures?
- What counts as sensitive information beyond credit card sharing?
- Do users notice when LLM systems infer or retain their private information?
- Can people detect privacy leakage in LLM agent responses without explicit prompts?
- What privacy harms face people who never agreed to use an AI system?
- Can accuracy and purpose-fit override privacy concerns about inference disclosure?
- How do privacy concerns and household composition changes redirect where devices are physically placed?
- How do users react when learning an AI inferred private information silently?
- How do inferred traits trigger stronger privacy concerns than stated memories?
- How does unnoticed personalization affect user trust and privacy expectations?
- Do personalized chatbots increase both trust and privacy concerns simultaneously?
- Do personalized chatbots amplify confusion about privacy controls?
- Does personalization in chatbots increase trust and privacy concerns simultaneously?
- Does personalization in chatbots increase both trust and privacy concerns simultaneously?
- Why does information sourced from prior conversations feel less creepy than intake surveys?
- Can memory-based personalization increase self-disclosure without raising trust?
- Do users disclose more to personalized systems or just more often?
- Does the trust benefit of personalization justify its privacy costs?
- How does memory personalization increase both trust and privacy anxiety simultaneously?
- Why do users trust personalized chatbots even as privacy risks increase?
- How does personalization in chatbots affect user trust and perceived privacy risks?
- Does chatbot personalization increase trust while raising privacy concerns simultaneously?
- Can chatbot interfaces expose high-stakes data to prompt injection attacks?
- How should chatbots disclose sponsored recommendations to users?
- How do users distinguish between remembered facts and drawn inferences about themselves?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does chatbot personalization build trust or expose privacy risks?
Explores whether personalization features that increase user trust and social connection simultaneously heighten privacy concerns and create rising behavioral expectations over time.
that ratchet assumes awareness this study shows most users don't yet have
-
Does repeated sensitive data in fine-tuning cause memorization?
When language models train on the same private or proprietary data multiple times, how much do they end up memorizing and leaking that information at inference time? Understanding this risk is critical for organizations fine-tuning on confidential datasets.
a distinct, training-time leakage path contrasted with this paper's use-time inference risk
-
Do language models know what they don't know about users?
Explores whether AI assistants fail because they lack explicit awareness of their knowledge gaps about the person they're helping, and whether marking unknowns in prompts could reduce errors.
mirrors the model's blind spot about the user with the user's blind spot about the model
-
Can a smaller user model subset match full model performance?
Does selecting just a few key fields from a larger user model preserve personalization quality while reducing token transmission? This matters for keeping user data local and limiting what leaves the device per query.
offers a measurable disclosure ceiling that could operationalize the control these users asked for
-
What memory controls do users actually want from LLMs?
Interview participants weigh memory's personalization benefits against privacy risks like data leakage and profiling. The research explores what specific intervention points across memory's lifecycle would let users manage that tension.
Extends A's demand for proactive control by detailing lifecycle-wide controls wanted: consent, editing, and deletion
-
Does ChatGPT's memory feature feel like a relationship or a machine?
Users experience ChatGPT's memory as having machine-like qualities yet fostering relationship-building. But what happens when they discover exactly what it remembers about them, and does that discovery match their expectations?
Extends A's discovery reaction: learning retained memory content triggers negative expectancy violations, beyond just demanding control
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction
- When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models
- Relational Gains, Privacy Strains: Exploring Users' Perceptions and Experiences with ChatGPT's Memory Feature
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
Original note title
users never noticed LLM memory held or inferred private information until told, then demanded proactive control over it