Instinct's AI Assistant Raises Concerns Over Privacy and Security
A new personal AI assistant, Instinct, has been making waves in the tech community for its impressive capabilities. However, concerns have also been raised about its potential impact on user privacy and security. The agent, which is still in private testing, has been described as feeling ‘like magic’ by some testers, but others have expressed worries about its terms of service and security model.
Instinct was created by a small team led by former Sierra research scientist Noah Shinn. It’s operated by Spear Street Technology, according to the company’s California business filings and PitchBook. The AI agent is currently operating in stealth mode, which has contributed to some of the concerns surrounding its use.
The way Instinct works is by connecting to users’ applications and devices, including email, messaging apps, calendar, audio, location, screen, and more. This allows it to perform various tasks on behalf of the user, such as booking appointments and reservations, scheduling rides, cleaning up inboxes, organizing important information, handling shopping, finding cheap flights, and much more.
Despite its impressive capabilities, some testers have raised concerns about Instinct’s approach to customer privacy and security. This has led to a timely question: are the trade-offs of giving AI this level of access and autonomy worth it? For instance, several people have circulated screenshots from Instinct’s Terms of Service, which grant the company a broad ‘perpetual and irrevocable’ license to access, use, host, cache, store, reproduce, transmit, display, publish, distribute, and modify any user materials, including for training its AI models.
The terms also detail how Instinct can receive information from users’ devices, including screen captures, cursor movements, and keyboard inputs. Furthermore, the company is allowed to enter into ‘agreements, commitments, or transactions’ on behalf of users, which would be binding. This has raised concerns about user control and agency over their own data.
One early adopter, Peter Yang, pointed out that Instinct wouldn’t delete his Gmail records when asked. However, the team later fixed this issue by adding a tool for deleting external data in its settings. Another person, Claire Vo, found that Instinct was still summarizing her inbox after disconnecting its access. When she asked Instinct what happened, the bot confirmed that the emails were stored in plain text for later searches.
Many others have questioned the security model of Instinct as well. One tester expressed concern when they discovered that Instinct could pull a sign-up code from their email inbox to complete a particular task – in this case, booking a table at a restaurant via Resy. Hello Patient co-founder Alex Cohen wrote that once he found out how easily Instinct could be phished, he deleted his account.
Moxxie Ventures founder Katie Jacobs Stanton shared that Instinct broke her trust when it sent an email on her behalf without first checking with her. She noted that ‘we’re trading privacy and control for hyper-personalized AI tools (AI notetakers, personalized AI agents, etc), often without fully understanding the trade.’ This highlights the dilemma posed by personal AI agents: as they become more powerful, trust becomes increasingly important.
Michael Mignano, founder of Anchor, which was acquired by Spotify and now a GP at Union Square Ventures, noted that products like Instinct are going to ‘change modern security norms for consumers,’ adding that people will increasingly hand over passwords to third-party apps without understanding how or what they’re storing. This raises concerns about the long-term implications of using such AI assistants.
Interest in Instinct and personal AI has been growing since the arrival of OpenClaw, a popular personal AI assistant that became known for its powerful capabilities. Its founder joined OpenAI to help work on the next generation of personal agents. Another messaging-based assistant, Poke, also recently exited to Cognition.
The team behind Instinct hasn’t yet responded to concerns or complaints about their product. The company has been quiet, preferring to keep a low profile despite growing interest in its AI assistant. TechCrunch has heard from multiple investors that Kleiner Perkins and Conviction have invested in the startup and those rounds have now closed.
Requests for comment sent to both the startup’s main email address and Shinn directly haven’t yet been returned. As Instinct continues to operate in stealth mode, it remains to be seen how these concerns will be addressed and whether users will continue to trust their personal AI assistants with sensitive information.