Generative AI Privacy Risks: What Business Leaders Need to Know

Generative AI is rapidly gaining popularity across various industries, but understanding AI privacy risks is crucial for success. The Cisco 2023 Consumer Privacy Survey found that 39% of consumers have entered work information, and over 25% have entered personal information, account numbers, and health and ethnicity information. 

88% of those consumers worried that their data could be shared with others. Their concern isn’t misplaced.

Despite policies allegedly protecting your data, many generative AI companies are less-than-transparent regarding data storage and use. While this may convince some business owners that they should write AI off entirely, the risk does have its reward: AI has the potential to add between $240 and $460 billion to the high-tech sector, and in 2024, AI software has led to revenue increases of over 5% in supply chain and inventory management.

That said, it’s important to know generative AI’s privacy risks and how best to mitigate them. 

What Are the Security Risks of Generative AI?

Though the exact security and privacy risks that each AI model faces are unique, there are a few common ones that users should keep an eye out for. Most models, by default, incorporate user prompts and conversations into their training. 

The companies behind these models also have access to user data, though some limit access to a few key staff members for specific purposes. As no two models are the same, the details may differ. 

Here are a few common tools, and the security risks that come with them.

ChatGPT Security Concerns

OpenAI’s ChatGPT model is one of, if not the most well-known, generative AI models currently on the market. It grows smarter and more capable with each update, making it a versatile tool for anything from planning a daily routine to editing content and helping programmers refine their code. Unfortunately, useful as it may be, the AI privacy concerns have grown with each new iteration.

Rui Zhu, a PhD candidate at Indiana University Bloomington, extracted the contact information for over 30 New York Times employees from GPT-3.5 Turbo. While he and his team were using ChatGPT’s application programming interface (API) over its public interface, Zhu’s findings revealed that the AI not only retains, but could also output specific individuals’  personal information. 

Though OpenAI insists that its Large Language Models (LLMs) do not copy or store information in a database, the company isn’t always forthright about the data used for its training model. Dr. Prateek Mittal, a professor in the Department of Electrical and Computer Engineering at Princeton University, noted that AI companies couldn’t guarantee that these models had not learned sensitive information in the same article.

“To the best of my knowledge, no commercially available large language models have strong defenses to protect privacy,” Dr. Mittal said. “I think that presents a huge risk.”

Google Gemini and AI Privacy Risks

Google Gemini is the brainchild of the eponymous company and enjoys all the benefits that come with it. However, Google openly discourages Gemini users from sharing sensitive information with the chatbot. “Chats with Gemini aren’t end-to-end encrypted,” Google explains. The Gemini Apps Privacy Hub reinforces this, emphasizing that “Google collects your Gemini Apps conversations, related product usage information, info about your location, and your feedback.” 

Google has also informed users that their information can, and will, be privy to outside eyes. “To help with quality and improve our products,” Google says, “...human reviewers read, annotate, and process your Gemini Apps conversations.” Though it assures users it takes steps to protect their privacy, it makes no secret that they must still be mindful of what they share.

Despite these warnings, users can and often do share personal information with Gemini. Without a company-protected alternative, your employees may feed sensitive information to Google.

Copilot AI Security Risks for Companies

Copilot is Microsoft’s answer to the AI arms race and now comes pre-packaged in Windows 10 and 11. While it was introduced as a helpful digital assistant and chatbot to support Microsoft 365 users, it has raised significant privacy concerns. 

The Information Commissioner's Office (ICO) reported that they would be “making inquiries with Microsoft” over Copilot’s Recall feature, which privacy campaigners referred to as a “privacy nightmare.” Recall can search users’ entire past activity, including their files, photos, emails, and browsing history. Not only that, but Recall takes screenshots every few seconds, which it also searches.

The feature has raised alarm bells for many, with experts noting that it could capture passwords or proprietary and confidential data. While Microsoft assures users that it has “built privacy into Recall’s design” and even made the feature opt-in after backlash, not everyone is convinced. For instance, the US House of Representatives banned congressional staffers’ use of Copilot over concerns of House data leaking to non-House approved cloud services. 

How Can You Protect Your Privacy Using AI? 

Troubling as these AI privacy concerns might be, they don’t negate the many uses generative AI models offer. Users can enjoy AI without compromising data security, but they must understand the risks and know how to avoid them. 

Check AI Privacy Policies

Knowledge is power, and knowing the terms and conditions of different generative AI models helps users understand how their data is handled. Websites’ terms and conditions provide valuable insight into users’ rights, the information they store, and what they may use your data for. OpenAI’s privacy policy, for instance, informs users that they may “disclose Personal Information to our affiliates, meaning an entity that controls, is controlled by, or is under common control with OpenAI.” 

As these policies may change over time or after updates, users should reread old policies to stay informed. 

Adjust Your Privacy Settings

After confirming that there are no deal-breakers in a model’s privacy policy, users should explore the privacy and security controls and adjust them to their liking. Many websites allow users to opt out of having their conversations used to train models. Some also offer the option to erase conversations automatically over a set period.

Stay Mindful of AI Security Practices

While adjusting your privacy settings is a step in the right direction, users shouldn’t depend on it to protect their data. The best way to safeguard sensitive information from generative AI is not inputting it. Users should only share information they feel comfortable going public, as while they can limit what AI models learn from their data, some companies store information for a period before deleting it. This information is vulnerable to data leaks and may still be accessed by the company behind the AI.

AI Privacy and Personalized AI

Generative AI’s privacy concerns may outweigh the benefits for some companies, discouraging them from using these models. Fortunately, there are ways for users to take AI into their own hands and reduce privacy risks. 

Companies building their own AI solutions, such as an AI chatbot, might integrate an LLM like Open AI’s GPT-4. However, using a closed-source LLM as part of a closed system can reduce access and data handling, including training data. 

Developers must consider how malicious parties might interfere with systems using third-party technology as the foundation for a custom AI. They may exploit vulnerable APIs with harmful commands, collect sensitive data, or spread destructive code. Companies can counter this by implementing strong access controls, performing thorough risk assessments, and educating employees on cybersecurity to mitigate data leaks. 

Companies can also secure their data by incorporating different privacy-enhancing techniques into their models, such as those that support anonymization. For instance, the AI could be trained to ignore or not remember prompts featuring keywords and built to prevent unintentional data leaks by regularly scrubbing its memory banks.

AI Proof-of-Concepts and Data Security

An AI Proof-of-Concept (PoC) is a limited, scaled-down version of an intended AI product. At FullStack, our clients use them to prove the feasibility of an AI concept before investing in full AI development. Developing a PoC before implementing a full-scale AI project can have a few benefits for securing sensitive data. 

From a security standpoint, the biggest benefit of PoC-first development is that PoCs are trained on a limited data set. Because PoCs are scaled down, the data set needed to train them is much smaller—and, therefore, less risky. Companies with critical security concerns, such as healthcare companies looking to implement AI, can develop an AI PoC first and train it on limited, redacted, or synthesized data. 

Then, the development team can assess and refine the system in a controlled environment without exposing the company to data security risks. Once the AI PoC has been refined and deemed compliant, the company may be able to scale its use. 

The Future of Digital Privacy

Generative AI is advancing by the day, and while this brings innovations, it also creates new privacy concerns that users must stay mindful of in the future. The ways that companies handle data will change as new tools are released and AI models update. By understanding these risks and taking proactive steps to address them, users can enjoy the advantages of AI without compromising their data security.

FullStack helps clients explore AI possibilities—without data security risks—with our AI PoC development. Contact us today to explore secure AI development and learn more about our proofs-of-concept.