ChatGPT, Copilot, and Your Data: What's Actually at Risk?
Your team is probably using ChatGPT and Copilot right now. Here's what data risks you should actually worry about.
Written and reviewed by The Technology Office · Independent technology advisory
Your team is already using ChatGPT and Copilot. You probably don't have a formal policy about it. And you might not realize what data risks you've created.
Here's what's actually at risk—and what you can do about it.
The Reality of Shadow AI
Most organizations don't know how much generative AI is being used.
Common scenarios:
- Engineers using ChatGPT to write code
- Marketers using ChatGPT to draft copy
- Finance using Copilot to analyze data
- Sales using ChatGPT to write proposals
- Support teams using Copilot for response suggestions
All happening without formal approval or governance.
This is "shadow AI"—and it creates real risks.
What Data Gets Exposed
Confidential Information
When someone pastes customer data, financial information, or proprietary code into ChatGPT, where does that data go?
Answer: OpenAI's servers. And OpenAI uses it to train their models (unless you've opted into a paid enterprise plan).
Risk: Your confidential information becomes part of a public AI model. Competitors could get access to it. Regulators could question it.
Customer Data
If your team uploads customer information (names, emails, purchase history, health data) to ChatGPT, you could be violating privacy laws.
Risk: GDPR fines (up to 4% of revenue). Privacy Act violations. Customer lawsuits.
Employee Information
HR teams sometimes use ChatGPT to analyze employee data or write performance reviews.
Risk: Employee privacy violations. Discrimination risk if AI output is biased.
Code and Intellectual Property
Engineers using ChatGPT to write code might be uploading proprietary code to train OpenAI's models.
Risk: Loss of IP protection. Competitors getting access to your code. License violation risk (if your code is GPL or similar).
Security and Access Information
Someone might paste API keys, passwords, or system access information into ChatGPT for troubleshooting.
Risk: Complete security breach. Attackers now have access to your systems.
What You Can't Control
Even if you restrict data access, you have limited control over OpenAI:
- OpenAI changes terms regularly (and has expanded what they do with your data)
- OpenAI is not liable if your data is exposed (their terms say so)
- OpenAI shares data with third parties (depending on your plan)
- OpenAI's data retention policies are opaque (where is your data stored, for how long?)
Enterprise plans give you better control, but they're expensive and still don't give you full visibility.
The Privacy Law Problem
GDPR (Europe)
If you process any European customer data, GDPR applies. Using ChatGPT without consent or proper data processing agreements could be a violation.
Fine: Up to 4% of annual revenue or €20 million, whichever is higher.
Privacy Act (Australia)
Similar requirements around customer data. If you're handling Australian customer data, you need proper safeguards.
HIPAA (Healthcare)
If you're in healthcare and you use ChatGPT with patient data, that's a violation.
PCI-DSS (Payment Data)
If you handle payment data, you can't use ChatGPT with that data.
Most organizations are violating at least one of these without realizing it.
What Good Governance Looks Like
1. AI Usage Policy
Clear policy on what tools are approved and how they can be used.
Not: "ChatGPT is banned." Better: "ChatGPT is approved for drafting and brainstorming, NOT for customer data, confidential information, or code. Use the enterprise version only."
2. Data Classification
Label your data by sensitivity:
- Public (can be used with any tool)
- Internal (limited tools only)
- Confidential (restricted to authorized systems)
- Regulated (strict controls, no public AI tools)
3. Approved Tools
Maintain a list of approved AI tools and their approved uses.
Include data access restrictions for each tool.
4. Training
Train staff on what data can and can't be used with generative AI.
One mistake (pasting customer data into ChatGPT) can trigger legal exposure.
5. Monitoring
Monitor tool usage. You probably have unauthorized tools in use right now.
6. Privacy Impact Assessment
For any new AI tool, assess privacy risks before approving it.
What to Do Right Now
Immediate (This Week)
- Ask your team: What generative AI tools are you using? (Get honest answers, don't punish people.)
- Map what data is being used with these tools.
- Identify compliance gaps.
Short-term (This Month)
- Draft a generative AI usage policy.
- Classify your data by sensitivity.
- Choose approved tools (including enterprise versions if needed).
- Train staff.
Ongoing
- Monitor usage.
- Review quarterly.
- Update policy as tools and regulations evolve.
The Real Risk
The risk isn't that generative AI is inherently unsafe. The risk is that you're using it without governance.
Good governance doesn't ban AI. It enables safe AI use.
Most organizations are somewhere between "completely unmanaged" and "well-governed." If you're unsure whether you have proper governance in place, that's worth investigating.
One unmanaged incident—employee posting customer data to ChatGPT, for example—could cost you millions in regulatory fines and reputation damage.
Getting governance right now is much cheaper than dealing with the consequences later.
Mentioned services
These service and regional NSW pages expand on the topics covered in this article.
Need help applying this?
Bring in senior technology leadership without the full-time overhead.
The Technology Office works with Sydney and regional NSW businesses on embedded CIO support, Head of IT leadership, governance, vendor management, cost optimisation, crisis stabilisation, and no-cost technology reviews as a lighter first step.