TL;DR

Advanced persistent threats didn’t disappear; they moved closer to identity. In 2025, the most effective actors don’t always drop exotic malware - they live off the land inside your cloud tenant, harvest tokens, grant OAuth consents, and hide in mailboxes and service principals long enough to finish their objective. This field guide meets that reality head-on: it focuses on identity and SaaS-centric tradecraft, provides ATT&CK-mapped detections you can deploy, and lays out a pragmatic 90-day rollout your SecOps team can execute starting now.

What changed since 2023 (and why your defenses must, too)

Over the last 18–24 months, the center of gravity for serious intrusions shifted to identity and SaaS. Microsoft’s public write-ups of Midnight Blizzard (APT29) document the actor creating and consenting malicious OAuth apps to grant mailbox-wide access - persistence without a noisy implant. In one case, the actor used a legacy test app to obtain full_access_as_app (Exchange Online / Office 365) permissions across mailboxes, exemplifying the new playbook: compromise → consent → live quietly inside identity fabric. 

Meanwhile, Volt Typhoon campaigns highlighted how nation-state actors burrow into critical infrastructure using valid accounts and living-off-the-land (LOTL) techniques to maintain long-term, low-noise persistence - often in places with thin logging. U.S. government guidance explicitly calls LOTL a hallmark of Volt Typhoon’s tradecraft and provides operational mitigations defenders can adopt. And this isn’t unique to Volt Typhoon: a Bitdefender analysis of 700,000 incidents found 84% of high-severity attacks involved LOTL binaries, underscoring how widespread these tactics have become. 

On the destructive side, Sandworm/APT44 activity continued to blend espionage, sabotage, and information ops, with 2024–2025 reporting underscoring its integrated playbook and critical-infrastructure targeting. That matters because your tabletop scenarios should assume multi-vector pressure, not a tidy “single incident.”

Bottom line: the APT of 2025 is identity-centric, cloud-aware, and patient. Your defense must treat identity signals, Software-as-a-Service (SaaS) posture, and detections as first-class citizens alongside Endpoint Detection and Response (EDR) and Extended Detection and Response (XDR.)

APT lifecycle - rewritten for the identity era

The classic phases still apply, but each step now has an identity/SaaS flavor:

  • Initial access: password spray, consent phishing, supplier compromise, or token replay.
  • Establish persistence: OAuth application grants, service principals with excessive app roles, mailbox rules that hide replies. (No binary, no beacon - just platform features.)
  • Discovery & lateral movement: enumerate tenants, mailboxes, SharePoint/Drive, cloud roles; pivot across test/production or subsidiaries via OAuth apps.
  • Privilege escalation: abuse conditional access gaps, legacy auth, unmanaged device joins, or dormant high-privilege accounts.
  • Collection & exfiltration: mailbox search/export, staged file syncing, exfil via sanctioned apps to look “normal.”
  • Action on objectives: espionage, prepositioning for disruption, or financial fraud.

If your program still frames APTs as “EDR + patch” problems, you’ll miss the quiet parts.

Defender’s map: ATT&CK × identity & SaaS (high-value techniques)

Below are some examples and what to watch. Use them as seeds for your own coverage map.

ATT&CK technique

What it looks like in 2025

Signals to log & hunt

First moves to defend

T1550 – Use of Stolen Tokens

Replayed OAuth/refresh tokens from unusual IPs/devices

IdP sign-in logs, token sign-in telemetry, geo/ASN anomalies

Phishing-resistant MFA, Conditional Access device risk, revoke & reissue tokens

T1098 – Account Manipulation

New service principals, consented apps with excessive scopes

IdP audit: app registrations/grants; M365/Azure/Google Workspace admin logs

App consent policies, admin consent workflow, review dormant/high-privilege SPs

T1114 – Email CollectionBurst of mailbox rules (move/delete/forward), eDiscovery exportsExchange/Workspace audit: rule changes, Search-Mailbox/eDiscovery opsAlert on atypical rule creation; restrict eDiscovery; separate “investigator” roles
T1078 – Valid AccountsLOTL with legit creds on VPN/RDP/SSHVPN/Zero Trust network logs, device posture, unusual time/ASNPAM/JIT for admins; block legacy auth; enforce device compliance
T1021 – Remote ServicesCloud admin portals, remote powershell, SaaS admin APIsCloud admin API calls, unusual admin actions, new API tokensJust-enough admin, break-glass accounts with extra scrutiny, admin session recording
T1567 – Exfil over Web ServicesData pulled via sanctioned SaaS (Drive/SharePoint/OneDrive)DLP/SSPM logs: mass download/sync, public link creation, external sharesDLP policies, download/sync thresholds, sensitive label enforcement
T1036 – Masquerading / LotLBuilt-in tools, scheduled tasks, certutil/powershell blendsEDR command-line, script block logging, LOLBin usage

Constrain scripting, application control for admin tiers, PowerShell logging

The combination of malicious OAuth apps and LOTL is the signature of modern APT persistence. Build detections where your adversaries actually live - your IdP and SaaS logs.

Detection engineering you can actually ship

For each detection you get (a) the objective, (b) primary signals no matter your vendor, (c) where to pull those signals on Windows/macOS/Linux, and (d) an IdP/SaaS angle. Snippets use Sigma-style fields (generic) -  translate to your SIEM syntax.

1) Suspicious OAuth consent spike (new app + high-privilege scopes)

Objective: Catch consent-phishing/token-based persistence that bypasses endpoint controls.
Primary signals (any stack): New OAuth app registration or enterprise app install; high-scope grants; unusual granter; bursty approvals.

  • Windows/macOS/Linux: Mostly N/A (identity layer), but correlate with first-seen device or headless browser use on the approving host (EDR telemetry).

  • IdP/SaaS:


    • Entra/Okta/Google: Service/app creation, admin/user consent, high scopes (mail/files/all, offline_access, repo_admin, etc.).

    • GitHub/Slack/Atlassian: New OAuth apps, token grants with org-wide scopes.

Sigma-style sketch

title: OAuth Consent Spike With High-Risk Scopes
logsource: identity
detection:
  selection:
    event.category: ["iam","application"]
    event.action: ["app_consent_granted","oauth_app_installed","service_principal_created"]
    oauth.scopes|contains_any:
      - "Files.ReadWrite.All"
      - "Mail.ReadWrite"
      - "offline_access"
      - "repo"
      - "admin:*"
  timeframe: 30m
  condition: selection | count() by user.id > 2
fields: [user.id, app.id, oauth.scopes, source.ip, user.agent]
level: high

2) Mass mailbox rule creation/changes (multi-platform mail)

Objective: Find stealthy mail forwarding/hiding used for collection or BEC.
Primary signals: New/modified inbox rules, forwarding to external, “move to RSS/Archive,” auto-delete.

  • Windows/macOS/Linux: If using Outlook/Apple Mail/Thunderbird local rules, watch for changes in client config files + EDR network beacons immediately after.

  • IdP/SaaS (any email provider): Unified audit/mailbox audit logs: New-InboxRule/Update-InboxRule, external forward enabled, auto-forward policy violations, Gmail filters API changes.

Sigma-style sketch

title: Burst Of Suspicious Mailbox Rules
logsource: email
detection:
  selection:
    event.action: ["mailbox_rule_created","mailbox_rule_updated"]
    rule.actions|contains_any: ["forward_external","delete","move_to_archive","mark_read"]
  timeframe: 30m
  condition: selection | count() by mailbox.address >= 3
fields: [mailbox.address, rule.name, rule.conditions, rule.actions, actor.user, source.ip]
level: high

3) High-risk token use (impossible travel + privileged action)

Objective: Tie an anomalous sign-in to a sensitive admin/API action shortly after.
Primary signals: Geo/ASN/device anomalies; privileged API calls (create user, change roles, modify SSO/SCIM, rotate keys).

  • Windows: Event logs for new device sign-in + subsequent directory/privileged tooling (e.g., PowerShell Graph/AD modules).

  • macOS: Unified Logging entries for new browser/device; EDR network telemetry to IdP endpoints followed by admin APIs.

  • Linux: journald/auth logs for first-seen user-agent; CLI SDK calls (aws, gcloud, az) with admin actions.

  • IdP/SaaS: Correlate sign-in risk or impossible travel with “role_assignment,” “policy_update,” “admin_consent,” “SCIM_provision.”

Sigma-style sketch

title: Impossible Travel Followed By Privileged Action
logsource: identity
detection:
  signin_anom:
    event.action: "login_success"
    auth.risk: "impossible_travel"
  admin_action:
    event.action|contains_any: ["role_assignment","policy_update","key_rotation","create_service_principal"]
  timeframe: 15m
  condition: signin_anom by user.id then admin_action by user.id within 15m
fields: [user.id, source.geo, source.asn, target.resource, action.name]
level: critical

4) New service principal / robot identity with broad roles

Objective: Detect non-human identities gaining powerful org/tenant roles (hard to notice, very durable).
Primary signals: Creation of app/service account; assignment of org-wide roles or app roles; escalation of existing SP/robot.

  • Windows/macOS/Linux: Track local credential stores/CLI profiles (e.g., ~/.aws/credentials, ~/.config/gcloud/, Azure CLI token cache) created right after SP creation.

  • IdP/Cloud:
    • Entra/Okta/Google/AWS/IAM: create_app, create_service_principal, attach_role_policy, UpdateAssumeRolePolicy, iam.serviceAccount.addKey.
    • Kubernetes: ServiceAccount with ClusterRoleBinding.

Sigma-style sketch

title: Newly Created Non-Human Identity With Broad Privileges
logsource: cloud
detection:
  create_id:
    event.action|contains_any: ["create_service_principal","iam.createUser","iam.createRole","projects.serviceAccounts.create"]
  high_priv:
    privileges|contains_any: ["*","Owner","Administrator","ClusterAdmin","iam:*","ec2:*","s3:*"]
  condition: create_id and high_priv within 10m
fields: [actor.user, principal.id, privileges, source.ip]
level: high

5) Exfiltration via sanctioned SaaS (endpoint + SaaS together)

Objective: Catch “looks-legit” data staging/sync to approved tools at abnormal volume or timing.
Primary signals: Abnormal download/sync size or rate; creation of public links on sensitive libraries; off-hours spikes.

  • Windows/macOS/Linux (endpoint): EDR network flows to sanctioned SaaS domains with unusual byte counts; filesystem events indicating bulk reads of labeled files before sync.

  • SaaS: SharePoint/Google Drive/Box/Dropbox/Confluence: file download/create-link/audit events; DLP hits; sudden permission changes to “anyone with link.”

Sigma-style sketch

title: Sanctioned SaaS Data Exfil Pattern
logsource: saas
detection:
  burst_downloads:
    event.action|contains_any: ["file_download","file_sync"]
    bytes_out > 500000000   # 500MB (tune per org)
  public_link:
    event.action: "link_created"
    link.visibility: "public"
  timeframe: 1h
  condition: burst_downloads by user.id and public_link by user.id within 1h
fields: [user.id, saas.app, bytes_out, file.labels, link.url]
level: high

Platform signal cheat-sheet

  • Windows: Security/Event Logs (4624/4625/4648), PowerShell logs, Sysmon, EDR process + network trees.
  • macOS: Unified Logging (log show), EndpointSecurity/EDR events, keychain/launchd changes, new configuration profiles.
  • Linux: auditd/auditbeat, journald (_SYSTEMD_UNIT, SYSLOG_IDENTIFIER), shell history anomalies, package manager logs.
  • Identity/SSO: Okta System Log, Entra/Azure AD Audit/Sign-in, Google Workspace Admin, PingFederate, Duo, Auth0.
  • Cloud/IaC: AWS CloudTrail, Azure Activity/Graph, GCP Admin Activity, Kubernetes audit, GitHub/Azure DevOps/GitLab audit.
  • SaaS apps: M365/Google Drive/Box/Dropbox/Slack/Atlassian/GitHub audit streams; DLP/SSPM findings.
  • Email: Microsoft 365/Exchange Online, Gmail audit, Proofpoint/Mimecast logs, DMARC aggregate/forensic where available.

Tuning & deployment tips (non-Microsoft)

  • If you’re Okta-first, replace “consent events” with: app.oauth2.client.application.created, system.api_token.create, user.mfa.factor.deactivate, app.oauth2.scope.grant.
  • If you’re Google Workspace-first, monitor: CREATE_DATA_ACCESS_LEVEL, AUTHORIZE_API_CLIENT_ACCESS, Gmail CREATE_FILTER with forward/external actions.
  • On macOS fleets, lean on your EDR for network + process lineage around identity events and profile installs; add Configuration Profiles monitoring to flag unexpected SSO/MDM changes.
  • On Linux servers, add CLI SDK telemetry (aws/gcloud/az) and watch for first-seen credentials files + subsequent high-priv API calls.
  • Wherever possible, join identity + endpoint + SaaS: most of these detections “light up” when you correlate cross-source within short windows.

Control stack by maturity (choose your next step, not all steps)

You don’t need everything on day one. Build upward in deliberate stages, starting where the biggest gains come fastest - identity and detections - and then graduate to deeper coverage as your team and telemetry mature.

Level 1: Baselines that bite.
Get the fundamentals working for you every single day. Enforce MFA, moving to phishing-resistant methods where practical. Block legacy protocols that bypass modern protections. Use Conditional Access to evaluate user, device, and session risk before granting entry. Turn on audit logging across your identity provider, email, and core SaaS platforms, and light up alerts for mailbox-rule changes and new OAuth grants. Make reporting suspicious prompts a one-click action in mail and chat. Treat CISA’s living-off-the-land guidance as your minimum operating floor for configuration and hygiene.

Level 2: Identity-aware telemetry.
Once the basics hold, deepen visibility where today’s intrusions actually live: identity and SaaS. Stream IdP, email, and SaaS audit logs into your SIEM/XDR and add ITDR so you can reason about users, tokens, app grants, and privileged actions—not just endpoints. Deploy SSPM to surface risky OAuth consents and SaaS misconfigurations, and use EASM/CAASM to discover exposed assets and shadow IT. Prioritize quiet-signal detections: unusual OAuth scopes on new apps, bursts of mailbox rules, replayed tokens tied to admin API calls, newly created service principals gaining powerful roles, and anomalous downloads from labeled repositories. Pair detections with automatic containment steps—revoke tokens, disable suspect app grants, force re-auth—so response is measured in minutes.

Level 3: Operate like a hunter.
With telemetry in place, set explicit ATT&CK coverage goals and hunt the techniques that matter in your environment. Replace standing admin rights with tiered administration and just-in-time elevation so privileged sessions are rare, short, and heavily logged. Monitor service principals like crown jewels: owner verification, scope reviews, and alerting on privilege changes. Add targeted deception around high-value apps and identities to raise the cost of lateral movement. Run regular tabletop exercises that stress identity abuse paths, and feed what you learn back into detections and access policy.

Treat traditional endpoint controls and network monitoring as necessary - but recognize they’re not sufficient on their own for long-dwell, identity-centric campaigns. The combination that keeps pace in 2025 is an identity-first control plane, ITDR-driven detections, and disciplined operations with owners, SLAs, and audit-ready evidence.

A 90-day execution plan 

Days 0–15: Instrument & baseline
Name the three human-plus-identity risks you actually see. Pipe IdP, email, and cloud audit logs into your SIEM/XDR. Turn on mailbox-rule/OAuth-consent alerts. Record a 60-second exec kickoff so managers carry the message.

Days 16–45: Enforce & detect
Enforce Conditional Access baselines (block legacy, step-up for risky sign-ins). Publish the first two detections above and tune. Inventory OAuth apps and service principals; disable unused, review privileged ones with owners. Hold a 2-hour tabletop: consent phishing + mailbox rule abuse + exfil tasking.

Days 46–75: Hunt & harden
Add detections 3 - 5. Introduce JIT for admin roles. Enable DLP/SSPM policies for sensitive stores. Run a phishing drill focused on consent prompts; remediate kindly but quickly.

Days 76–90: Report & iterate
Ship a one-page brief: what detections fired, what you tuned, what identity posture improved (fewer privileged SPs, fewer risky OAuth grants), and what changed in MTTD/containment. Capture artifacts (screenshots/exports) into your audit pack.

Are we already in an APT? Read the room through identity and SaaS

Forget generic “symptom” lists. In long-dwell intrusions today, the tells are quiet and identity-shaped and they’re all visible if you collect the right logs. Start with application consents and service principals: a newly registered OAuth app that requests sweeping Graph scopes (think mail or files at tenant scale), or a service principal that appears with broad roles and no clear owner. That’s not hygiene; that’s a foothold.

Pivot to mail telemetry. Bursts of inbox rules that silently move or delete messages, sudden forwarding to external recipients, and eDiscovery or export activity that no one recalls initiating are classic signs of mailbox harvesting and staging for exfiltration. Correlate sign-ins with admin actions as well: a “valid account” login from an impossible location followed minutes later by privileged API calls is the living-off-the-land pattern in miniature - credentials first, admin work second.

Watch dormant privilege waking up. A long-idle global admin, app registration permission, or powerful service principal springing to life for a short window and then going quiet again often marks hands on a backdoor rather than a routine task. Finally, follow the data. In sanctioned SaaS like SharePoint, OneDrive, Google Drive, or Box, look for spikes in downloads or syncs from sensitive libraries - especially when they happen at odd hours or from unfamiliar networks. Network tools may shrug at this traffic; your SaaS and DLP logs will not.

If you can describe these signals with timestamps, actors, scopes/roles, and follow-on actions, and you can’t find a legitimate change ticket, you don’t just have “weirdness.” You have a high-confidence APT lead, and your next move is containment: revoke tokens, disable the suspect app grant, lock or rotate impacted accounts and service principals, and begin a wider hunt for the same patterns across tenants.

People and process: training that shrinks the blast radius

Technology alone won’t hold. The people closest to identity and SaaS must recognize and report the quiet indicators APTs produce. Teach everyone how consent prompts work, what a suspicious OAuth screen looks like, and where to report it immediately. For IT, cloud, and engineering, go beyond concepts: practice hardening identity providers, reviewing OAuth apps, locking down service principals, and responding to mailbox rule abuse in hands-on labs. For SOC, practice the detections above until response times drop.

Cybrary fits as your role-based learning and practice engine: enroll IT/cloud/dev/SOC personas in the paths and labs that mirror their jobs, pair phishing/tabletop exercises with relevant lessons as pre-work or remediation, and export completion/lab outcomes into your evidence pack. 

Build Your Quarterly Cyber Risk Snapshot

Make your quarterly update a crisp narrative that a director can absorb in two minutes. Start with what changed: describe how identity posture improved - fewer always-on admins, a smaller set of privileged service principals, and a reduction in risky OAuth consents. Note the expansion of detection coverage by naming the ATT&CK techniques you can now see reliably, and add one human signal that matters, such as a higher first-hour phishing-report rate.

Then translate that movement into operational risk. Explain, in plain language, that fewer valid-account intrusions are progressing to mailbox or data-access stages, and that the ones that do are being contained faster because alerts arrive earlier and with better context.

Close with what happens next. Commit to a small, concrete set of actions for the coming quarter - two new detections you’ll ship, one access-policy change you’ll enforce, and one exercise you’ll run - and give each an owner and a date.

Attach concise evidence, not a data dump: a link to your consent-log review, a screenshot of a tuned detection firing and being triaged, and the updated ATT&CK-by-identity matrix that shows coverage at a glance. Presented the same way every quarter, this single page becomes a trustworthy record of progress for both the board / auditors and it keeps your program focused on outcomes rather than activity.

Appendix: five starter detections (pseudocode to adapt)

New app consent with risky scopes

What it does: Flags users granting OAuth consent to a new application that requests powerful scopes (e.g., full mailbox/file access) within the last hour, excluding known privileged admin accounts.
Why it matters: Consent phishing and malicious app registration are common APT persistence paths that bypass MFA and endpoint tools.
How to read it:

  • Filters for “ConsentToApp” events and checks if requested scopes include high-risk permissions.
  • Excludes a safelisted admin group to reduce noise from legitimate admin activity.
  • Buckets by 1-hour windows and surfaces who approved what app and scopes.
    Tuning tips: Maintain an allowlist of known apps/scopes; consider alerting only on new apps or first-time consents per user.
    Investigate next: Validate the app publisher, review all granted scopes, revoke consent if suspicious, and hunt for token refresh activity from unfamiliar IPs/devices.
where Event == "ConsentToApp"
and Scopes has_any ("full_access_as_app","Mail.ReadWrite","Files.ReadWrite.All")
and Actor not_in (PrivilegedAdmins)
| summarize count() by Actor, AppId, Scopes, bin(TimeGenerated, 1h)

Mailbox rule storm

What it does: Detects bursts of inbox rule creation or changes on a single mailbox in a short period (e.g., >3 rules in 30 minutes).
Why it matters: Attackers often create rules to hide or forward messages (collection/BEC staging) without tripping login alerts.
How to read it:

  • Looks for “New-InboxRule” or “Set-InboxRule” operations.
  • Counts rule changes per user within a 30-minute window and flags unusually high activity.

Tuning tips:

  • Add rule-content filters (forward to external, move to RSS/Archive, delete) for higher fidelity.
  • Suppress during known migrations or when admins are bulk-standardizing rules.
  • Investigate next: List new/changed rules, destinations (external domains), and who/what performed the action (user vs service). Disable suspicious rules and review concurrent sign-ins.
where Operation in ("New-InboxRule","Set-InboxRule")
| summarize RuleCount = count() by UserId, bin(TimeGenerated, 30m)
| where RuleCount > 3

Impossible travel + privileged action

What it does: Detects bursts of inbox rule creation or changes on a single mailbox in a short period (e.g., >3 rules in 30 minutes).
Why it matters: Attackers often create rules to hide or forward messages (collection/BEC staging) without tripping login alerts.
How to read it:

  • Looks for “New-InboxRule” or “Set-InboxRule” operations.
  • Counts rule changes per user within a 30-minute window and flags unusually high activity.

Tuning tips:

  • Add rule-content filters (forward to external, move to RSS/Archive, delete) for higher fidelity.
  • Suppress during known migrations or when admins are bulk-standardizing rules.
  • Investigate next: List new/changed rules, destinations (external domains), and who/what performed the action (user vs service). Disable suspicious rules and review concurrent sign-ins.
let risky_signins = (SigninLogs | where RiskLevel >= "medium" | project UserId, TimeGenerated);
AdminActions
| lookup risky_signins on UserId
| where AdminActions.TimeGenerated between (risky_signins.TimeGenerated .. risky_signins.TimeGenerated + 15m)

New service principal with elevated role

What it does: Surfaces creation or modification events where a new application/service principal (non-human identity) is granted high-privilege roles or app permissions.
Why it matters: Robot identities with tenant-wide rights are stealthy persistence; they can access mail, files, or directories without interactive logins.
How to read it:

  • Watches directory audit logs for app role assignments.
  • Flags when the assigned role is powerful (e.g., Exchange.ManageAsApp, Directory.ReadWrite.All).

Tuning tips:

  • Maintain a registry of approved service principals and expected roles; alert only on new SPs or new high-priv grants.
  • Include who granted the role and from where to assist triage.
  • Investigate next: Validate business justification with the owner/team, rotate app secrets, and revoke the role if unapproved. Review follow-on API usage by that SP.
DirectoryAudit
| where Operation == "AddAppRoleAssignment"
and AppRole in ("Exchange.ManageAsApp","Directory.ReadWrite.All", "Mail.Read")

Exfil pattern via sanctioned SaaS

What it does: Detects potential data exfiltration through approved cloud storage by looking for unusually high download/sync volume of labeled-sensitive content in short windows.
Why it matters: APTs often use “allowed” apps (SharePoint/Drive/Box) to blend in while staging or exfiltrating data.

How to read it:

  • Aggregates file operations by user and sensitivity label over 1 hour.
  • Alerts when volume exceeds a threshold on “Confidential/Restricted” sites/libraries.

Tuning tips:

  • Calibrate the byte/volume threshold per team (engineering vs HR will differ).
  • Add conditions for public link created or sharing to external domains to increase fidelity.
  • Investigate next: Identify which files were accessed, sharing changes, time of day, and destination links. Temporarily suspend external sharing for the user/library and review endpoint activity leading up to the burst.
SharePointFileOperation
| summarize Downloads = count() by UserId, SiteSensitivityLabel, bin(TimeGenerated, 1h)
| where SiteSensitivityLabel in ("Confidential","Restricted") and Downloads > threshold

Tune thresholds, add allow-lists for known automations, and wire pages to analyst runbooks.

Conclusion

APTs are not a malware genre; they are an operational style. In 2025, that style is defined by identity: consent prompts, tokens, service principals, and SaaS admin APIs - plus the old LOTL discipline that keeps noise low. If you log where the adversary lives, ship detections that fit your tenant, harden identity in 90 days, and train the people nearest the blast radius to spot the quiet tells, you’ll make serious intrusions expensive again.

Ready to help your team level up? Find out how Cybrary can get you there. Request a demo, today.

The Open Worldwide Application Security Project (OWASP) is a community-led organization and has been around for over 20 years and is largely known for its Top 10 web application security risks (check out our course on it). As the use of generative AI and large language models (LLMs) has exploded recently, so too has the risk to privacy and security by these technologies. OWASP, leading the charge for security, has come out with its Top 10 for LLMs and Generative AI Apps this year. In this blog post we’ll explore the Top 10 risks and explore examples of each as well as how to prevent these risks.

LLM01: Prompt Injection

Those familiar with the OWASP Top 10 for web applications have seen the injection category before at the top of the list for many years. This is no exception with LLMs and ranks as number one. Prompt Injection can be a critical vulnerability in LLMs where an attacker manipulates the model through crafted inputs, leading it to execute unintended actions. This can result in unauthorized access, data exfiltration, or social engineering. There are two types: Direct Prompt Injection, which involves "jailbreaking" the system by altering or revealing underlying system prompts, giving an attacker access to backend systems or sensitive data, and Indirect Prompt Injection, where external inputs (like files or web content) are used to manipulate the LLM's behavior.

As an example, an attacker might upload a resume containing an indirect prompt injection, instructing an LLM-based hiring tool to favorably evaluate the resume. When an internal user runs the document through the LLM for summarization, the embedded prompt makes the LLM respond positively about the candidate’s suitability, regardless of the actual content.

How to prevent prompt injection:

  1. Limit LLM Access: Apply the principle of least privilege by restricting the LLM's access to sensitive backend systems and enforcing API token controls for extended functionalities like plugins.
  2. Human Approval for Critical Actions: For high-risk operations, require human validation before executing, ensuring that the LLM's suggestions are not followed blindly.
  3. Separate External and User Content: Use frameworks like ChatML for OpenAI API calls to clearly differentiate between user prompts and untrusted external content, reducing the chance of unintentional action from mixed inputs.
  4. Monitor and Flag Untrusted Outputs: Regularly review LLM outputs and mark suspicious content, helping users to recognize potentially unreliable information.

LLM02: Insecure Output Handling

Insecure Output Handling occurs when the outputs generated by a LLM are not properly validated or sanitized before being used by other components in a system. Since LLMs can generate various types of content based on input prompts, failing to handle these outputs securely can introduce risks like cross-site scripting (XSS), server-side request forgery (SSRF), or even remote code execution (RCE). Unlike Overreliance (LLM09), which focuses on the accuracy of LLM outputs, Insecure Output Handling specifically addresses vulnerabilities in how these outputs are processed downstream.

As an example, there could be a web application that uses an LLM to summarize user-provided content and renders it back in a webpage. An attacker submits a prompt containing malicious JavaScript code. If the LLM’s output is displayed on the webpage without proper sanitization, the JavaScript will execute in the user’s browser, leading to XSS. Alternatively, if the LLM’s output is sent to a backend database or shell command, it could allow SQL injection or remote code execution if not properly validated.

How to prevent Insecure Output Handling:

  1. Zero-Trust Approach: Treat the LLM as an untrusted source, applying strict allow list validation and sanitization to all outputs it generates, especially before passing them to downstream systems or functions.
  2. Output Encoding: Encode LLM outputs before displaying them to end users, particularly when dealing with web content where XSS risks are prevalent.
  3. Adhere to Security Standards: Follow the OWASP Application Security Verification Standard (ASVS) guidelines, which provide strategies for input validation and sanitization to protect against code injection risks.

LLM03: Training Data Poisoning

Training Data Poisoning refers to the manipulation of the data used to train LLMs, introducing biases, backdoors, or vulnerabilities. This tampered data can degrade the model's effectiveness, introduce harmful biases, or create security flaws that malicious actors can exploit. Poisoned data could lead to inaccurate or inappropriate outputs, compromising user trust, harming brand reputation, and increasing security risks like downstream exploitation.

As an example, there could be a scenario where an LLM is trained on a dataset that has been tampered with by a malicious actor. The poisoned dataset includes subtly manipulated content, such as biased news articles or fabricated facts. When the model is deployed, it may output biased information or incorrect details based on the poisoned data. This not only degrades the model’s performance but can also mislead users, potentially harming the model’s credibility and the organization’s reputation.

How to prevent Training Data Poisoning:

  1. Data Validation and Vetting: Verify the sources of training data, especially when sourcing from third-party datasets. Conduct thorough checks on data integrity, and where possible, use trusted data sources.
  2. Machine Learning Bill of Materials (ML-BOM): Maintain an ML-BOM to track the provenance of training data and ensure that each source is legitimate and suitable for the model’s purpose.
  3. Sandboxing and Network Controls: Restrict access to external data sources and use network controls to prevent unintended data scraping during training. This helps ensure that only vetted data is used for training.
  4. Adversarial Robustness Techniques: Implement strategies like federated learning and statistical outlier detection to reduce the impact of poisoned data. Periodic testing and monitoring can identify unusual model behaviors that may indicate a poisoning attempt.
  5. Human Review and Auditing: Regularly audit model outputs and use a human-in-the-loop approach to validate outputs, especially for sensitive applications. This added layer of scrutiny can catch potential issues early.

LLM04: Model Denial of Service

Model Denial of Service (DoS) is a vulnerability in which an attacker deliberately consumes an excessive amount of computational resources by interacting with a LLM. This can result in degraded service quality, increased costs, or even system crashes. One emerging concern is manipulating the context window of the LLM, which refers to the maximum amount of text the model can process at once. This makes it possible to overwhelm the LLM by exceeding or exploiting this limit, leading to resource exhaustion.

As an example, an attacker may continuously flood the LLM with sequential inputs that each reach the upper limit of the model’s context window. This high-volume, resource-intensive traffic overloads the system, resulting in slower response times and even denial of service. As another example, if an LLM-based chatbot is inundated with a flood of recursive or exceptionally long prompts, it can strain computational resources, causing system crashes or significant delays for other users.

How to prevent Model Denial of Service:

  1. Rate Limiting: Implement rate limits to restrict the number of requests from a single user or IP address within a specific timeframe. This reduces the chance of overwhelming the system with excessive traffic.
  2. Resource Allocation Caps: Set caps on resource usage per request to ensure that complex or high-resource requests do not consume excessive CPU or memory. This helps prevent resource exhaustion.
  3. Input Size Restrictions: Limit input size according to the LLM's context window capacity to prevent excessive context expansion. For example, inputs exceeding a predefined character limit can be truncated or rejected.
  4. Monitoring and Alerts: Continuously monitor resource utilization and establish alerts for unusual spikes, which may indicate a DoS attempt. This allows for proactive threat detection and response.
  5. Developer Awareness and Training: Educate developers about DoS vulnerabilities in LLMs and establish guidelines for secure model deployment. Understanding these risks enables teams to implement preventative measures more effectively.

LLM05: Supply Chain Vulnerabilities

Supply Chain attacks are incredibly common and this is no different with LLMs, which, in this case refers to risks associated with the third-party components, training data, pre-trained models, and deployment platforms used within LLMs. These vulnerabilities can arise from outdated libraries, tampered models, and even compromised data sources, impacting the security and reliability of the entire application. Unlike traditional software supply chain risks, LLM supply chain vulnerabilities extend to the models and datasets themselves, which may be manipulated to include biases, backdoors, or malware that compromises system integrity.

As an example, an organization uses a third-party pre-trained model to conduct economic analysis. If this model is poisoned with incorrect or biased data, it could generate inaccurate results that mislead decision-making. Additionally, if the organization uses an outdated plugin or compromised library, an attacker could exploit this vulnerability to gain unauthorized access or tamper with sensitive information. Such vulnerabilities can result in significant security breaches, financial loss, or reputational damage.

How to prevent Supply Chain Vulnerabilities:

  1. Vet Third-Party Components: Carefully review the terms, privacy policies, and security measures of all third-party model providers, data sources, and plugins. Use only trusted suppliers and ensure they have robust security protocols in place.
  2. Maintain a Software Bill of Materials (SBOM): An SBOM provides a complete inventory of all components, allowing for quick detection of vulnerabilities and unauthorized changes. Ensure that all components are up-to-date and apply patches as needed.
  3. Use Model and Code Signing: For models and external code, employ digital signatures to verify their integrity and authenticity before use. This helps ensure that no tampering has occurred.
  4. Anomaly Detection and Robustness Testing: Conduct adversarial robustness tests and anomaly detection on models and data to catch signs of tampering or data poisoning. Integrating these checks into your MLOps pipeline can enhance overall security.
  5. Implement Monitoring and Patching Policies: Regularly monitor component usage, scan for vulnerabilities, and patch outdated components. For sensitive applications, continuously audit your suppliers’ security posture and update components as new threats emerge.

LLM06: Sensitive Information Disclosure

Sensitive Information Disclosure in LLMs occurs when the model inadvertently reveals private, proprietary, or confidential information through its output. This can happen due to the model being trained on sensitive data or because it memorizes and later reproduces private information. Such disclosures can result in significant security breaches, including unauthorized access to personal data, intellectual property leaks, and violations of privacy laws.

As an example, there could be an LLM-based chatbot trained on a dataset containing personal information such as users’ full names, addresses, or proprietary business data. If the model memorizes this data, it could accidentally reveal this sensitive information to other users. For instance, a user might ask the chatbot for a recommendation, and the model could inadvertently respond with personal information it learned during training, violating privacy rules.

How to prevent Sensitive Information Disclosure:

  1. Data Sanitization: Before training, scrub datasets of personal or sensitive information. Use techniques like anonymization and redaction to ensure no sensitive data remains in the training data.
  2. Input and Output Filtering: Implement robust input validation and sanitization to prevent sensitive data from entering the model’s training data or being echoed back in outputs.
  3. Limit Training Data Exposure: Apply the principle of least privilege by restricting sensitive data from being part of the training dataset. Fine-tune the model with only the data necessary for its task, and ensure high-privilege data is not accessible to lower-privilege users.
  4. User Awareness: Make users aware of how their data is processed by providing clear Terms of Use and offering opt-out options for having their data used in model training.
  5. Access Controls: Apply strict access control to external data sources used by the LLM, ensuring that sensitive information is handled securely throughout the system

LLM07: Insecure Plugin Design

Insecure Plugin Design vulnerabilities arise when LLM plugins, which extend the model’s capabilities, are not adequately secured. These plugins often allow free-text inputs and may lack proper input validation and access controls. When enabled, plugins can execute various tasks based on the LLM’s outputs without further checks, which can expose the system to risks like data exfiltration, remote code execution, and privilege escalation. This vulnerability is particularly dangerous because plugins can operate with elevated permissions while assuming that user inputs are trustworthy.

As an example, there could be a weather plugin that allows users to input a base URL and query. An attacker could craft a malicious input that directs the LLM to a domain they control, allowing them to inject harmful content into the system. Similarly, a plugin that accepts SQL “WHERE” clauses without validation could enable an attacker to execute SQL injection attacks, gaining unauthorized access to data in a database.

How to prevent Insecure Plugin Design:

  1. Enforce Parameterized Input: Plugins should restrict inputs to specific parameters and avoid free-form text wherever possible. This can prevent injection attacks and other exploits.
  2. Input Validation and Sanitization: Plugins should include robust validation on all inputs. Using Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) can help identify vulnerabilities during development.
  3. Access Control: Follow the principle of least privilege, limiting each plugin's permissions to only what is necessary. Implement OAuth2 or API keys to control access and ensure only authorized users or components can trigger sensitive actions.
  4. Manual Authorization for Sensitive Actions: For actions that could impact user security, such as transferring files or accessing private repositories, require explicit user confirmation.
  5. Adhere to OWASP API Security Guidelines: Since plugins often function as REST APIs, apply best practices from the OWASP API Security Top 10. This includes securing endpoints and applying rate limiting to mitigate potential abuse.

LLM08: Excessive Agency

Excessive Agency in LLM-based applications arises when models are granted too much autonomy or functionality, allowing them to perform actions beyond their intended scope. This vulnerability occurs when an LLM agent has access to functions that are unnecessary for its purpose or operates with excessive permissions, such as being able to modify or delete records instead of only reading them. Unlike Insecure Output Handling, which deals with the lack of validation on the model’s outputs, Excessive Agency pertains to the risks involved when an LLM takes actions without proper authorization, potentially leading to confidentiality, integrity, and availability issues.

As an example, there could be an LLM-based assistant that is given access to a user's email account to summarize incoming messages. If the plugin that is used to read emails also has permissions to send messages, a malicious prompt injection could trick the LLM into sending unauthorized emails (or spam) from the user's account.

How to prevent Excessive Agency:

  1. Restrict Plugin Functionality: Ensure plugins and tools only provide necessary functions. For example, if a plugin is used to read emails, it should not include capabilities to delete or send emails.
  2. Limit Permissions: Follow the principle of least privilege by restricting plugins’ access to external systems. For instance, a plugin for database access should be read-only if writing or modifying data is not required.
  3. Avoid Open-Ended Functions: Avoid functions like “run shell command” or “fetch URL” that provide broad system access. Instead, use plugins that perform specific, controlled tasks.
  4. User Authorization and Scope Tracking: Require plugins to execute actions within the context of a specific user's permissions. For example, using OAuth with limited scopes helps ensure actions align with the user’s access level.
  5. Human-in-the-Loop Control: Require user confirmation for high-impact actions. For instance, a plugin that posts to social media should require the user to review and approve the content before it is published.
  6. Authorization in Downstream Systems: Implement authorization checks in downstream systems that validate each request against security policies. This prevents the LLM from making unauthorized changes directly.

LLM09: Overreliance

Overreliance occurs when users or systems trust the outputs of a LLM without proper oversight or verification. While LLMs can generate creative and informative content, they are prone to “hallucinations” (producing false or misleading information) or providing authoritative-sounding but incorrect outputs. Overreliance on these models can result in security risks, misinformation, miscommunication, and even legal issues, especially if LLM-generated content is used without validation. This vulnerability becomes especially dangerous in cases where LLMs suggest insecure coding practices or flawed recommendations.

As an example, there could be a development team using an LLM to expedite the coding process. The LLM suggests an insecure code library, and the team, trusting the LLM, incorporates it into their software without review. This introduces a serious vulnerability. As another example, a news organization might use an LLM to generate articles, but if they don’t validate the information, it could lead to the spread of disinformation.

How to prevent Overreliance:

  1. Regular Monitoring and Review: Implement processes to review LLM outputs regularly. Use techniques like self-consistency checks or voting mechanisms to compare multiple model responses and filter out inconsistencies.
  2. Cross-Verification: Compare the LLM’s output with reliable, trusted sources to ensure the information’s accuracy. This step is crucial, especially in fields where factual accuracy is imperative.
  3. Fine-Tuning and Prompt Engineering: Fine-tune models for specific tasks or domains to reduce hallucinations. Techniques like parameter-efficient tuning (PET) and chain-of-thought prompting can help improve the quality of LLM outputs.
  4. Automated Validation: Use automated validation tools to cross-check generated outputs against known facts or data, adding an extra layer of security.
  5. Risk Communication: Clearly communicate the limitations of LLMs to users, highlighting the potential for errors. Transparent disclaimers can help manage user expectations and encourage cautious use of LLM outputs.
  6. Secure Coding Practices: For development environments, establish guidelines to prevent the integration of potentially insecure code. Avoid relying solely on LLM-generated code without thorough review.

LLM10: Model Theft

Model Theft refers to the unauthorized access, extraction, or replication of proprietary LLMs by malicious actors. These models, containing valuable intellectual property, are at risk of exfiltration, which can lead to significant economic and reputational loss, erosion of competitive advantage, and unauthorized access to sensitive information encoded within the model. Attackers may steal models directly from company infrastructure or replicate them by querying APIs to build shadow models that mimic the original. As LLMs become more prevalent, safeguarding their confidentiality and integrity is crucial.

As an example, an attacker could exploit a misconfiguration in a company’s network security settings, gaining access to their LLM model repository. Once inside, the attacker could exfiltrate the proprietary model and use it to build a competing service. Alternatively, an insider may leak model artifacts, allowing adversaries to launch gray box adversarial attacks or fine-tune their own models with stolen data.

How to prevent Model Theft:

  1. Access Controls and Authentication: Use Role-Based Access Control (RBAC) and enforce strong authentication mechanisms to limit unauthorized access to LLM repositories and training environments. Adhere to the principle of least privilege for all user accounts.
  2. Supplier and Dependency Management: Monitor and verify the security of suppliers and dependencies to reduce the risk of supply chain attacks, ensuring that third-party components are secure.
  3. Centralized Model Inventory: Maintain a central ML Model Registry with access controls, logging, and authentication for all production models. This can aid in governance, compliance, and prompt detection of unauthorized activities.
  4. Network Restrictions: Limit LLM access to internal services, APIs, and network resources. This reduces the attack surface for side-channel attacks or unauthorized model access.
  5. Continuous Monitoring and Logging: Regularly monitor access logs for unusual activity and promptly address any unauthorized access. Automated governance workflows can also help streamline access and deployment controls.
  6. Adversarial Robustness: Implement adversarial robustness training to help detect extraction queries and defend against side-channel attacks. Rate-limit API calls to further protect against data exfiltration.
  7. Watermarking Techniques: Embed unique watermarks within the model to track unauthorized copies or detect theft during the model’s lifecycle.

Wrapping it all up

As LLMs continue to grow in capability and integration across industries, their security risks must be managed with the same vigilance as any other critical system. From Prompt Injection to Model Theft, the vulnerabilities outlined in the OWASP Top 10 for LLMs highlight the unique challenges posed by these models, particularly when they are granted excessive agency or have access to sensitive data. Addressing these risks requires a multifaceted approach involving strict access controls, robust validation processes, continuous monitoring, and proactive governance.

For technical leadership, this means ensuring that development and operational teams implement best practices across the LLM lifecycle starting from securing training data to ensuring safe interaction between LLMs and external systems through plugins and APIs. Prioritizing security frameworks such as the OWASP ASVS, adopting MLOps best practices, and maintaining vigilance over supply chains and insider threats are key steps to safeguarding LLM deployments. Ultimately, strong leadership that emphasizes security-first practices will protect both intellectual property and organizational integrity, while fostering trust in the use of AI technologies.

Start learning with Cybrary

Create a free account

Related Posts

All Blogs