(outline as of 9/13/2015)

Course Title:Cryptography

Section 1-1 Topic:Introduction To Cryptography

Learning Objectives:This introduction video provides an outline of the curriculum covered in Cybrary's in-depth Cryptography course.

Confidentiality, Integrity, Availability and Authentication

How companies use cryptography

How penetration testers can also use cryptography

Tools, Techniques and Attacks

Academics discusses history of encryption.

Academics simply encrypt or hash, why aren't people using hashing more?

Professional is just, encrypt or not, hash or just verify

Section 1-2 Topic:Key Concepts of Cryptography:PKIS & Encryption

Learning Objectives:Cryptography is one of the most underrated courses of study in the industry. Of those who do study it, issues with comprehension tend to hinder individual mastery. By taking this course, learners will finally be able to grasp all the critical concepts, theories and practices associated with Cryptography.

This Cryptography presentation discusses and demonstrates the key concepts of Cryptography from attacks, PKIs and Encryption in detail.

You’ll learn about the difference between public and private keys and about the similarities and differences between symmetry & asymmetry. We’ll also discuss the concept of integrity and confidentiality and their relationships to/with protocols.

This Cybrary Cryptography course will help you master the basics of Cryptography as you begin to develop the discipline needed to become an accomplished pen tester.

Keys and Principles

1.       Keys

1a. Symmetric - also referred to as same keys, private key, symmetric key - same (confidentially)

Different Version/Ways/Procedures to get PlainText to CipherText

Symmetric Cipher Examples:

AES, DES, 3DES, IDEA, CAST, twofish, Blowfish, RC2, Serpent, Rijndael - (there are others, chart at end of section)

PGP, OTP (One Time password)

Mnemonic device to remember some Symmetric Ciphers (how it reads, sounds and then corresponds, in English):

Hey! These 3 guys, had an idea to cast their rod to fish for blowfish, but instead received, serpents in the rain.

A! Dees 3 guys, had an IDEA to CAST their rod TWOFISH for BLOWFISH, but instead Received, Serpents in the Rain.

AES DES 3DES                 IDEA      CAST                 TWOFISH        BLOWFISH                       RC2           Serpent          Rijndael

1b. Asymmetric - public/private key pair - what one key does, the other key undoes

(confidentially, authentication, nonrepudiation)

Asymmetric Cipher Examples:

DH (key exchange), El Gamal (open source), RSA (internet), Elliptical Curve (for limited processor machines),

Knapsack, DSA (popular as a US standard)

Mnemonic device to remember some Asymmetric Ciphers (how it reads and then corresponds, in English):

D-E-R-E-K-S

Diffie Hellman, El Gamal, RSA, Elliptical Curve, Knapsack, DSA

2.       Principles (C, I, A)

Principles of CIA (confidentiality, integrity and authentication) along with PAIN will be discussed in detail later, focus on concepts and definitions.

Substitution/Transposition - PlainText to CipherText, and CipherText to PlainText

Disk Encryption (further detail later)

Integrity

Detect unauthorized change via output of Message Digest, also known as, a hash.

MD5 (128bit), SHA (160 bit, also variable), Hashing Calculator

PKI

A public key infrastructure (PKI) is a set of hardware, software, people, policies, and procedures needed to create, manage, distribute, use, store, and revoke digital certificates and manage public-key encryption.

CA, RA, X.509

IKE (internet Key exchange), ISAKMP, CRL (certification revocation list), OCSP (dynamic, checks x.509 cerrtification)

Attacks

Known Plaintext

Known CipherText

Chosen CipherText (needs physical access to machine)

Chosen Plaintext (choose what gets decrypted, needs physical access to)

Protocols

SSH (replaced Telnet)

HTTPS (SSL/TLS with security protocol, wraps all upper layer application in cryptographic wrapper)

IPSec (Network Layer, analogy truck:authentication header/integrity check, payload:integrity check and confidentiality check)

SSL/TLS (protocols)

Section 1-3 Topic:Introduction to Cryptographic Tools

Section 1-4 Topic:Services & Threats of Cryptography

Section 2-1 Topic:Integrity Checking:Modifications

Learning Objectives:These videos, while separate, build on each other and are all linked.

Explain how cryptography is used to protect CIA.  ‘CIA’ can also be a ‘PAIN’ – all defined…

1. Confidentiality (C of CIA) (also Privacy, P of PAIN) - prevent unauthorized disclosure of data

Threats

a. Social Engineering:Opening Unsigned Emails (Spear Phishing, Whaling Attack)

Defense Controls:Physical, Technical, Mail Filter, Administrative Training

Ideas:Separation of Duties, Need to Know

b. Media Reuse:USB Thumb Drive reuse is common.

Sanitizing can also be referred to as Zero'ing

Destroying a hard drive, completely, requires incineration.

Degaussing method only works for Magnetic Media.

Deleting only removes pointer to file and Formatting can be recovered.

c. Eavesdropping:protocol (packet) analyzer/ sniffer. Wireshark is popular.

Replay Attack:Capturing a packet that contains a password and reusing;

This allows access to ‘use’ a password without never knowing the actual password

Zero Knowledge Proof:Kerberos allows end user to prove the password, without sending

               password across network

Hack Attack Discussion:Directory Services:Query DNS, reply to send Credential to Server 1

2. Integrity (I of CIA, I of PAIN)

Accidental Modifications are checked by message digests, or hashes.

(digital representation of file - checks if identical)

Intentional (Malicious) Modification can be checked by using:

Message Authentication Code (MAC) or

Digital Signatures (both provide an extra step of authentication)

3. Authenticity (A of CIA, A of PAIN)

How to prove is truly came from the sender, is it valid?

4. Non repudiation (N of PAIN)

Combination of Integrity and Authenticity – must have both!

Sender can't dispute sending the message or contents - we use digital signatures for this

(further detailed explanations will follow on how)

Mnemonic Device, some remember it as CIA, or PAIN - P(rivacy) A(uthentication) I(ntegrity) N(on Repudiation)

 Section 2-2 Topic:CipherText

Learning Objectives:Learn how to properly create CipherText and the processes behind it in this lecture series on Cryptography.

Privacy/Confidentiality

Plaintext + Initialization Vector + Algorithm + Key = CipherText

Plaintext is just plain, non-encrypted text or data.

Initialization Vector (IV) adds randomness and complexity.  Randomized Starting Point

~Computer cannot generate a random number, so we have a pseudo random number.

~We could take any number of variable pieces of data to generate a pseudo random number such as:

A Date:12/12/1999 would be 34.

Algorithm (aka Cipher, is a collection of math functions)

               ~Symmetric Cryptography can also be referred to as:

                               Private Key

                               Secret Key

                               Session Key

                               Shared Key (i.e. shared between two parties)

               ~Any Symmetric Algorithm, to be discussed later, will be either:

                               Block

                               Stream

                ~Asymmetric

                               Discrete logarithms

                               Factorization

Key (Crypto Variable) is the instruction on how to use the algorithm (how to use the math)

Section 2-3 Topic:Master Pattern Analysis with CrypTool

Learning Objectives:Learn how to perform pattern analysis with CryptTool to identify trends during the encryption process.

CrypTool Lab

CrypTool is a non-academic analysis tool that offers a fundamental understanding of encryption algorithms.

Can be found at:https://www.cryptool.org

Section 2-4 Topic:Symmetric Encryption

Learning Objectives:The Symmetric Encryption discussion opens by reviewing the usage of random numbers, explaining what Replay Attack is, the importance of unique information on data packets and in sequencing and defining key terms including Nonce and Key Generation.

Then, you’re introduced to Symmetric Encryption with a basic summary of how it works, what ciphers are, the different types of Symmetric Ciphers, what happens with block ciphers, and what the difference means to encryption and decryption as a process.

As a penetration tester, you must know the difference between confusion and diffusion.  You’ll observe how that correlates to how algorithms work and the importance of strong math in configuring algorithms.  And you’ll learn what happens as algorithms are used and what you need to look for in your testing results.

Cryptographic Uses of Random Numbers

               ~There is very little true randomness (or entropy) so instead we use pseudo random.

               Nonces - process of stamping uniqueness on packet without being sequential.

                               Note:Adds uniqueness and helps to stop replay attacks

               Key Generation - keys must be unique, (pseudo) random and of a suitable length

                               Note:All things being equal, longer will offer more protection.

Symmetric Encryption (same key on both ends - real world example:a house key)

               Ciphers

                               Confusion - Confusion relates to complex math and substitution

                               Diffusion - Diffusion relates to permutations or rounds

                               (DES went through 16 rounds, 3DES goes through 48 times/rounds/permutations)

                                Bit length is not necessarily the final determination of the strength of a key.

                               Open - Kerckhoffs Principles, algorithm should be open

                               Key Generation – (definition above)

Example Algorithm:

[R1]+2, [R2]-2, [R3]*2, [R4]/2, ... [R6]^2 - each [R] would be considered an SBox (or substitution box)

                Block Ciphers (slower, but more secure)

                               ECB

                               CBC

                               DES

                               3DES (secure, very process intensive, but AES would be preferred)

                               AES (Advanced Encryption Standard)

Section 2-5 Topic:Stream Ciphers, XOR and RC4

Learning Objectives:As a continuation of learning about Symmetric Encryption, this lesson explores the other type of cipher:Stream Ciphers.

We’ll go much deeper into what happens during XOR (which stands for exclusive Or), what transposition and substitution is, and whether or not multiple transposition and substitution is preferred. You’ll also observe a rich demonstration of how XOR provides encryption, learn what RC4 is and what takes place at the encryption and decryption level.

As a Penetration Tester, you must understand the key functions of each of these events, what their impact is on each other and how to determine when to use one over another in Symmetric Encryption.

StreamCiphers

               One bit at a time

                               Transposition (THE becomes HET; moving over 1 character)

                               Substitution (A for T, B for H, C for E, assigning different characters, etc.),

               XOR (Exclusive Or)

                               How XOR works, since it is a stream, it goes bit by bit

                               XOR Example

                               A real world example could be WEP/WPA which uses RC4

                                               (fast encryption and decryption, trades security for speed)

                               WPA2 uses AES which is more secure.

Section 2-6 Topic:Pros and Cons of Symmetric Cryptography

Learning Objectives:As we wrap up our lesson on Symmetric Cryptography, we must address the pros and cons of this encryption method.

You’ll discuss its greatest pro—speed. But as we know, sometimes being too fast occurs at a cost. Those costs will be discussed when we cover the cons of Symmetric Cryptography, such as Out of Band Key Distribution, lack of authenticity, issues with integrity, and non-repudiation.

Knowing what the boundaries are with your encryption resources will enable you to make accurate decisions and advise decision-makers with specific points of fact.

Con:

               Out of band key distribution (no straight forward means of key exchange)

               Not Scalable (every unique conversation requires a key)

               No 'A'uthenticity (no court room authenticity, both parties share the key)

               No 'I'ntegrity (corruption happens, it cannot detect corruption or packet drops)

               No 'N'on Repudiation (requires both Authenticity and Integrity)

Pro:

               Does provide 'P'(rivacy/Confidentiality) of 'PAIN'

               FAST, FAST, FAST! (It can be noticably real world faster)

Section 2-7 Topic:Public Key Cryptography

Learning Objectives:This final lesson in Section 2 of the Cryptography module focuses on public key cryptography.

Every user on the network is given a public and private key for encryption. This lesson discusses how the public and private key exchange happens, how Asymmetric Cryptography works, what has to happen so that public and private key exchanges can authenticate, and what the advantage/disadvantages are to employing Asymmetric Cryptography.

And finally, there is a contrast discussion on Symmetric vs. Asymmetric Cryptography discussion introducing how penetration testers can get the best out of both worlds.

Public Key Cryptography

               Two keys issued to each user (simplified example)

                               ~1 public key and 1 private key (a key pair)

                               Done automatically at the application layer with Outlook, Browser, etc.

               Anything encrypted with one key (public) can only be decrypted with the other (private) and vice versa

               Provides key exchange, scalability and non-repediation

               SLOW

Section 3-1 Topic:Asymmetric Cryptography Functions

Learning Objectives:As we wrap up our lesson on Cryptography, we spend our last section exploring Asymmetric Cryptography in detail with several labs to supplement the lecture knowledge.

A penetration tester must be able to understand what happens at each stage of encryption/decryption phase and how those transactions affect or trigger the outcomes of other events.

In this session, we learn how Asymmetric Cryptography delivers all four encryption services:privacy, authenticity, integrity, non-repudiation (PAIN). We’ll then we find out what happens at each state of asymmetric cryptography.

At the conclusion of this lesson, you’ll know what info is exchanged between public and private key, how packet verification occurs, how a hash is created and how hashing algorithms are used.

Asymetric Cryptography P.A.I.N.

               Privacy

               Sender always encrypt with the receiver's PUBLIC key.

               Receiver decrypts with receiver's PRIVATE key which only that receiver has.

                Authenticity

               Sender will encrypt \"something\" with the sender's PRIVATE key.

               When the receiver is able to successfully decrypt that 'something' with that sender's PUBLIC key,                                                   that proves it was encrypted with the sender's PRIVATE key, which only the sender has.

                Example:I am sending message; my content is encrypted with receiver’s public key.

                               The time stamp is encrypted with senders (my) private key - this can prove that *I* sent it.

                Integrity

               Create a message digest (hash) with a hashing algorithm

                               (MD5:128 bit; used less, SHA-1:160 bit; used less, SHA 256:256 bit)

               Asymmetric and Symmetric both use keys

               Hashes do not use keys, they only use math functions

                Example:Very rudimentary, but concept is important - hashes are one way functions

                               H E L   L    O           (every characters numeric value, or the message, then added together)

                               8 5 12 12 15 = 52                (this value is the message digest/hash)

               If you ONLY know the 52 (hash), trying to figure out the corresponding additive values would be arduous.    Coupled with higher bit level encryption, the effort could take longer than we have time in the universe.

                However, if they both match, the message has not been modified.  Very easy one way math.

                Non-Repudiation

               Sender encrypts hash with the sender’s private key. Receiver decrypts hash with sender’s public key.           Receiver hashes document - if hashes match, receiver gets a guaranteed message has not been modified.                  This is called a digital signature.

                Example:              

*It is generally accepted that you cannot reverse a hash*

                SSL - secure sockets layer – Asymmetrical (for all above), Symmetrical for data exchange (speed)

Section 3-2 Topic:Public Key Exchange

Learning Objectives:we continue our focus on Asymmetric Cryptography with a detailed diagram and discussion on secure public key exchange and how it works. You’ll learn the entire process from start to finish and why it’s a faster, more reliable encryption process.

You’ll also learn about Certificates of Authority (CA), what comprises the certificate revocation list (CRL), what a online certification status protocol (OCSP) is, and why they are each essential to the encryption process. We’ll discuss the entire PKI (public key infrastructure) and what role each of its components has.

And finally, we’ll observe then explore what CA hashing looks like during encryption and decryption at the browser level, and learn why thorough PKI knowledge and training in Symmetric & Asymmetric Encryption for netadmins and cyber security professionals is essential.

SSL/TLS

Example:

[Desktop clients] -> uses https:// to reach -> [BofA servers]

               1. Client requests via https to server

               2. BofA sends public key* to client key

               3. Client generates a symmetric session key (i.e. 8439...)

                               how does it get this session key to server?*

               4. Client encrypts the session key with servers public key

                               (since server will be able to decode with private key)

               Result:Enables Asymmetric Key Exchange/Symmetric Data Exchange - Secure Channel Setup

*This model still allows Man In The Middle (MITM) attacks, we must go further for assurance and authenticity.

               To cure this, we must have the following public key being verified by a CA via public key infrastructure

               BofA server proves to Verisign CA (certificate of authority) that they are truly BofA

                               (other CA’s would be Thawte, Baltimore, Entrust, etc.)

               The CA then gives a certificate back to bind public/private key to server

*How do we know it came from Verisign, a trusted authority?

               Verisign sends certificate with private key (can be decrypted with Verisigns public key).

*Your web browser may be pre-loaded with CA's - trust gained by having CA's public key.

Could set up our own CA, where we 'vouch' for ourselves.  While that may not work for web based transactions, but for internal network transactions, it may be an acceptable trade-off.

CRL (pronounced krill) - Certificate Revocation List

OCSP - Online Certificate Status Protocol

Section 3-3 Topic:Advanced Encryption Package

Learning Objectives:The Advanced Encryption Package lab demonstrates use of an advanced encryption analysis components and how to use them to dig deeper in your file, password and data analysis.

You’ll observe creation of a password protected text file, see how the Advanced Encryption Package tool rates the strength of assigned password, and uses powerful algorithms to generate secure encryption for network files and directories.

In the Advanced Encryption Package lab, you’ll learn the advantages and disadvantages in the integrity principle vs. confidentiality principle and how those differences impact Symmetric vs. Asymmetric encryption. You’ll also see how to perform secure deletions, and learn how to use this tool as a key generator and PKI manager.

Advanced Encryption Package Lab

Can be found (Free Trial) at: http://www.aeppro.com/

Section 3-4 Topic:Hashing and Hash Collisions

Learning Objectives:This session of our Cryptography module introduces an extensive overview on Hashing. It begins with mapping out exactly what a Hash does and where it falls short and why it’s a disadvantage being one-directional.

You’ll also explore Hash collisions—what they are, how they happen, what it takes to recreate them, what happens when there is no integrity and how that is determined by the penetration tester.

Hashing

               Hash functions

                               Scenario 1:A MITM could just intercept and change the hash!

                                               Without authenticity, hash is only useful for detecting accidental changes/corruption.

                               Scenario 2:Digital Signature adds authentication to the integrity, we get true non repudiation:                                                       protects against accidental or intentional changes...this all requires a PKI ($ overhead)

                               Scenario 3:Introduce a solution; use a

                                               M(essage)A(uthentication)C(ode) = Message + Symmetric Key + Hashing Algorithm

                                               (this would be a step below having/paying for the PKI overhead)

               Passwords, credit cards, PII

               Digital Forensics

                               Hash the drive before doing forensic analysis (write protect the drive).

                               Even opening a file could change the hash.

               Malware analysis and virus detection

               Hash Collisions

                               Two different pieces of text product the same hash.  Collision resistant hashes are the goal.

                               If a collision is achieved, you now can produce the hash even though it was not the original.

               MAC

               HMAC

Section 3-5 Topic:Hash Calc

Learning Objectives:The Hash Calc lab teaches you what  information is gained from using Hash Calculators to analyze files within a directory, or specifically targeted files. For example, the output of the Hash Calc tool includes the message digest about the target as well as all its relevant integrity algorithms.

The lab also discloses several integrity algorithms not output from the Hash Calc tool that provide a wider range of output datasets. Used in conjunction with the Hash My Files, and Advanced Encryption Package tool Hash Calc can be a very revealing “go-to” suite of resources for cryptographic analysis.

HashCalc  is a fast and easy-to-use calculator that allows to compute message digests, checksums and HMACs for files, as well as for text and hex strings. It offers a choice of 13 of the most popular hash and checksum algorithms for calculations.

Can be found at:http://www.slavasoft.com/hashcalc/

Section 3-6 Topic:Digital Signature, SHA2(256) and RSA

Learning Objectives:In this sub-part of our previous Hash lesson, we take a closer, more intimate look at digital signatures, SHA2(256). We’ll look at Asymmetric Algorithm, RSA, what is met by satisfying the digital signature standard, and what the entire packet transfer looks like when hashing takes place.

We will also introduce concepts behind protecting data at rest.

Several key definitions are discussed included full disk encryption, file encryption, code signing with a discussion on how they differ, what occurs in each of those data transactions, and why that sequence of events is important for Penetration Testers to know as part of the Asymmetric encryption process.

Digital Signatures

               Hash (remember, hashes are just one way math)

                               MD4, SHA-1, SHA-2

               Asymmetric Algorithm

                               RSA - satisfied the Digital Signature Standard

Protecting Data At Rest

               File Encryption - some files or properties may be readable if hard drive is moved to another computer OS                                   (i.e. Windows Hard Drive using Encrypting File System is physically moved to a Linux Box)

               Full Disk Encryption - modern motherboard have TPM (trusted platform module)

                               BitLocker, PGP, etc. achieve this.

               Code Signing - distributed code accessed on Internet (Java, Javascript, ActiveX) should be digitally signed.

Section 3-7 Topic:

Hash My Files

Learning Objectives:

This lab demonstrates the Hash My Files tool which analyzes files by specific file or by directory and validates their integrity. The Hash My Files process report is basic and the output can be exported to a series of different formats including HTML.

You’ll also observe a demonstration of how the Hash My Files tool works and what you can learn from the information it provides.

Because of their strong algorithms, security professionals rely heavily on hashing as a reliable Cryptographic analysis tool for integrity validation.

HashMyFiles tool analyzes files by specific file or by directory and validates their integrity.

HashMyFiles is small utility that allows you to calculate the MD5 and SHA1 hashes of one or more files in your system. You can easily copy the MD5/SHA1 hashes list into the clipboard, or save them into text/html/xml file.

HashMyFiles can also be launched from the context menu of Windows Explorer, and display the MD5/SHA1 hashes of the selected file or folder.

Can be found at:

http://www.nirsoft.net/

Tools Used

CrypTool - http://www.cryptool.org

Advanced Encryption Package - http://www.aeppro.com

HashCalc - http://www.slavasoft.com/hashcalc/

HashMyFiles - http://www.nirsoft.net/

(links cannot be guaranteed, as they are maintained independently)

END OF OUTLINE

The Open Worldwide Application Security Project (OWASP) is a community-led organization and has been around for over 20 years and is largely known for its Top 10 web application security risks (check out our course on it). As the use of generative AI and large language models (LLMs) has exploded recently, so too has the risk to privacy and security by these technologies. OWASP, leading the charge for security, has come out with its Top 10 for LLMs and Generative AI Apps this year. In this blog post we’ll explore the Top 10 risks and explore examples of each as well as how to prevent these risks.

LLM01: Prompt Injection

Those familiar with the OWASP Top 10 for web applications have seen the injection category before at the top of the list for many years. This is no exception with LLMs and ranks as number one. Prompt Injection can be a critical vulnerability in LLMs where an attacker manipulates the model through crafted inputs, leading it to execute unintended actions. This can result in unauthorized access, data exfiltration, or social engineering. There are two types: Direct Prompt Injection, which involves "jailbreaking" the system by altering or revealing underlying system prompts, giving an attacker access to backend systems or sensitive data, and Indirect Prompt Injection, where external inputs (like files or web content) are used to manipulate the LLM's behavior.

As an example, an attacker might upload a resume containing an indirect prompt injection, instructing an LLM-based hiring tool to favorably evaluate the resume. When an internal user runs the document through the LLM for summarization, the embedded prompt makes the LLM respond positively about the candidate’s suitability, regardless of the actual content.

How to prevent prompt injection:

  1. Limit LLM Access: Apply the principle of least privilege by restricting the LLM's access to sensitive backend systems and enforcing API token controls for extended functionalities like plugins.
  2. Human Approval for Critical Actions: For high-risk operations, require human validation before executing, ensuring that the LLM's suggestions are not followed blindly.
  3. Separate External and User Content: Use frameworks like ChatML for OpenAI API calls to clearly differentiate between user prompts and untrusted external content, reducing the chance of unintentional action from mixed inputs.
  4. Monitor and Flag Untrusted Outputs: Regularly review LLM outputs and mark suspicious content, helping users to recognize potentially unreliable information.

LLM02: Insecure Output Handling

Insecure Output Handling occurs when the outputs generated by a LLM are not properly validated or sanitized before being used by other components in a system. Since LLMs can generate various types of content based on input prompts, failing to handle these outputs securely can introduce risks like cross-site scripting (XSS), server-side request forgery (SSRF), or even remote code execution (RCE). Unlike Overreliance (LLM09), which focuses on the accuracy of LLM outputs, Insecure Output Handling specifically addresses vulnerabilities in how these outputs are processed downstream.

As an example, there could be a web application that uses an LLM to summarize user-provided content and renders it back in a webpage. An attacker submits a prompt containing malicious JavaScript code. If the LLM’s output is displayed on the webpage without proper sanitization, the JavaScript will execute in the user’s browser, leading to XSS. Alternatively, if the LLM’s output is sent to a backend database or shell command, it could allow SQL injection or remote code execution if not properly validated.

How to prevent Insecure Output Handling:

  1. Zero-Trust Approach: Treat the LLM as an untrusted source, applying strict allow list validation and sanitization to all outputs it generates, especially before passing them to downstream systems or functions.
  2. Output Encoding: Encode LLM outputs before displaying them to end users, particularly when dealing with web content where XSS risks are prevalent.
  3. Adhere to Security Standards: Follow the OWASP Application Security Verification Standard (ASVS) guidelines, which provide strategies for input validation and sanitization to protect against code injection risks.

LLM03: Training Data Poisoning

Training Data Poisoning refers to the manipulation of the data used to train LLMs, introducing biases, backdoors, or vulnerabilities. This tampered data can degrade the model's effectiveness, introduce harmful biases, or create security flaws that malicious actors can exploit. Poisoned data could lead to inaccurate or inappropriate outputs, compromising user trust, harming brand reputation, and increasing security risks like downstream exploitation.

As an example, there could be a scenario where an LLM is trained on a dataset that has been tampered with by a malicious actor. The poisoned dataset includes subtly manipulated content, such as biased news articles or fabricated facts. When the model is deployed, it may output biased information or incorrect details based on the poisoned data. This not only degrades the model’s performance but can also mislead users, potentially harming the model’s credibility and the organization’s reputation.

How to prevent Training Data Poisoning:

  1. Data Validation and Vetting: Verify the sources of training data, especially when sourcing from third-party datasets. Conduct thorough checks on data integrity, and where possible, use trusted data sources.
  2. Machine Learning Bill of Materials (ML-BOM): Maintain an ML-BOM to track the provenance of training data and ensure that each source is legitimate and suitable for the model’s purpose.
  3. Sandboxing and Network Controls: Restrict access to external data sources and use network controls to prevent unintended data scraping during training. This helps ensure that only vetted data is used for training.
  4. Adversarial Robustness Techniques: Implement strategies like federated learning and statistical outlier detection to reduce the impact of poisoned data. Periodic testing and monitoring can identify unusual model behaviors that may indicate a poisoning attempt.
  5. Human Review and Auditing: Regularly audit model outputs and use a human-in-the-loop approach to validate outputs, especially for sensitive applications. This added layer of scrutiny can catch potential issues early.

LLM04: Model Denial of Service

Model Denial of Service (DoS) is a vulnerability in which an attacker deliberately consumes an excessive amount of computational resources by interacting with a LLM. This can result in degraded service quality, increased costs, or even system crashes. One emerging concern is manipulating the context window of the LLM, which refers to the maximum amount of text the model can process at once. This makes it possible to overwhelm the LLM by exceeding or exploiting this limit, leading to resource exhaustion.

As an example, an attacker may continuously flood the LLM with sequential inputs that each reach the upper limit of the model’s context window. This high-volume, resource-intensive traffic overloads the system, resulting in slower response times and even denial of service. As another example, if an LLM-based chatbot is inundated with a flood of recursive or exceptionally long prompts, it can strain computational resources, causing system crashes or significant delays for other users.

How to prevent Model Denial of Service:

  1. Rate Limiting: Implement rate limits to restrict the number of requests from a single user or IP address within a specific timeframe. This reduces the chance of overwhelming the system with excessive traffic.
  2. Resource Allocation Caps: Set caps on resource usage per request to ensure that complex or high-resource requests do not consume excessive CPU or memory. This helps prevent resource exhaustion.
  3. Input Size Restrictions: Limit input size according to the LLM's context window capacity to prevent excessive context expansion. For example, inputs exceeding a predefined character limit can be truncated or rejected.
  4. Monitoring and Alerts: Continuously monitor resource utilization and establish alerts for unusual spikes, which may indicate a DoS attempt. This allows for proactive threat detection and response.
  5. Developer Awareness and Training: Educate developers about DoS vulnerabilities in LLMs and establish guidelines for secure model deployment. Understanding these risks enables teams to implement preventative measures more effectively.

LLM05: Supply Chain Vulnerabilities

Supply Chain attacks are incredibly common and this is no different with LLMs, which, in this case refers to risks associated with the third-party components, training data, pre-trained models, and deployment platforms used within LLMs. These vulnerabilities can arise from outdated libraries, tampered models, and even compromised data sources, impacting the security and reliability of the entire application. Unlike traditional software supply chain risks, LLM supply chain vulnerabilities extend to the models and datasets themselves, which may be manipulated to include biases, backdoors, or malware that compromises system integrity.

As an example, an organization uses a third-party pre-trained model to conduct economic analysis. If this model is poisoned with incorrect or biased data, it could generate inaccurate results that mislead decision-making. Additionally, if the organization uses an outdated plugin or compromised library, an attacker could exploit this vulnerability to gain unauthorized access or tamper with sensitive information. Such vulnerabilities can result in significant security breaches, financial loss, or reputational damage.

How to prevent Supply Chain Vulnerabilities:

  1. Vet Third-Party Components: Carefully review the terms, privacy policies, and security measures of all third-party model providers, data sources, and plugins. Use only trusted suppliers and ensure they have robust security protocols in place.
  2. Maintain a Software Bill of Materials (SBOM): An SBOM provides a complete inventory of all components, allowing for quick detection of vulnerabilities and unauthorized changes. Ensure that all components are up-to-date and apply patches as needed.
  3. Use Model and Code Signing: For models and external code, employ digital signatures to verify their integrity and authenticity before use. This helps ensure that no tampering has occurred.
  4. Anomaly Detection and Robustness Testing: Conduct adversarial robustness tests and anomaly detection on models and data to catch signs of tampering or data poisoning. Integrating these checks into your MLOps pipeline can enhance overall security.
  5. Implement Monitoring and Patching Policies: Regularly monitor component usage, scan for vulnerabilities, and patch outdated components. For sensitive applications, continuously audit your suppliers’ security posture and update components as new threats emerge.

LLM06: Sensitive Information Disclosure

Sensitive Information Disclosure in LLMs occurs when the model inadvertently reveals private, proprietary, or confidential information through its output. This can happen due to the model being trained on sensitive data or because it memorizes and later reproduces private information. Such disclosures can result in significant security breaches, including unauthorized access to personal data, intellectual property leaks, and violations of privacy laws.

As an example, there could be an LLM-based chatbot trained on a dataset containing personal information such as users’ full names, addresses, or proprietary business data. If the model memorizes this data, it could accidentally reveal this sensitive information to other users. For instance, a user might ask the chatbot for a recommendation, and the model could inadvertently respond with personal information it learned during training, violating privacy rules.

How to prevent Sensitive Information Disclosure:

  1. Data Sanitization: Before training, scrub datasets of personal or sensitive information. Use techniques like anonymization and redaction to ensure no sensitive data remains in the training data.
  2. Input and Output Filtering: Implement robust input validation and sanitization to prevent sensitive data from entering the model’s training data or being echoed back in outputs.
  3. Limit Training Data Exposure: Apply the principle of least privilege by restricting sensitive data from being part of the training dataset. Fine-tune the model with only the data necessary for its task, and ensure high-privilege data is not accessible to lower-privilege users.
  4. User Awareness: Make users aware of how their data is processed by providing clear Terms of Use and offering opt-out options for having their data used in model training.
  5. Access Controls: Apply strict access control to external data sources used by the LLM, ensuring that sensitive information is handled securely throughout the system

LLM07: Insecure Plugin Design

Insecure Plugin Design vulnerabilities arise when LLM plugins, which extend the model’s capabilities, are not adequately secured. These plugins often allow free-text inputs and may lack proper input validation and access controls. When enabled, plugins can execute various tasks based on the LLM’s outputs without further checks, which can expose the system to risks like data exfiltration, remote code execution, and privilege escalation. This vulnerability is particularly dangerous because plugins can operate with elevated permissions while assuming that user inputs are trustworthy.

As an example, there could be a weather plugin that allows users to input a base URL and query. An attacker could craft a malicious input that directs the LLM to a domain they control, allowing them to inject harmful content into the system. Similarly, a plugin that accepts SQL “WHERE” clauses without validation could enable an attacker to execute SQL injection attacks, gaining unauthorized access to data in a database.

How to prevent Insecure Plugin Design:

  1. Enforce Parameterized Input: Plugins should restrict inputs to specific parameters and avoid free-form text wherever possible. This can prevent injection attacks and other exploits.
  2. Input Validation and Sanitization: Plugins should include robust validation on all inputs. Using Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) can help identify vulnerabilities during development.
  3. Access Control: Follow the principle of least privilege, limiting each plugin's permissions to only what is necessary. Implement OAuth2 or API keys to control access and ensure only authorized users or components can trigger sensitive actions.
  4. Manual Authorization for Sensitive Actions: For actions that could impact user security, such as transferring files or accessing private repositories, require explicit user confirmation.
  5. Adhere to OWASP API Security Guidelines: Since plugins often function as REST APIs, apply best practices from the OWASP API Security Top 10. This includes securing endpoints and applying rate limiting to mitigate potential abuse.

LLM08: Excessive Agency

Excessive Agency in LLM-based applications arises when models are granted too much autonomy or functionality, allowing them to perform actions beyond their intended scope. This vulnerability occurs when an LLM agent has access to functions that are unnecessary for its purpose or operates with excessive permissions, such as being able to modify or delete records instead of only reading them. Unlike Insecure Output Handling, which deals with the lack of validation on the model’s outputs, Excessive Agency pertains to the risks involved when an LLM takes actions without proper authorization, potentially leading to confidentiality, integrity, and availability issues.

As an example, there could be an LLM-based assistant that is given access to a user's email account to summarize incoming messages. If the plugin that is used to read emails also has permissions to send messages, a malicious prompt injection could trick the LLM into sending unauthorized emails (or spam) from the user's account.

How to prevent Excessive Agency:

  1. Restrict Plugin Functionality: Ensure plugins and tools only provide necessary functions. For example, if a plugin is used to read emails, it should not include capabilities to delete or send emails.
  2. Limit Permissions: Follow the principle of least privilege by restricting plugins’ access to external systems. For instance, a plugin for database access should be read-only if writing or modifying data is not required.
  3. Avoid Open-Ended Functions: Avoid functions like “run shell command” or “fetch URL” that provide broad system access. Instead, use plugins that perform specific, controlled tasks.
  4. User Authorization and Scope Tracking: Require plugins to execute actions within the context of a specific user's permissions. For example, using OAuth with limited scopes helps ensure actions align with the user’s access level.
  5. Human-in-the-Loop Control: Require user confirmation for high-impact actions. For instance, a plugin that posts to social media should require the user to review and approve the content before it is published.
  6. Authorization in Downstream Systems: Implement authorization checks in downstream systems that validate each request against security policies. This prevents the LLM from making unauthorized changes directly.

LLM09: Overreliance

Overreliance occurs when users or systems trust the outputs of a LLM without proper oversight or verification. While LLMs can generate creative and informative content, they are prone to “hallucinations” (producing false or misleading information) or providing authoritative-sounding but incorrect outputs. Overreliance on these models can result in security risks, misinformation, miscommunication, and even legal issues, especially if LLM-generated content is used without validation. This vulnerability becomes especially dangerous in cases where LLMs suggest insecure coding practices or flawed recommendations.

As an example, there could be a development team using an LLM to expedite the coding process. The LLM suggests an insecure code library, and the team, trusting the LLM, incorporates it into their software without review. This introduces a serious vulnerability. As another example, a news organization might use an LLM to generate articles, but if they don’t validate the information, it could lead to the spread of disinformation.

How to prevent Overreliance:

  1. Regular Monitoring and Review: Implement processes to review LLM outputs regularly. Use techniques like self-consistency checks or voting mechanisms to compare multiple model responses and filter out inconsistencies.
  2. Cross-Verification: Compare the LLM’s output with reliable, trusted sources to ensure the information’s accuracy. This step is crucial, especially in fields where factual accuracy is imperative.
  3. Fine-Tuning and Prompt Engineering: Fine-tune models for specific tasks or domains to reduce hallucinations. Techniques like parameter-efficient tuning (PET) and chain-of-thought prompting can help improve the quality of LLM outputs.
  4. Automated Validation: Use automated validation tools to cross-check generated outputs against known facts or data, adding an extra layer of security.
  5. Risk Communication: Clearly communicate the limitations of LLMs to users, highlighting the potential for errors. Transparent disclaimers can help manage user expectations and encourage cautious use of LLM outputs.
  6. Secure Coding Practices: For development environments, establish guidelines to prevent the integration of potentially insecure code. Avoid relying solely on LLM-generated code without thorough review.

LLM10: Model Theft

Model Theft refers to the unauthorized access, extraction, or replication of proprietary LLMs by malicious actors. These models, containing valuable intellectual property, are at risk of exfiltration, which can lead to significant economic and reputational loss, erosion of competitive advantage, and unauthorized access to sensitive information encoded within the model. Attackers may steal models directly from company infrastructure or replicate them by querying APIs to build shadow models that mimic the original. As LLMs become more prevalent, safeguarding their confidentiality and integrity is crucial.

As an example, an attacker could exploit a misconfiguration in a company’s network security settings, gaining access to their LLM model repository. Once inside, the attacker could exfiltrate the proprietary model and use it to build a competing service. Alternatively, an insider may leak model artifacts, allowing adversaries to launch gray box adversarial attacks or fine-tune their own models with stolen data.

How to prevent Model Theft:

  1. Access Controls and Authentication: Use Role-Based Access Control (RBAC) and enforce strong authentication mechanisms to limit unauthorized access to LLM repositories and training environments. Adhere to the principle of least privilege for all user accounts.
  2. Supplier and Dependency Management: Monitor and verify the security of suppliers and dependencies to reduce the risk of supply chain attacks, ensuring that third-party components are secure.
  3. Centralized Model Inventory: Maintain a central ML Model Registry with access controls, logging, and authentication for all production models. This can aid in governance, compliance, and prompt detection of unauthorized activities.
  4. Network Restrictions: Limit LLM access to internal services, APIs, and network resources. This reduces the attack surface for side-channel attacks or unauthorized model access.
  5. Continuous Monitoring and Logging: Regularly monitor access logs for unusual activity and promptly address any unauthorized access. Automated governance workflows can also help streamline access and deployment controls.
  6. Adversarial Robustness: Implement adversarial robustness training to help detect extraction queries and defend against side-channel attacks. Rate-limit API calls to further protect against data exfiltration.
  7. Watermarking Techniques: Embed unique watermarks within the model to track unauthorized copies or detect theft during the model’s lifecycle.

Wrapping it all up

As LLMs continue to grow in capability and integration across industries, their security risks must be managed with the same vigilance as any other critical system. From Prompt Injection to Model Theft, the vulnerabilities outlined in the OWASP Top 10 for LLMs highlight the unique challenges posed by these models, particularly when they are granted excessive agency or have access to sensitive data. Addressing these risks requires a multifaceted approach involving strict access controls, robust validation processes, continuous monitoring, and proactive governance.

For technical leadership, this means ensuring that development and operational teams implement best practices across the LLM lifecycle starting from securing training data to ensuring safe interaction between LLMs and external systems through plugins and APIs. Prioritizing security frameworks such as the OWASP ASVS, adopting MLOps best practices, and maintaining vigilance over supply chains and insider threats are key steps to safeguarding LLM deployments. Ultimately, strong leadership that emphasizes security-first practices will protect both intellectual property and organizational integrity, while fostering trust in the use of AI technologies.

Start learning with Cybrary

Create a free account

Related Posts

All Blogs