Recently, we researched a project on Portainer, the go-to open-source tool for managing Kubernetes and Docker environments. With more than 30K stars on GitHub, Portainer gives you a user-friendly web interface to deploy and monitor containerized applications easily.

Since Portainer is an open-source, we thought CodeQL, an advanced code analysis tool, be a good fit to check its codebase for any security issues. CodeQL analyzes the code deeply by turning it into a database that you can query for patterns that might indicate security flaws.

In this blog, we will show how we used CodeQL to find these vulnerabilities and even wrote custom queries to find a specific vulnerability . Let’s dive in.

tl;dr

During the research, we found two vulnerabilities in Portainer:

1. CVE-2024-33661: Two Blind Server-Side Request Forgery (SSRF) vulnerabilities.
2. CVE-2024-33662: Cryptographic flaw in the implementation of AES-OFB.

Portainer Overview

As stated above, Portainer is an open-source management tool that provides a simple and intuitive interface for deploying and managing Docker containers (Figure 1). For example, instead of using complex Docker commands, you can easily start or stop containers with just a few clicks.

Figure 1 – Portainer web app

Since Portainer is an open-source written in Golang (supported by CodeQL), we decided to explore whether CodeQL could assist us find security issues as we did in the past.

About CodeQL

CodeQL is a powerful code analysis framework originally developed by Semmle and now maintained by GitHub. It is freely available for use on open-source projects. It allows developers and researchers to perform variant analysis to find security vulnerabilities by querying code databases generated using CodeQL.

CodeQL supports many languages such as C/C++, C#, Java, JavaScript, Python and Golang.

Once we generate a code database, a structured representation of your codebase that includes all relevant code files and their relationships, we can use premade queries developed by GitHub and the community or write and use custom queries.

Setting Up a CodeQL Environment

We started by installing the CodeQL extension in Visual Studio Code. After that, we downloaded the starter workspace. The next step was to generate a database for our working project, in our case Portainer.

It is possible to use the CodeQL CLI to generate a database; however, in some cases (like ours) there is an easier option in the CodeQL extension to download the database by pasting the GitHub URL of the repository (Figure 2)

Figure 2 – Download CodeQL database from GitHub

Within the workspace, we have access to many premade CodeQL queries (Figure 3). For example, a typical query can check for SQL injection vulnerabilities.

The premade queries are divided into categories; one of them is security, where we can run up to 20 queries by default (adjustable in the extension settings) simultaneously. This enables the detection of potential security vulnerabilities within the codebase.

Figure 3 – Premade CodeQL queries for Go

Running CodeQL Queries

We ran all the premade security queries (CWE-020, CWE-022, etc.) as well as experimental one, investigated each finding and assessed that most of them were not exploitable — and some were false positives (Figure 4).

Figure 4 – Query history after running the queries

After some investigation, we discovered two notable findings that led to blind Server-Side Request Forgery (SSRF) vulnerabilities.

We found these vulnerabilities through the CWE-918 query, which uses the Request Forgery query library to search for instances in which untrusted user input is applied in network requests. For example, the query might detect a case in which a user-supplied URL is passed directly to a function that makes HTTP requests, potentially allowing an attacker to perform SSRF.

Double Blind SSRFs (CVE-2024-33661)

SSRF is a security vulnerability that allows an attacker to induce the server-side application to make HTTP requests to an unintended destination, potentially leading to unauthorized access to internal resources. Blind SSRF is a type of SSRF attack in which the attacker cannot see the response from the server directly.

Our investigation into the first vulnerability began with CodeQL query results that highlighted potential starting points (source) and end points (sink) within the codebase (Figure 5)..

Figure 5 – CodeQL SSRF results

These results pointed us toward the /api/templates/helm endpoint. We observed that the function helmRepoSearch, which handles GET requests to /api/templates/helm, contains the following code:

 

func (handler *Handler) helmRepoSearch(w http.ResponseWriter, r *http.Request) *httperror.HandlerError {
    repo := r.URL.Query().Get("repo")
    ...
    result, err := handler.helmPackageManager.SearchRepo(searchOpts)
    ...
}

Code Snippet 1 – helmRepoSearch implementation

Here, the repo variable is extracted from the query parameter, validated and forwarded to the SearchRepo function, which then makes an HTTP GET request:

func (hbpm *helmBinaryPackageManager) SearchRepo(searchRepoOpts options.SearchRepoOptions) ([]byte, error) {
   ...
  url.Path = path.Join(url.Path, "index.yaml")
  resp, err := client.Get(url.String())
  ...
}

Code Snippet 2 – SearchRepo implementation

This means that when a URL like /api/templates/helm?repo=http://<IP>:<PORT> is provided, it will request http://<IP>:<PORT>/index.yaml (Figure 6). This setup allows us to control the URL the server requests.

Figure 6 – Regular behavior of Portainer to get the Helm’s index YAML file

Although SearchRepo appends index.yaml to the URL, we can bypass this intended behavior by configuring a server to redirect index.yaml to an arbitrary internal service we control.

An attacker can set up a server that listens on port 8081 and has an endpoint named /index.yaml that redirects to any location the attacker wants, such as an internal services within the same network (e.g., internal APIs or databases).

The attacker can send a request to this local endpoint by crafting the URL /api/templates/helm?repo=https://attacker.com:8081.

In turn, Portainer will receive the request and send a GET request to https://attacker.com:8081/index.yaml.

To demonstrate this, we created an HTTP web server on Portainer’s local server that listens on port 1337 and includes an endpoint /web.

The attacker server will then receive this request and because the server has an endpoint named index.yaml, it will redirect the request to http://127.0.0.1:1337/web (Figure 7).

Figure 7 – Blind SSRF redirects Portainer to an internal service

By setting up this redirection, we were able to demonstrate how an attacker can exploit this SSRF vulnerability to make the server access internal services that would otherwise be protected.

Second Blind SSRF

The second blind SSRF vulnerability is similar to the first one. This vulnerability resides in the helmShow function, which handles GET requests to /api/templates/helm/{command:chart|values|readme} in the following code:

 
func (handler *Handler) helmShow(w http.ResponseWriter, r *http.Request) *httperror.HandlerError {
	repo := r.URL.Query().Get("repo")
	...
	chart := r.URL.Query().Get("chart")
	if chart == "" {
		return httperror.BadRequest("Bad request", errors.New("missing `chart` query parameter"))
	}

	cmd, err := request.RetrieveRouteVariableValue(r, "command")
	if err != nil {
		cmd = "all"
		log.Debug().Str("default_command", cmd).Msg("command not provided, using default")
	}

	showOptions := options.ShowOptions{
		OutputFormat: options.ShowOutputFormat(cmd),
		Chart:        chart,
		Repo:         repo,
	}
	result, err := handler.helmPackageManager.Show(showOptions)
	...
}

Code Snippet 3 – helmShow implementation

The helmShow function handles URL paths in the following way:

/api/templates/helm/{command:chart|values|readme}?repo=http://<repository_url>&chart={chart_name}

Code Snippet 4 – helmShow handles URL paths

The function extracts the parameters and sends them to the helmPackageManager.Show(..) function:

 

// Show runs `helm show   --repo ` with specified show options.
// The show options translate to CLI arguments which are passed in to the helm binary when executing install.
func (hbpm *helmBinaryPackageManager) Show(showOpts options.ShowOptions) ([]byte, error) {
	if showOpts.Chart == "" || showOpts.Repo == "" || showOpts.OutputFormat == "" {
		return nil, errRequiredShowOptions
	}

	args := []string{
		string(showOpts.OutputFormat),
		showOpts.Chart,
		"--repo", showOpts.Repo,
	}

	result, err := hbpm.run("show", args, showOpts.Env)
	if err != nil {
		return nil, errors.New("the request failed since either the Helm repository was not found or the chart does not exist")
	}

	return result, nil
}

Code Snippet 5 – Show implementation

This function assembles the parameters into Helm CLI command and calls hbpm.run which is a library to run Helm CLI with the following command:

 
helm show <command:chart|values|readme> <chart_name> --repo http://<repository_url>

Code Snippet 6 – helm CLI command

The helm command will search for the repository URL with the index.yaml file by sending a GET request to the repository URL. As we did in the first case, we can set the repository URL to our server and redirect GET requests to “index.yaml” to any place we want.

We reported these issues to Portainer, and they fixed it quickly on version 2.20.0 and assigned it with CVE-2024-33661.

Here is PoC exploitation demo for the two vulnerabilities:

 

Insecure AES-OFB Implementation (CVE-2024-33662)

While examining the APIs in Portainer, we discovered a different vulnerability class related to the backup functionality. Portainer allows users to download a backup file of its configurations with an optional password for encryption (Figure 8).

Figure 8- Portainer download backup functionality

This process involves a POST request to /backup, which invokes the CreateBackupArchive function. This function creates a tar.gz archive with Portainer server configuration and encrypts it using the provided password by calling the encrypt function:

 
func encrypt(path string, passphrase string) (string, error) {
	in, err := os.Open(path)
	if err != nil {
		return "", err
	}
	defer in.Close()

	outFileName := fmt.Sprintf("%s.encrypted", path)
	out, err := os.Create(outFileName)
	if err != nil {
		return "", err
	}

	err = crypto.AesEncrypt(in, out, []byte(passphrase))

	return outFileName, err
}

Code Snippet 7 – encrypt implementation

The encrypt function, in turn, calls to AesEncrypt, which uses the AES-256 encryption with Output Feedback (OFB) mode:

// NOTE: has to go with what is considered to be a simplistic in that it omits any
// authentication of the encrypted data.
// Person with better knowledge is welcomed to improve it.
// sourced from https://golang.org/src/crypto/cipher/example_test.go

var emptySalt []byte = make([]byte, 0)

// AesEncrypt reads from input, encrypts with AES-256 and writes to the output.
// passphrase is used to generate an encryption key.
func AesEncrypt(input io.Reader, output io.Writer, passphrase []byte) error {
	// making a 32 bytes key that would correspond to AES-256
	// don't necessarily need a salt, so just kept in empty
	key, err := scrypt.Key(passphrase, emptySalt, 32768, 8, 1, 32)
	if err != nil {
		return err
	}

	block, err := aes.NewCipher(key)
	if err != nil {
		return err
	}

	// If the key is unique for each ciphertext, then it's ok to use a zero
	// IV.
	var iv [aes.BlockSize]byte
	stream := cipher.NewOFB(block, iv[:])

	writer := &cipher.StreamWriter{S: stream, W: output}
	// Copy the input to the output, encrypting as we go.
	if _, err := io.Copy(writer, input); err != nil {
		return err
	}

	return nil
}

Code Snippet 8 – Show implementation

However, upon closer inspection of the AesEncrypt implementation, we identified several security issues:

1. No salting: The key derivation function (KDF) does not use salting. Salting is essential to prevent pre-computed dictionary attacks and ensure that the same password generates different keys each time.
2. Zero IV: The initialization vector (IV) is set to zero. An IV is crucial in modes like OFB to ensure that the same plaintext encrypts to different ciphertexts each time.
3. No message authentication: The encrypted data lacks an HMAC (Hash-based Message Authentication Code), which is vital for ensuring the integrity and authenticity of the ciphertext. An attacker could modify the ciphertext, and the system would be unable to detect this tampering.

We reported this issue to Portainer, and they fixed it in version 2.20.2 and assigned it with CVE-2024-33662.

A Dangerous Mix: Zero IV and AES-OFB Mode

Let’s focus now on the combination of zero IV and AES-OFB mode. Using a NULL IV is risky in any AES Mode that requires it, but it is especially dangerous when used in OFB Mode. Why?

Let’s have a look:
When using AES encryption, the different encryption modes define how blocks of plaintext are transformed into ciphertext using a block cipher. Different modes offer different properties and levels of security.

Output Feedback (OFB) mode is a type of stream cipher mode that requires a unique and unpredictable IV for each encryption session. If the IV is not unique, as in the case of a zero IV, the same plaintext block will always encrypt to the same ciphertext block (Figure 9), which causes it to be vulnerable to replay attacks and pattern analysis.

Figure 9 – Encrypting the same text leads to the same encrypted stream

The encryption process involves generating a keystream (IV + key/password) independently of the plaintext, which is then XORed with the plaintext to produce the ciphertext (Figure 10).

Figure 10 – OFB implementation from Wikipedia

This stream cipher-like behavior has a unique implication:

If the same key and IV are used, the same keystream is generated for any plaintext. This creates a significant security risk because if different plaintexts are encrypted with the same key and IV, the resulting ciphertexts will reveal patterns or similarities, making it easier for attackers to deduce the original data or launch other cryptographic attacks.

Let’s examine what happens when we XOR two ciphertexts that were encrypted using AES-OFB with the same keystream.

Given:

• c1 – is the ciphertext corresponding to plaintext p1 using keystream k.
• c2 – is the ciphertext corresponding to plaintext p2 using keystream k.

The encryption process can be described as:

• c1 = p1 ⊕ k
• c2 = p2 ⊕ k

Where ⊕ denotes the XOR operation.

If we XOR the two ciphertexts c1 and c2:

• c1 ⊕ c2 = ( p1 ⊕ k ) ⊕ ( p2 ⊕ k )

Using the properties of XOR, which is associative and commutative:

• c1 ⊕ c2 = p1 ⊕ k ⊕ p2 ⊕ k

Since k ⊕ k = 0 (XORing any value with itself yields zero):

• c1 ⊕ c2 = p1 ⊕ p2 ⊕ ( k ⊕ k )
• c1 ⊕ c2 = p1 ⊕ p2 ⊕ 0
• c1 ⊕ c2 = p1 ⊕ p2

Thus, when we XOR the two ciphertexts c1 and c2, we get the XOR of the two plaintexts p1 and p2:

• c1 ⊕ c2 = p1 ⊕ p2

This result highlights a potential vulnerability in the use of OFB mode with a static keystream for multiple plaintexts. If an attacker knows or can guess some part of one plaintext (say p1), they can use c1⊕c2 to potentially deduce parts of p2.

Therefore, it is important to use a unique keystream for each encryption operation. Typically, this is achieved by using a unique IV for each encryption stream, ensuring the keystream is different — even if the key remains the same.

CodeQL Query to Find Insecure AES-OFB

As far as we know, this pattern is not covered by the “stock” CodeQL queries, so we decided to write two queries that can find similar vulnerabilities. While these queries might contain a few false positives that can be improved, we found them to be sufficient at this time.

If you are not familiar with CodeQL, writing queries can be challenging at first. However, several resources helped in this process:

• Asking questions in the CodeQL Slack community (Special thanks to Owen Mansel-Chan from GitHub, who helped us write the queries)
• ChatGPT
• CodeQL “Quick Evaluation” feature in Visual Studio
• CodeQL tutorials and code examples

Query to Find Empty Salt

The first query we are going to build will look for instances using an empty salt (Code Snippet 9):

var emptySalt []byte = make([]byte, 0)

func AesEncrypt(input io.Reader, output io.Writer, passphrase []byte) error {
    key, err := scrypt.Key(passphrase, emptySalt, 32768, 8, 1, 32)

Code Snippet 9 – AesEncrypt calls scrypt.Key

 

The query achieves its purpose by looking for empty byte arrays that passed as the second argument to the scrypt.Key, corresponding to the salt parameter (emptySalt).

In CodeQL, we can create path queries to visualize how information flows from a function to a function through the codebase. These queries are especially useful for analyzing data flow, as they track a variable’s journey from its potential starting points (sources) to its possible endpoints (sinks). To model these paths, our query must specify the sources, the sinks and the intermediate data flow steps that connect them.

We used predicates, which behave like functions by taking inputs and returning results, to define the source and the sink. We started by implementing the isSource predicate to identify all the empty byte arrays:

 
predicate isSource(DataFlow::Node source) {
  exists(DataFlow::CallNode cn, CallExpr ce |
    ce = cn.getExpr() and
    ce.getTarget() = Builtin::make() and
    ce.getArgument(0).getType().(SliceType).getElementType() = Builtin::byte().getType() and
    ce.getArgument(1).getIntValue() = 0 and
    source = cn.getResult()
  )
}

Code Snippet 10 – CodeQL isSource

Let’s cover the crucial points:

• ce.getTarget() = Builtin::make(): Walking over all function calls and checking if they are a built-in Go “make” function — a function that initializes slices in Go.
• ce.getArgument(0).getType().(SliceType).getElementType() = Builtin::byte().getType(): Ensures the first argument is a slice of bytes.
• ce.getArgument(1).getIntValue() = 0: Checks that the second argument is an integer with a value of 0 (an empty array).

We can improve it by including cases in which the emptySalt variable is initialized through a wrapper function.

The second predicate (isSink) identifies where the scrypt.Key function is called with the empty byte array (from the isSource predicate) as the salt parameter:

 
predicate isSink(DataFlow::Node sink) {
    exists(DataFlow::CallNode cn |
      cn.getTarget().hasQualifiedName("golang.org/x/crypto/scrypt", "Key") and
      sink = cn.getArgument(1)
   )}
}

Code Snippet 11 – CodeQL isSink

Predicate isSink details:

• cn.getTarget().hasQualifiedName(“golang.org/x/crypto/scrypt”, “Key”): Finds all instances where the scrypt.Key function is called.
• sink = cn.getArgument(1): Binds sink to the second argument (the salt parameter) of the scrypt.Key function call.

The final query is:

module FlowConfig implements DataFlow::ConfigSig {
  predicate isSource(DataFlow::Node source) {
    exists(DataFlow::CallNode cn, CallExpr ce |
      ce = cn.getExpr() and
      ce.getTarget() = Builtin::make() and
      ce.getArgument(0).getType().(SliceType).getElementType() = Builtin::byte().getType() and
      ce.getArgument(1).getIntValue() = 0 and
      source = cn.getResult()
    )
  }

  predicate isSink(DataFlow::Node sink) {
    exists(DataFlow::CallNode cn |
      cn.getTarget().hasQualifiedName("golang.org/x/crypto/scrypt", "Key") and
      sink = cn.getArgument(1)
   )}
}

module Flow = DataFlow::Global;

import Flow::PathGraph

from Flow::PathNode source, Flow::PathNode sink
where
  Flow::flowPath(source, sink)
select sink.getNode(), source, sink, "Empty salt in scrypt call"

 Code Snippet 12 – CodeQL query to find empty salt

Query to Find Empty IV

The second query we are going to build searches for instances where an empty IV is used inside cipher.NewOFB (Code Snippet 13):

var iv [aes.BlockSize]byte
stream := cipher.NewOFB(block, iv[:])

Code Snippet 13 – Calling OFB mode with empty IV

The query does this by looking for byte arrays that are passed as the second argument to cipher.NewOFB, corresponding to the IV byte array.

As we did in the previous query, we started by implementing the isSource predicate to identify all the declarations of ArrayType variables:

 
predicate isSource(DataFlow::Node source) {
  exists(Type type, ArrayTypeExpr ate, Expr arrayElementTypeExpr  |

    arrayElementTypeExpr = ate.getElement() and
    type instanceof ArrayType
  )
}

Code Snippet 14 – CodeQL isSource

Predicate isSource details:

• arrayElementTypeExpr = ate.getElement(): Gets a list of array types.
• type instanceof ArrayType: Ensures that the type being examined is indeed an array type.

The second predicate (isSink) identifies where the cipher.NewOFB function is called with the empty byte array (from the isSource predicate) as the IV parameter:

 

predicate isSink(DataFlow::Node sink) { 
  exists(DataFlow::CallNode cn | 
    cn.getTarget().hasQualifiedName("crypto/cipher", "NewOFB") and 
    sink = cn.getArgument(1) 
  )} 
}

Code Snippet 15 – CodeQL isSink

Predicate isSink details:
– cn.getTarget().hasQualifiedName(“crypto/cipher “, “NewOFB”): Finds all the calls by cipher.NewOFB function.
– sink = cn.getArgument(1): Binds sink to the second argument (the IV parameter) of the cipher.OFB function call.

Finally, the complete query is as follows:

module FlowConfig implements DataFlow::ConfigSig {
  predicate isSource(DataFlow::Node source) {
    exists(Type type, ArrayTypeExpr ate, Expr arrayElementTypeExpr  |
      arrayElementTypeExpr = ate.getElement() and
      type instanceof ArrayType
    )
  }

  predicate isSink(DataFlow::Node sink) {
    exists(DataFlow::CallNode cn |
      cn.getTarget().hasQualifiedName("crypto/cipher", "NewOFB") and
      sink = cn.getArgument(1)
   )}
}

module Flow = DataFlow::Global<FlowConfig>;

import Flow::PathGraph

from Flow::PathNode source, Flow::PathNode sink
where
  Flow::flowPath(source, sink)
select sink.getNode(), source, sink, "Empty IV in NewOFB call"
 

Code Snippet 16 – CodeQL query to find empty IV

Final Insights

In this blog post, we examined two vulnerabilities found in Portainer. The first issue, found through CodeQL, was a blind SSRF vulnerability in the /api/templates/helm endpoint.

The second vulnerability, found through a manual static analysis, was an insecure AES-OFB implementation in the backup functionality, where the encryption process lacked essential security measures like salting, a proper initialization vector (IV) and HMAC, exposing the backup files to potential attacks.

These findings underscore the importance of rigorous security practices, especially in tools used for managing sensitive infrastructure like Portainer. By leveraging CodeQL and enhancing CodeQL with custom queries, security researchers/developers can systematically uncover and address vulnerabilities, contributing to safer and more secure software ecosystems.

Disclosure Timeline

February, 13, 2024 — Initial report to Portainer about the two blind SSRF vulnerabilities; they acknowledged on the same day and created an internal tracking ID: EE-6722.

February, 20, 2024 — Initial report to Portainer about the vulnerable AES-OFB implementation.

February 21, 2024 —Portainer acknowledged the vulnerable AES-OFB implementation and created an internal tracking ID: EE-6764.

February 22, 2024 —Portainer fixed the blind SSRFs (EE-6764) but didn’t publish it yet.

March 3, 2024 — We asked for an update for the blind SSRF vulnerabilities.

March 4, 2024 —Portainer answered that the fix for the blind SSRF vulnerabilities will be included in the next Portainer release.

March, 19, 2024 — Portainer released version 2.20.0 with the fix for the blind SSRFs (assigned CVE-2024-33661).

March, 26, 2024 — We asked for an update on both vulnerabilities; Portainer answered on the same day that the blind SSRF (EE-6722) has been fixed in the latest Portainer release 2.20 and EE-6764 is planned for 2.20.x Point release.

May 1, 2024 — Portainer released version 2.20.2 with the fix for the vulnerable AES-OFB implementation (assigned CVE-2024-33662).

 

Eviatar Gerzi is a principal cyber researcher at CyberArk Labs.