Oxia - CraftedSignal Threat Feed

Oxia TLS Certificate Chain Validation Failure

hello@craftedsignal.io — Tue, 30 Jan 2024 12:00:00 +0000

The Oxia database platform, specifically versions 0.16.1 and earlier, contains a vulnerability in its TLS configuration parsing. The trustedCertPool() function within the common/security/tls.go file is designed to load trusted Certificate Authority (CA) certificates for mTLS authentication. However, the function only processes the first PEM-encoded certificate block within a specified CA file. This means that if a CA file contains a bundle of certificates, such as an intermediate CA and a root CA, only the first certificate is loaded and the rest are silently discarded. This issue was reported on GitHub as GHSA-7jrq-q4pq-rhm6 and can lead to a significant degradation of security posture.

Attack Chain

An administrator configures Oxia to use mTLS for client authentication, specifying a CA bundle file containing both intermediate and root CA certificates via the trustedCaFile setting.
Oxia's trustedCertPool() function attempts to load the CA certificates from the specified file.
The pem.Decode() function is called, but it only parses the first PEM block (e.g., the intermediate CA certificate).
Subsequent certificates within the CA bundle (e.g., the root CA certificate) are silently ignored because the code doesn't iterate over all PEM blocks.
A legitimate client attempts to connect to Oxia using a certificate chain that is correctly signed by the intermediate CA, which is in turn signed by the root CA.
Oxia only trusts the intermediate CA certificate that was loaded.
Certificate chain validation fails because the root CA is not trusted, resulting in the error "x509: certificate signed by unknown authority".
The legitimate client connection is rejected, effectively breaking mTLS authentication.

Impact

The vulnerability results in the failure of certificate chain validation in mTLS deployments of Oxia. This means that legitimate clients with properly chained certificates are rejected, leading to a denial of service. Operators might be forced to disable client certificate verification, which severely weakens the security posture of Oxia deployments. All Oxia versions up to and including 0.16.1 are vulnerable if TLS with the trustedCaFile configuration is used.

Recommendation

Apply the patch provided by Oxia that iterates over all PEM blocks in the CA file (reference: GHSA-7jrq-q4pq-rhm6).
As a temporary workaround, use CA files containing only a single certificate (the direct issuer of client certificates) for the trustedCaFile configuration (reference: GHSA-7jrq-q4pq-rhm6).
Monitor Oxia server logs for "x509: certificate signed by unknown authority" errors after configuring mTLS, which may indicate this vulnerability is being triggered.

Oxia Server Crash via Session Heartbeat Race Condition

hello@craftedsignal.io — Tue, 09 Jan 2024 12:00:00 +0000

A denial-of-service vulnerability exists in Oxia, an open-source distributed database, due to a race condition in the session heartbeat handling mechanism. This vulnerability can be exploited by a remote client to crash the Oxia data server process. The root cause lies in the interaction between the heartbeat() method in session.go and the KeepAlive() function in session_manager.go. Specifically, a time-of-check-to-time-of-use (TOCTOU) gap occurs when KeepAlive() releases the session manager's read lock before calling heartbeat(), allowing the session to be closed concurrently. All Oxia versions up to and including 0.16.1 are affected. Defenders should prioritize implementing detection mechanisms to identify potentially malicious KeepAlive patterns and plan for upgrading to patched versions.

Attack Chain

A client establishes a session with the Oxia data server.
The client sends regular KeepAlive requests to the server to maintain the session.
The server's KeepAlive() function receives the request and releases the session manager's read lock.
Before heartbeat() is called, the session expires or is closed due to other factors.
The heartbeat() method attempts to send a signal on the closed heartbeatCh channel.
Due to the race condition between session closure and the heartbeat() call, the server panics.
The Oxia data server process crashes, leading to a denial-of-service.

Impact

Successful exploitation of this vulnerability results in a complete denial-of-service. The Oxia data server process crashes, interrupting all data services relying on that server. The number of affected systems depends on the scale of Oxia deployment. Organizations utilizing Oxia versions up to and including 0.16.1 are vulnerable. The impact can range from service disruption to data unavailability.

Recommendation

Upgrade to a patched version of Oxia that includes the fixes for the heartbeat handling race condition.
Implement rate limiting on incoming KeepAlive requests to mitigate the impact of rapid requests (refer to network_connection log source).
Deploy the provided Sigma rule to detect anomalous patterns of KeepAlive requests (see the Sigma rule targeting webserver logs).
Monitor webserver logs for unusual spikes in KeepAlive requests (refer to webserver log source).

Oxia Bearer Token Exposure in Debug Logs

hello@craftedsignal.io — Tue, 09 Jan 2024 12:00:00 +0000

Oxia, a distributed database, is vulnerable to exposing sensitive bearer tokens in its debug logs. Specifically, when OpenID Connect (OIDC) authentication fails, the complete JWT, including header, payload, and signature, is logged in plaintext at the DEBUG level. This occurs in versions 0.16.1 and earlier. If debug logging is enabled in production environments, these tokens are written to application logs and potentially any connected log aggregation systems. An attacker gaining access to these logs could then extract valid JWT tokens and use them to impersonate legitimate users. The vulnerability stems from the validateTokenWithContext() function within oxiad/common/rpc/auth/interceptor.go. Defenders should ensure that debug logging is disabled in production or implement detections to identify potential token leakage.

Attack Chain

A user attempts to authenticate to an Oxia service using OIDC.
The validateTokenWithContext() function in oxiad/common/rpc/auth/interceptor.go is invoked to validate the provided JWT token.
If the token validation fails, the complete token is logged at DEBUG level using slog.String("token", token).
The debug logs containing the full bearer token are written to the application log files.
An attacker gains unauthorized access to the application log files or the connected log aggregation system.
The attacker extracts valid JWT tokens from the compromised logs.
The attacker replays the extracted JWT token to authenticate as a legitimate user.
The attacker gains unauthorized access to Oxia resources and data, performing actions as the compromised user.

Impact

The exposure of bearer tokens can lead to unauthorized access to Oxia resources. An attacker with access to application logs can extract valid JWT tokens and impersonate legitimate users. The impact depends on the privileges associated with the compromised user account. This vulnerability affects all Oxia deployments using OIDC authentication with debug logging enabled in production, potentially impacting a large number of users and sensitive data stored within the database.

Recommendation

Ensure DEBUG-level logging is never enabled in production environments as a primary mitigation step (Workarounds).
Deploy the Sigma rule to detect instances of failed authentication attempts where the full bearer token is logged, and investigate immediately ([Sigma rule: Detect Oxia Token Leak in Logs]).
Upgrade to a patched version of Oxia that redacts the token in log output, preserving only the last 8 characters.
Review and harden access controls for application logs and log aggregation systems to prevent unauthorized access.