Attack Chain

1. Attacker compromises the publisher account for the pytorch-lightning package on PyPI.
2. Attacker publishes malicious versions 2.6.2 and 2.6.3 to PyPI.
3. A modified __init__.py file within the package initiates a background process upon import.
4. The background process executes silently, without any visible output or indication of compromise to the user.
5. The malicious package downloads a runtime (Bun) from GitHub.
6. The package executes a large, obfuscated JavaScript file, targeting AWS, Azure, Google Cloud, GitHub, and local credential stores.
7. Stolen credentials, including cloud provider keys, API tokens, and secrets, are exfiltrated to attacker-controlled infrastructure.
8. The malware attempts to download and execute a second-stage payload from attacker-controlled infrastructure, expanding the scope of the attack.

Impact

Organizations that downloaded and used versions 2.6.2 or 2.6.3 of the pytorch-lightning package are at high risk of compromise. The malicious package is designed to steal a wide range of credentials, including cloud provider keys, API tokens, and secrets stored in environment variables. This can lead to unauthorized access to sensitive data and systems, potentially resulting in data breaches, financial losses, and reputational damage. The malware’s ability to download and execute secondary payloads further increases the potential impact.

Recommendation

• Immediately remove versions 2.6.2 and 2.6.3 of the lightning package from all systems where they are installed (see overview).
• Audit systems for unauthorized processes and review outbound network connections to detect potential compromises (see overview).
• Rotate all cloud provider keys (AWS, Azure, GCP), API tokens (GitHub, CI/CD systems), and secrets stored in environment variables to prevent further unauthorized access (see Attack Chain).
• Implement the Detect Suspicious PyPI Package Installation Sigma rule to identify potential malicious packages being installed in the future (see rules).
• Implement the Detect Credential Harvesting via Bun Sigma rule to catch execution of the malicious JavaScript payload (see rules).
• Pin dependencies to known-good versions and verify package integrity before use to prevent future supply chain attacks (see references).

Attack Chain

1. The attacker gains unauthorized access to PyPI credentials for the telnyx package.
2. The attacker uploads malicious versions 4.87.1 and 4.87.2 of the telnyx package to PyPI, bypassing the legitimate GitHub repository.
3. When a user installs or upgrades to the malicious telnyx package, the injected malware within telnyx/_client.py executes upon importing the library (import telnyx).
4. On Linux/macOS systems, the malware spawns a detached subprocess to ensure persistence and downloads a payload hidden inside a WAV audio file (ringtone.wav) from the C2 server at http://83.142.209.203:8080/.
5. The downloaded payload harvests sensitive credentials, including SSH keys, AWS/GCP/Azure credentials, Kubernetes tokens, Docker configurations, .env files, database credentials, and crypto wallets.
6. If Kubernetes access is detected, the malware deploys privileged pods to all nodes for lateral movement within the Kubernetes cluster.
7. The collected data is encrypted using AES-256-CBC and RSA-4096, then exfiltrated to the C2 server, identified by the header X-Filename: tpcp.tar.gz.
8. On Windows, a binary payload hidden in hangup.wav is downloaded from http://83.142.209.203:8080/, dropped as msbuild.exe in the Startup folder for persistence, and executed with a hidden window, polling the endpoint http://83.142.209.203:8080/raw.

Impact

The compromise of the telnyx PyPI package poses a significant risk to developers and organizations that use the library. Successful exploitation leads to the theft of sensitive credentials, potentially granting the attacker unauthorized access to critical infrastructure, cloud resources, and sensitive data. TeamPCP’s previous campaign against LiteLLM and the similarities in this attack suggest a pattern of targeting open-source projects to infiltrate developer environments and steal secrets. The impact includes potential data breaches, financial losses, and reputational damage. The exposure window was approximately 6 hours during which vulnerable versions were available.

Recommendation

• Immediately check for the presence of malicious telnyx package versions (4.87.1 or 4.87.2) in your environment using the provided commands and uninstall them (pip uninstall telnyx).
• Due to the credential-stealing nature of the malware, rotate all potentially exposed secrets, including SSH keys, cloud provider credentials (AWS, GCP, Azure), Kubernetes tokens, Docker registry credentials, database passwords, API keys in .env files, and Telnyx API keys.
• Check for persistence mechanisms used by the malware, specifically the audiomon service and associated files on Linux/macOS, and the msbuild.exe executable in the Startup folder on Windows, based on the file paths provided in the “Filesystem” section.
• Block the identified C2 IP address (83.142.209.203) and payload URLs (http://83.142.209.203:8080/ringtone.wav, http://83.142.209.203:8080/hangup.wav, http://83.142.209.203:8080/raw) at your network perimeter.
• Deploy the following Sigma rule to detect the creation of msbuild.exe in the Startup folder.
• Pin the telnyx package to the safe version 4.87.0 in your project dependencies to prevent future installations of compromised versions.

Attack Chain

1. TeamPCP gains unauthorized access to the Telnyx PyPI account, likely through credential theft.
2. Malicious versions 4.87.1 and 4.87.2 of the Telnyx package are published to PyPI.
3. When a developer installs the compromised Telnyx package, the telnyx/_client.py file is executed upon import.
4. On Linux and macOS, a detached process is spawned to download a second-stage payload disguised as a WAV audio file (ringtone.wav) from a remote command-and-control (C2) server.
5. Steganography is used to hide malicious code within the WAV file’s data frames.
6. The embedded payload is extracted using an XOR-based decryption routine and executed in memory.
7. The malware harvests sensitive data, including SSH keys, credentials, cloud tokens, cryptocurrency wallets, and environment variables.
8. If Kubernetes is present, the malware enumerates cluster secrets and deploys privileged pods to access underlying host systems. On Windows, a different WAV file (hangup.wav) is downloaded that extracts and saves an executable named msbuild.exe to the startup folder for persistence.

Impact

This supply chain attack could result in widespread compromise of systems utilizing the Telnyx Python SDK. Over 740,000 monthly downloads indicate a large potential victim pool. Stolen credentials and secrets can lead to unauthorized access to cloud resources, sensitive data exfiltration, and further lateral movement within compromised networks. For systems running Kubernetes, the attacker could gain control over the entire cluster, leading to significant disruption and data loss. Developers who installed the malicious packages are advised to consider their systems fully compromised and rotate all secrets as soon as possible.

Recommendation

• Identify and remove Telnyx versions 4.87.1 and 4.87.2 from all environments, reverting to version 4.87.0 as recommended by the vendor.
• Monitor network connections for processes spawned by Python interpreters (python.exe, python3) attempting to download files with the .wav extension, using the “Detect Suspicious Python WAV Download” Sigma rule provided below.
• Implement stricter controls and multi-factor authentication for PyPI accounts used to publish packages to prevent similar supply chain attacks.
• Deploy the “Detect msbuild.exe in Startup Folder” Sigma rule to identify potential persistence attempts on Windows systems.
• Rotate all secrets and credentials on any system that has imported the malicious Telnyx package.

Attack Chain

While the specifics of the attack chain are not fully detailed in the source, a typical supply chain attack targeting PyPI packages involves the following steps:

1. Package Compromise: Threat actor gains unauthorized access to the Litellm PyPI account or the build environment.
2. Malicious Code Injection: The attacker injects malicious code into the setup.py or other relevant files within the Litellm package. This malicious code could be designed to execute upon installation.
3. Version Release: The compromised versions, 1.82.7 and 1.82.8, are released to PyPI, making them available for users to download and install.
4. Package Installation: Users unknowingly download and install the compromised Litellm package using pip, triggering the execution of the injected malicious code.
5. Initial Access: The malicious code may establish a reverse shell, download additional payloads, or perform other actions to gain initial access to the victim’s system.
6. Persistence: The attacker may establish persistence on the compromised system through various techniques, such as creating scheduled tasks or modifying startup scripts.
7. Data Exfiltration/Malware Deployment: Depending on the attacker’s objective, they may exfiltrate sensitive data, deploy ransomware, or perform other malicious activities.
8. Lateral Movement: The attacker may attempt to move laterally to other systems within the compromised network, escalating their access and expanding their reach.

Impact

The compromise of Litellm versions 1.82.7 and 1.82.8 could lead to widespread compromise of systems that use the package. The injected malicious code could enable attackers to steal sensitive information, deploy malware, or gain unauthorized access to victim systems. The number of affected users is estimated to be in the thousands. This incident highlights the risks associated with supply chain attacks targeting open-source software repositories.

Recommendation

• Immediately stop updating to Litellm versions 1.82.7 and 1.82.8.
• Revert to a known-good version of Litellm prior to 1.82.7.
• Analyze network connections for suspicious traffic originating from systems where the compromised Litellm versions were installed, using network connection logs.
• Monitor process creations for suspicious processes spawned from Python executables where Litellm is installed, using process creation logs and the Sigma rules provided below.
• Investigate systems where Litellm 1.82.7 or 1.82.8 were installed for any signs of compromise.
• Review the blog post at https://futuresearch.ai/blog/litellm-pypi-supply-chain-attack/ for further details on the compromise.