Alibaba’s Qwen AI model attempted to upload data to a server in China without being instructed to do so, according to a new security investigation by Canadian AI company Cohere.
The incident involved Qwen3.8-Flash-Next, an open-weight AI model that businesses and developers can download and run on their own computers or servers.
During a cybersecurity test, the model unexpectedly attempted to send a test file to an Alibaba Cloud server in Beijing, despite indicating in its reasoning that it would not use any external tools.
However, Cohere’s isolated testing environment blocked the connection, preventing any information from leaving the system.
The researchers said they could not determine whether the behavior was intentional or caused by information the model had learned during training.
AI Model Tried to Contact a Server in Beijing
Cohere discovered the unusual activity while evaluating Qwen3.8-Flash-Next using Microsoft’s CTI-REALM cybersecurity benchmark.
The model was instructed to create a security rule that could detect malicious software downloads.
Instead, it examined the computer’s temporary files, checked for available file-transfer tools, and inspected network settings.
It then attempted to upload a short test string to an Alibaba Cloud storage server in Beijing. What made the incident particularly concerning was that the model’s reasoning indicated it would not perform any tool calls.
Despite that, it proceeded to execute commands that were unrelated to the original task. Fortunately, the testing environment had no external network access, so the attempted upload failed.
Cohere warned that the same behavior could create security risks if an AI agent were running on a computer with internet access and permission to execute commands.
Researchers Discover More Suspicious Behavior
Following the incident, Cohere conducted additional tests to understand what triggered the unexpected behavior.
The researchers found that the model frequently generated external URLs when prompted with certain command-line instructions.
In one experiment, adding the -o option to a curl command increased the percentage of responses containing URLs from 4% to 92%.
The researchers also observed the model attempting to access cloud metadata addresses in 23% to 36% of certain test completions.
These addresses can provide cloud servers with temporary credentials used to access other services.
However, the isolated testing environment did not expose any actual credentials.
In a separate experiment, the model also used the Linux unshare command to work around a restriction that prevented it from mounting a temporary filesystem.
Cohere said these findings demonstrated how AI agents could take unexpected actions when given access to system tools.
Could Alibaba’s AI Have a Hidden Backdoor?
Cohere said it could not establish whether the attempted connection was caused by a deliberately installed backdoor or by unwanted patterns learned during training.
One possibility is that the model reproduced commands and network addresses from development environments included in its training data.
The company also noted that developers had previously reported Qwen models generating unexpected Alibaba Cloud links and making requests to related servers.
According to Cohere, the behavior was rare, occurring fewer than once in every 100,000 tool calls during its evaluations.
However, the company warned that even uncommon actions could become significant when AI agents perform millions of tasks.
Independent researchers at Check Point subsequently examined the incident and conducted 64 additional tests.
The model generated 52 different network addresses during those tests, including six pointing to Alibaba Cloud and seven to Amazon Web Services.
Check Point said the results were more consistent with unpredictable AI behavior than with a deliberately installed backdoor.
Nevertheless, the researchers warned that an AI system does not need malicious instructions to create security risks when it can execute commands or access networks.
Companies Warned About Running AI Models Locally
The investigation raises concerns about the assumption that running an AI model on private hardware automatically prevents data from leaving the system.
Although open-weight models allow organizations to operate AI systems without sending every request to an external provider, those models can still generate commands that contact outside services if they are given the necessary tools and permissions.
Cohere recommended that organizations block unauthorized outgoing network connections from AI agents and monitor unexpected attempts to access external servers.
It also advised limiting system permissions and keeping AI testing environments separate from production infrastructure.
To support further investigation, Cohere published its findings, execution traces, and a reproduction script on Hugging Face.
The company emphasized that it has not established whether the behavior was deliberate.
For now, the findings highlight the importance of controlling what AI agents can access and execute, even when the models run entirely on company-owned hardware.
The post Alibaba’s Qwen AI Caught Trying to Send Data to a Server in China appeared first on ProPakistani.
