Most system administrators look at virtual servers such as Amazon’s EC2 or RackSpace’s Cloud Servers as a boon to IT, a way to scale to meet problems quickly, easily, and less expensively. Server virtualization is all the rage, and virtualization has replaced “web 2.0″ in every vendors lexicon. The advantages of cloud based virtualized servers don’t just apply to your IT department. They also apply to the people that send spam, control botnets, and evade security. Here’s a look at the darker side of virtualized server applications.
Spammy Spam Spam
Late last year SpamHaus added both Amazon’s EC2 server IPs and Slice Host IPs to their block list. SpamHaus’ DNSBL is widely used, and this meant that users running mail servers on EC2 or Rack Space servers suddenly found their email being refused. This left the EC2 users in quite a conundrum. Email administrators weren’t going to stop using SpamHaus. It was reported that Amazon’s “hands off” approach exacerbated the situation, hurting both EC2 customers and Amazon’s reputation.
A scalable email architecture
To give you an idea of how serious this is, as a test, GeeForce stopped using SpamHaus, but retained the excellent NJABL and SpamCop’s RBL. In just a few hours we saw mail server CPU loads increase and a sizable increase in the number of spam messages that were making it into the queue. While they got tagged by spam assassin or dspam, these emails would have never made it past the inbound smtp server (ie Zone 1) with SpamHaus in place.
Today most spam is sent by computers that have been compromised and are part of a botnet. The rise of virtualized machines, coupled with the ability to script the start and stop of servers, have provided spammers a new tool. Now they have the ability to turn up servers with a lot of bandwidth quickly, spam through them and then take them down. The next time an instance is started it has a new IP address. This makes it harder for RBLs to be effective, with more “collateral damage” as a result.
Sending spam isn’t the only way spammers can use and abuse virtualized servers. Virtualized hosts can run bots to collect email, post spam to blogs, and to click on advertisements. Content can be hosted on virtulized servers with lots of bandwidth and that content can be moved, updated, and changed incredibly quickly thanks to programming hooks that virtual server providers build in. That means that the phishing or malware site that you block can change it’s location in a heartbeat. Unlike cheap shared hosting these websites have the bandwidth to deal with large influxes of traffic.
The same quick provisioning and moving of services also applies to botnet command and control. Virtual servers can be used to control large botnets, or even build large botnets. Why go to the trouble to compromise machines with low upstream bandwidth when you can rent machines for pennies that can push more bandwidth than most companies have? It’s not difficult to script and turn up several large servers that would only be live for a short time, but with enough bandwidth and throughput to wreak havoc on almost any website. MySpace does this for testing, others can do the same exact thing with criminal intent. An extortionist only needs to bring a big company down for 15 minutes to make their point.
A powerful dedicated attack machine
One of the primary aspects that makes virtualized servers so attractive to “hackers” is the ability to control large amounts of CPU power and bandwidth relatively cheaply. With the widespread use of pre-paid credit cards, purchasing that power can be anonymous. This provides the Internet attacker with a scalable, scriptable attack machine.
Imagine the power the hacker feels. Dictionary attacks can be launched against multiple targets, vulnerability scanners can process thousands of URLs, and the results mailed off site for the attacker to peruse at their convenience. Attackers are no longer limited by the fact that most consumer upstream bandwidth is relatively meager or limited CPU cycles dedicated to an attack. Today’s virtualized servers can be optimized for a specific attack, with most of the machine’s CPU dedicated to that end.
Location obfuscation
Example of using virtual servers to circumvent content monitoring
Our analysis of a recent attack against a web based email program showed that the penetration attempt came through a secured proxy running on a virtualized server. The server was taken down within an hour of the attack ending. These types of attacks often run through several servers chained together. Now instead of relying on a number of compromised machines with wildly variable latency between them, attackers can script the provisioning of several machines through several accounts all with low machine-to-machine latency. Add a couple of compromised consumer hosts in the middle of these chains and it becomes next to impossible to find the perpetrator.
Virtualized servers can also be used to avoid content filters, firewalls, and other related devices. The ability to easily route through multiple machines gives IT security officers a new problem to worry about. Traffic to a virtualized server might be legitimate, or it might be a user trying to circumvent the corporate network controls.
This type of proxying and network obfuscation has it’s legitimate purposes as well. Just as MySpace uses cloud servers to load test their application, there are Asian political activists that use virtual cloud servers to circumvent government censors and spying. Using several virtual cloud servers combined with CD/DVD based operating system distribution like dsl (Damn Small Linux) provides a layer of security for the activist that is very hard for a government to penetrate.
The unintended consequence
The human imagination is a wonderful thing. In the coming years we will see virtual servers used in ways that the developers of virtual environments never envisioned. The side effect is that some of those uses will be dishonest and criminal. For providers of virtual servers the race is about to begin in earnest to secure their servers against malicious use.
The consumers of virtual servers rely on the trustworthiness of the provider. Once a provider earns a reputation for poor security and response, network admins can be expected to block traffic from that provider. If such blocks become common place, then the virtual servers allocated by that provider are useless. Just as some email admins had to move their email out of the EC2 cloud, a poor security track record will force users to move their cloud based servers back to their own data centers or to other providers.
Cloud based virtual servers are a double edged sword. To keep from getting cut, IT professionals will have to modify their tactics, security policies, and update their assumptions. Virtual servers provide attractive pricing for high bandwidth servers that can be “scripted” into existence as needed for almost any application. Unfortunately some of those applications can be malicious and securing your cloud server harder than you imagine. Providing good recommendations to your company means understanding the threats as well as the benefits created by the expanding virtual server landscape.
UPDATE 4-17-2010: Since posting this article there have been 6 responses. All of them spam, and 4 of the 6 responses came from Amazon EC2 servers. The irony, spammers using bots on EC2 servers to try and post spam to an article about spammers using cloud based servers.