Skip to content

Moving a Paravirtualized EC2 legacy instance to a modern HVM one

I had to try a few things before I could get this right, so I thought I'd write about it. These steps are what ultimately worked for me. I had tried several other things to no success, which I'll list at the end of the post.

If you have Elastic Compute Cloud (EC2) instances on the "previous generation" paravirtualization based instance types, and want to convert them to the new/cheaper/faster "current generation", HVM instance types with SSD storage, this is what you have to do:

You'll need a donor Elastic Block Store (EBS) volume so you can copy data from it. Either shutdown the old instance and detach the EBS, or, as I did, snapshot the old system, and then create a new volume from the snapshot so that you can mess up without worrying about losing data. (I was also moving my instances to a cheaper data center, which I could only do by moving snapshots around). If you choose to create a new volume, make a note of which Availability Zone (AZ) you create it in.

Create a new EC2 instance of the desired instance type, configured with a new EBS volume set up the way you want it. Use a base image that's as similar to what you currently have as possible. Make sure you're using the same base OS version, CPU type, and that your instance is in the same AZ as your donor EBS volume. I mounted the ephemeral storage too as a way to quickly rollback if I messed up without having to recreate the instance from scratch.

Attach your donor EBS volume to your new instance as sdf/xvdf, and then mount them to a new directory I'll call /donor
mkdir /donor && mount /dev/xvdf /donor


Suggested: Mount your ephemeral storage on /mnt
mount /dev/xvdb /mnt
and rsync / to /mnt
rsync -aPx / /mnt/
If something goes wrong in the next few steps, you can reverse it by running
rsync -aPx --delete /mnt/ /
to revert to known working state. The rsync options tell rsync to copy (a)ll files, links, and directories, and all ownership/permissions/mtime/ctime/atime values; to show (P)rogress; and to not e(x)tend beyond a single file system (this leaves /proc /sys and your scratch and donor volumes alone).

Copy your /donor volume data to / by running
rsync -aPx /donor/ / --exclude /boot --exclude /etc/grub.d ...
. You can include other excludes (use paths to where they would be copied on the final volume, not the path in the donor system. The excluded paths above are for an Ubuntu system. You should replace /etc/grub.d with the path or paths where your distro keeps its bootloader configuration files. I found that copying /boot was insufficient because the files in /boot are merely linked to /etc/grub.d.

Now you should be able to reboot your instance your new upgraded system. Do so, detach the donor EBS volume, and if you used the ephemeral storage as a scratch copy, reset it as you prefer. Switch your Elastic IP, or change your DNS configuration, test your applications, and then clean up your old instance artifacts. Congratulations, you're done.

Notes:
Be careful of slashes. The rsync command treats /donor/ differently from /donor.

What failed:
Converting the EBS snapshot to an AMI and setting the AMI virtualization type as HVM, then launching a new instance with this AMI actually failed to boot (I've had trouble with this with PV instances too with the Ubuntu base image unless I specified a specific kernel, so I'm not sure whether to blame HVM or the Ubuntu base images.
Connecting a copy of the PV ebs volume to a running HVM system and copying /boot to the donor, then replacing sda1 with the donor volume also failed to boot, though I think if I'd copied /etc/grub.d too it might have worked. This might not get you an SSD backed EBS volume though, if that's desirable.

Profiteering and the singular ant problem

I found this video highly interesting. Why do we have to have such heavy blinders on about healthcare in this country? Regardless of the position you hold on the Affordable Care Act, everyone should understand the facts presented in this video:



As a summary:

  • We pay more per capita in taxes for healthcare than any other country, but only a few get the benefits of a single payer health system

  • We pay more as a percentage of GDP on healthcare than any other country

  • Our care is not better (as measured by outcomes) than any other country with socialized medicine

  • It's not one thing that causes high healthcare costs, but a variety of things, but the common causes dragged out for public vilification are not the most significant factors

  • One cannot negotiate for better prices for healthcare while receiving care, so prices will naturally increase to the level the market will bear, which is pretty high



I personally think it's unethical to profit by the treatment (or cause for that matter) of suffering. That doesn't mean the healthcare provider shouldn't be compensated for his or her time, equipment, and materials and that they shouldn't be allowed to expand their business or invest in research and development of new treatments, but the provider administrators shouldn't be beholden to stakeholders demanding higher profit margins, and administrators in turn demanding more high margin tests to be ordered by providers when deciding what care to give to a patient.

Government Social Programs vs. Private Charity

"Oh, if government didn't provide for the poor and needy, then charity would! Small local charities would be vastly more efficient than Government, and provide better services to boot!" This, first of all, ignores simple economies of scale; small local charities would have much more overhead/dollar and much less purchasing power than large national charities. Second, if natural charity worked with modern man, we'd still live in a Victorian era of asylums, orphanages, and debtors' prisons. It didn't. We don't, and we're better off today because we've recognized that fact. Could charity cover the gap if the Government closed down all it's social programs? My math says no. I wish I was wrong, but history has shown that the poor, diseased, or downdrodden are nearly universally ignored by society as a whole, so if we reduced taxes by the $1.3+ trillion spent every year on social programs, the $300 billion a year in charitable contributions wouldn't increase to cover the gap, even if we were able to get incredible efficiency gains. Charitable contributions nearly universally come from disposable income; we don't naturally sacrifice our own standard of living to raise another.

Not that Government run social programs are perfect: we waste tons of money by making Social Security payments mandatory, you receive them whether you really need them or not; by allowing welfare recipients to shop for themselves at retail (even at convenience store prices) vs. using the economies of scale to buy foodstuffs and other essentials at wholesale prices; using subsidies and tax credits instead of collective bargaining to make essential services (phone, water, power, housing) more affordable, etc; and paying most benefits out in CASH. There's even a measurable (sizeable even?) amount of graft and abuse. But despite their faults, and the levels of abuse, they provide some level of social safety net that didn't exist 30-80 years ago, and I think society is better for it.

An aside about the graft; Florida recently required all welfare recipients to take a drug test. They found that less than 2% of applicants failed the tests. From this we learned at least three things:
  1. that the 4th amendment doesn't allow the government to require invasive tests of worthiness for social programs.

  2. that contrary to popular belief, most welfare recipients are not drug addicts.

  3. that implementing bureaucracy to police graft may save some money, but not significant amounts.


So if you're going to propose to me that we "Abolish Social Security" or cut social services in some other way, you'd better be prepared to explain to me
  1. where those people will go for help when they need it.

  2. how those charities would receive the funding they would need to care for the large numbers of people who would need their services, and

  3. How that makes us a stronger society than the status quo.


Robin Hood Economics (or Hey Look Joseph's finally posting to his blog again!)

I find myself composing long replies to Facebook posts, but I feel Facebook is a poor medium for sharing such weighty ideas, so I'm going to start/resume posting here instead. Hopefully my friends and family recognize this as an opportunity to discuss civilly vs. as an attack on their deeply held political beliefs. I don't have all the answers, and I consider myself to be fiscally conservative and socially liberal, so none of the current political parties interest me much, and some of what's going on in Washington, D.C. and in Raliegh, N.C., really worries me.

Here's my rant for today:

Why is it that when we tax the poor to give to the rich it's called "social darwinism" or "lucrative government contracts" or "trickle down economics", and it's ok. But when we talk about taxing the rich to give to the poor it's "forced wealth redistribution", or "communism", (socialism doesn't mean what you think it means) and it's not ok? The former is really Prince John's feudalism, so wouldn't the latter be Robin Hood's economic policy? Which should we be cheering?

I guess my real point is: it's hard to have meaningful discussions about hot topics when the terms of the debate are set up to be so emotionally charged.

Red hots! Get Your Red Hots!

When I was in high school, I sold drinks and hot dogs for the University of Utah home football games to earn a little extra money*. I walked around the stadium with a big tray of sodas or an insulated box of hot dogs. I'd call out "Coke, Diet Coke, Sprite" or "Red hots! Get your red hots!". Ten years later, I find out that Red Hots are a real product, and they are awesome. Red Hots are stubby, thick hot dogs, colored red, with a bit of spiciness. I sat down to have a couple for lunch today with 'kraut and brown mustard and reminisced about cold Utah Saturday mornings at (then) Rice Stadium walking up and down the aisles and circling the stadium. If only the hot dogs I sold as "red hots" 16 years ago had been real Red Hots, I would have been able to sell twice as many.

* I say a little, but as a vendor, you started out selling drinks. A hard working vendor could make between $50 and $100 in about 2.5 hours (they stopped refilling trays at the end of the 3rd quarter) and you could watch the end of the game anywhere you wanted to. If you had been there long enough and worked hard enough, you get promoted to hot dogs. A hot dog vendor could make $150 to $200 in the same time frame. As a high schooler in the early 90s, that beat working at Hires.

Web Site Hosting Advice

Turbogears Occasionally friends, relatives, and clients ask me what they should do about creating and hosting a web site. When this happens, I find myself repeating, well, myself; so I thought I would put my thoughts on virtual paper for future reference. I will post a notice on this entry if my recommendations change at some future date. If you would like to consult with me about your particular setup, please contact me for consulting rates and availability.

Ok, you want a web site, good. First, get an idea of what your website will contain, how big it will be, what kind of content you will serve, and how much traffic it will receive. Will it DO something or SHOW something. If you're just starting out, or have no idea, any of the recommended plans will let you scale size and traffic for additional monthly fees, so don't worry too much about it.

If your goal is an informational, mostly text, but low volume, web site, just get a BlogSpot.com or other blog hosting account. They are free, minimally annoying, and with free image galleries and video hosting sites, can link to or embed video and photo content too. My Ward (a congregation in the LDS church) has a few of these sites for various extra activities, for example the youth group is presenting a "Fancy Dance" and Dessert Auction on Saturday Feb 19, 2011 to raise money for camp and activities this year, and uses BlogSpot to advertise. By the way, everyone is invited to the dance, and babysitting is provided, see the site for more information.

If your goal is to sell something, sell through the Amazon marketplace or Etsy.com if the products are crafty. Piggyback on top of an existing marketplace to jump start sales. If you're too big for that, I don't really have any advice. I don't have any experience in that space. I think that I would look for a host that provided merchant services (credit card processing for example) as part of the package.

If your goal is to host a medium volume dynamic application, use WebFaction. WebFaction is probably the best Shared Hosting service there is. They're one of the very few hosting providers that embraces Python application hosting, and I've run Pylons, TurboGears and CherryPy applications there. The hosting is cheap, fast, and it stays out of your way if you want it to. I host this blog, my personal e-mail and my business website on the base level account. I also host demo sites for clients when needed. The email service isn't spectacular, but it's functional as long as you have client side spam filtering like what is provided by Thunderbird. I like it because there are no set CPU limitations, the memory allotment is generous (email, OS, and even Database memory usage doesn't count against your quota, though the disk usage does), and the base disk space/bandwidth allocation is substantial. It also helps that WebFaction takes care of all data backups and operating system and hardware maintenance for you. WebFaction has one click installers for a large number of applications, so you don't have to know very much about Linux to get started, but if you do know what you're doing, you have SSH access, and everything that comes standard with a Linux shell account.

If you are planning on building a new application, take a look at Google App Engine. It lets you get going and host up to a certain threshold for free. Scaling up can be done fairly reasonably. Applications developed for App Engine can be run independently of Google, so you are not necessarily locked to Google as your hosting vendor.

I do not recommend any kind of Virtual Private Server hosting that isn't bundled as a Cloud offering. I've used three different VPS services, and two have all been slow and had high network latency (the third, Slice Host was bought and extended into Rackspace's cloud services, which I recommend below). Higher volume sites may do OK, but if the CPU, IO or Memory usage is too high for too long, your VPS can be rebooted or shut off. What this translates to is that you would have to hit a very small sweet spot to get good performance out of a VPS without getting shut down. Better hosting options exist.

If you do need system level access to a server of your own for some reason -- if for example you have an email processing system as part of your application -- or if you have requirements that extend beyond a single host, like high availability, then using a Cloud based VPS is desirable. Cloud computing nodes are designed for high performance application hosting. The overhead of virtualization is minimized by the use of advanced virtualization techniques (paravirtualization, CPU instruction sets, etc.) and by dedicating virtual resources to physical hardware. The management tools are typically excellent and, in the case of my two favorite cloud providers, there is an inherent benefit of a content delivery network (CDN) and Storage Attached to Network (SAN) which can serve as a scalable long term application storage or system backups. These two tools are used by very large websites to deliver content faster and more efficiently, and they're available on the Cloud for even the lowest rate plans. The intro level computing node at Amazon Elastic Compute Cloud (EC2) starts at 3¢/hour. Rackspace however has a node that start as low as $10.95/month (that's about 1.5¢/hour). There aren't as many third party software developers, and no external image providers (as far as I know) for Rackspace, but they have pretty good management tools, and a pretty good selection of base images to get you up and running pretty quickly.

EC2 was built for running short-lived computing (i.e., processor intensive) tasks, and it's pricing model and instance sizes reflect that. The instances and costs are very competitive to people looking at dedicated hosting. Rackspace's cloud is similarly designed, but has smaller instances, so it is cheap enough to use as a substitute for VPS or even shared hosting.

A former coworker of mine recently signed up for EC2 to host his blog using a promotional deal offered by Amazon's EC2. This deal lets you use the Micro instance for up to 750 hours per month for a whole year. Thereafter he's looking at a starting monthly rate of $21.60 plus storage and bandwidth charges. Of course using a Cloud node to host a blog is seriously overkill (as evidenced by his load average) unles he is doing much more with his site than visible at first glance. If he is uncomfortable with a free or even a paid blog hosting account, either WebFaction or Rackspace Cloud would be sufficient to host his site at about half the cost of EC2.

There is also dedicated hosting, but with the price point and performance of EC2 and Rackspace Cloud, you'd have to be very big indeed, or have special criteria not available for cloud nodes for the benefits to outweigh the costs.

Here's what I use for myself and my clients, and why I don't recommend VPS hosting:

As I mentioned above, I currently host my blog, email and business website on a WebFaction Shared Hosting plan. Shared Hosting starts at less than $10/month, with steep discounts for prepayment. I moved all the services off my VPS at Linode and shut it down since WebFaction was working so well. I found Linode to be sluggish and and network traffic to be high latency, but haven't felt that way about Webfaction.

With InMotionHosting's VPS offerings, performance was similar to or worse than Linode's. I had a client on the fully managed VPS plan costing $90/month. The VPS would bog down during traffic peaks and InMotion's system administrators would reboot the box (without any advance warning, without notice after the fact and without explanation of why). When things were peaceful, trying to log in to SSH could take 30-45 seconds, page loads for the main site or core application could take several seconds in spite of caching and being rather lightweight. InMotion always seemed to want to upsell to dedicated hosting when I mentioned the problems to their customer service representatives.

This site/application just passed through its busiest season on a Rackspace Cloud Server instance, and the it never even hiccuped. Final cost for hosting for the month? $24, and plenty of room to scale up if volume increases. I recommended the Rackspace Cloud Server because the application has an email processing system and the client has clients that could have been squeamish if their customers' names and email addresses were available on a shared host's shared database server (even though the database itself was not shared and was password protected).

As Promised to TriZPUG: EPDB

So I did some digging around after giving my off-the-cuff lightning talk at TriZPUG tonight and it looks like some other ex-rpathers (Thanks Dugan and Gafton!) have forked epdb. There's also the the rPath tree synchronized from here but this tree is missing some of the latest changes. The dugan tree is "python setup.py installable" now, instead of using make, and some shortcut documentation has been created, so I don't have to make this post as long as I thought I was going to have to.

For those who didn't see my little demo, epdb is like pdb (the standard Python debugger), but it adds multi-line text input, history and tab completion, nested debugging from the debug prompt, shortcuts to introspecting code, and a very nice post mortem debugger. Last, but not least, it also contains a server and client for remote debugging. The docs are still pretty sparse, but hopefully more attention can help fix that. I'd also be happy to answer questions about it.

Better E-mail validation

Turbogears Due to several shortcomings of the stock formencode email validator, I forked it and extended the test suite. This fixes the two most glaring issues I know of, namely the inability to handle unicode strings (international domains), and several problems with input checking (e.g., allowing commas) where invalid e-mail addresses make it through.

I did not write most of the code, I just refined it and added tests to exercise it. Let me know if it's useful to you, and if you find problems with it.

New Biofuel System

This technology is probably the only biofuel technology I'm really excited about. Unlike E85, it doesn't use food crops, and although algae based programs don't compete directly with food crops, they still require fermentation of cellulose, or refining of algae produced oil to create fuel. More links here and here. Joule Biotechnologies website here. I've been thinking for a while that we should be able to extract CO2 from the atmosphere and produce fuel. Now Joule has gone and built something that might be able to do that.