Planet OSUOSL

• TransactionExecutor
• TransactionExecutorWithLimitedBackoff
• TransactionExecutorWithInfiniteBackoff

The code can be found at: https://github.com/mozilla/socorro/blob/master/socorro/database/transaction_executor.py

These classes implement methods that accepts a function, a connection context to some resource and arbitrary function parameters. When instantiated and invoked, these classes will call the function passing it the connection and the additional parameters. The raising of an exception within the function indicates that a failure of the transaction: a rollback is automatically issued on the connection context. If the function succeeds and exits normally, then a 'commit' is issued on the connection context.

The first class in the list above is the degenerate single-shot case. It doesn't implement any retry behavior. If the function fails by raising an exception, then a rollback is issued on the connection and program moves on. Success results in a commit and the program moves on.

The latter two classes implement a retry behavior. If the function raises an exception, the Transaction class checks to see if the exception is of a type that is eligible for retry. If it is eligible, then a delay amount is selected and the thread sleeps. When it wakes, it tries to invoke the function again with the same parameters. The time delays are specified by a list of integers representing successive numbers of seconds to wait before trying again. For the class TransactionExecutorWithLimitedBackoff, when the list of time delays is exhausted the transaction is abandoned and the program moves on. The TransactionExecutorWithInfiniteBackoff will never give up, running the last time in the delay list over and over until the transaction finally succeeds or somebody kills the program.

How does the TransactionExecutor determine if an exception is eligible for retry? The connection context object is required to have a couple instance variables and methods to assist in the determination.

First, operational_exceptions defines a collection of exceptions that are eligible for the retry behavior. If one of the exceptions from this collection is raised, the retry behavior is triggered.

conditional_exceptions is a list of ambiguous exceptions that may or may not be eligible for retry. We encountered this with Postgres using psycopg2 on the ProgrammingError exception. Normally, this type of exception would not be retriable because it indicates a fundamental problem with a query such as a syntax error. Syntax errors are not retrible. However, sometimes we get network errors disguised as ProgrammingErrors; these are retriable.

If an exception found in the conditional_exceptions collection is raised, we have to further examine the error to determine if it should result in a failure or retry. The instance method is_operational_exception implemented by the connection class is used to determine in the current exception is retriable or not. In the case of Postgres, we look to the text of the exception to see if it contains the string “EOF”. We know that's a network error, not really a programming error so we can do a retry.

Is this class named poorly? Now that we've got many more external resources using this retry behavior and only Postgres is truly transactional, it seems that the name may not be right. Perhaps ExternalResourceActionRetrier?

This is a photo blog on how I used my Makerbot Replicator2, LED lights from Ikea, and components from Radio Shack & Fry's to make a cool addition to my camper.

I was beginning to look over my shoulder to see if advancing glaciers were going to overrun me.  It seemly has taken forever, but my rewrite last year of the Socorro processors has finally made it to production.  The switchover happened on Thursday afternoon (while I was at the dentist) from the old processor (Processor2008) to the new processor (Processor2012).

The new processor was written as a Fetch-Transform-Save Configman app.  This just means that the steps to process a Firefox crash were encoded in a modular manner and, through configuration, attached to the FTS framework.  Many of the Socorro apps follow that basic FTS model allowing them all to share the same multi-threading framework and storage schemes.

What's the advantage of doing this?

First, sharing code between apps allows us to reduce the shear size of the codebase that we have to maintain.  No more repeating similar mechanisms between apps.   This also allows the steps to accomplish a  task in a Socorro application to be isolated and reduced down to a single action.  For example, in the processor, there is now a single method called 'convert_raw_crash_to_processed_crash'.  That method knows nothing about storage mechanisms, queuing, or threading.  Its target action is its only concern.  The external framework handles all the other details.

Second, with that focus on a single action, the methods can be optimized.  Here's some stats comparing last weekend with Procssor2008 and this weekend with Processor2012:

Average Time from submission to completed processing:

Processor2008: 40 seconds
Processor2012: 16 seconds

Average Time for actual conversion from raw to processed:

Processor2008: 2.7 seconds
Processor2012: 1.7 seconds

I really don't think I need to say anything more. I'm just smiling today.

disclaimer: Processor2012 has only been running for a few days, long term stats will have to be collected to get a more focused picture of the performance characteristics.

“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.”

- Theodore Roosevelt

Talk @ http://svc2baltics.com/ in Vilnus, Lithuania, with a 1000+ students interested in starting up. 

Q&A in UAE w/entrepreneurs and @davemcclure at MAKE.

As I mentioned in my previous blog post, trying out Ganeti can be cumbersome and I went out and created a platform for testing it out using Vagrant. Now I have a PDF guide that you can use to walk through some of the basics steps of using Ganeti along with even testing a fail-over scenario. Its an updated version of a guide I wrote for OSCON last year. Give it a try and let me know what you think!

I totally forgot blogging about this!

Remember how I curate a collection of fail pets across the Interwebs? Sean Rintel is a researcher at the University of Queensland in Australia and has put some thought into the UX implications of whimsical error messages, published in his article: The Evolution of Fail Pets: Strategic Whimsy and Brand Awareness in Error Messages in UX Magazine.

In his article, Rintel attributes me with coining the term "fail pet".

Attentive readers may also notice that Mozilla's strategy of (rightly) attributing Adobe Flash's crashes with Flash itself by putting a "sad brick" in place worked formidably: Rintel (just like most users, I am sure) assumes this message comes from Adobe, not Mozilla:

Thanks, Sean, for the mention, and I hope you all enjoy his article.

Note: This is a cross-post of an article I published on the Mozilla Webdev blog this week.

During the course of this week, a number of high-profile websites (like LinkedIn and last.fm) have disclosed possible password leaks from their databases. The suspected leaks put huge amounts of important, private user data at risk.

What's common to both these cases is the weak security they employed to "safekeep" their users' login credentials. In the case of LinkedIn, it is alleged that an unsalted SHA-1 hash was used, in the case of last.fm, the technology used is, allegedly, an even worse, unsalted MD5 hash.

Neither of the two technologies is following any sort of modern industry standard and, if they were in fact used by these companies in this fashion, exhibit a gross disregard for the protection of user data. Let's take a look at the most obvious mistakes our protagonists made here, and then we'll discuss the password hashing standards that Mozilla web projects routinely apply in order to mitigate these risks.

A trivial no-no: Plain-text passwords

This one's easy: Nobody should store plain-text passwords in a database. If you do, and someone steals the data through any sort of security hole, they've got all your user's plain text passwords. (That a bunch of companies still do that should make you scream and run the other way whenever you encounter it.) Our two protagonists above know that too, so they remembered that they read something about hashing somewhere at some point. "Hey, this makes our passwords look different! I am sure it's secure! Let's do it!"

Poor: Straight hashing

Smart mathematicians came up with something called a hashing function or "one-way function" H: password -> H(password). MD5 and SHA-1 mentioned above are examples of those. The idea is that you give this function an input (the password), and it gives you back a "hash value". It is easy to calculate this hash value when you have the original input, but prohibitively hard to do the opposite. So we create the hash value of all passwords, and only store that. If someone steals the database, they will only have the hashes, not the passwords. And because those are hard or impossible to calculate from the hashes, the stolen data is useless.

"Great!" But wait, there's a catch. For starters, people pick poor passwords. Write this one in stone, as it'll be true as long as passwords exist. So a smart attacker can start with a copy of Merriam-Webster, throw in a few numbers here and there, calculate the hashes for all those words (remember, it's easy and fast) and start comparing those hashes against the database they just stole. Because your password was "cheesecake1", they just guessed it. Whoops! To add insult to injury, they just guessed everyone's password who also used the same phrase, because the hashes for the same password are the same for every user.

Worse yet, you can actually buy(!) precomputed lists of straight hashes (called Rainbow Tables) for alphanumeric passwords up to about 10 characters in length. Thought "FhTsfdl31a" was a safe password? Think again.

This attack is called an offline dictionary attack and is well-known to the security community.

Even passwords taste better with salt

The standard way to deal with this is by adding a per-user salt. That's a long, random string added to the password at hashing time: H: password -> H(password + salt). You then store salt and hash in the database, making the hash different for every user, even if they happen to use the same password. In addition, the smart attacker cannot pre-compute the hashes anymore, because they don't know your salt. So after stealing the data, they'll have to try every possible password for every possible user, using each user's personal salt value.

Great! I mean it, if you use this method, you're already scores better than our protagonists.

The 21st century: Slow hashes

But alas, there's another catch: Generic hash functions like MD5 and SHA-1 are built to be fast. And because computers keep getting faster, millions of hashes can be calculated very very quickly, making a brute-force attack even of salted passwords more and more feasible.

So here's what we do at Mozilla: Our WebApp Security team performed some research and set forth a set of secure coding guidelines (they are public, go check them out, I'll wait). These guidelines suggest the use of HMAC + bcrypt as a reasonably secure password storage method.

The hashing function has two steps. First, the password is hashed with an algorithm called HMAC, together with a local salt: H: password -> HMAC(local_salt + password). The local salt is a random value that is stored only on the server, never in the database. Why is this good? If an attacker steals one of our password databases, they would need to also separately attack one of our web servers to get file access in order to discover this local salt value. If they don't manage to pull off two successful attacks, their stolen data is largely useless.

As a second step, this hashed value (or strengthened password, as some call it) is then hashed again with a slow hashing function called bcrypt. The key point here is slow. Unlike general-purpose hash functions, bcrypt intentionally takes a relatively long time to be calculated. Unless an attacker has millions of years to spend, they won't be able to try out a whole lot of passwords after they steal a password database. Plus, bcrypt hashes are also salted, so no two bcrypt hashes of the same password look the same.

So the whole function looks like: H: password -> bcrypt(HMAC(password, local_salt), bcrypt_salt).

We wrote a reference implementation for this for Django: django-sha2. Like all Mozilla projects, it is open source, and you are more than welcome to study, use, and contribute to it!

What about Mozilla Persona?

Funny you should mention it. Mozilla Persona (née BrowserID) is a new way for people to log in. Persona is the password specialist, and takes the burden/risk away from sites for having to worry about passwords altogether. Read more about Mozilla Persona.

So you think you're cool and can't be cracked? Challenge accepted!

Make no mistake: just like everybody else, we're not invincible at Mozilla. But because we actually take our users' data seriously, we take precautions like this to mitigate the effects of an attack, even in the unfortunate event of a successful security breach in one of our systems.

If you're responsible for user data, so should you.

If you'd like to discuss this post, please leave a comment at the Mozilla Webdev blog. Thanks!

Silicon Valley comes to Beijing and Hong Kong! Awesome crew. 

In my home network, I use IPv4 addresses out of the 10.x.y.z/8 private IP block. After AT&T U-Verse contacted me multiple times to make me reconfigure my network so they can establish a large-scale NAT and give me a private IP address rather than a public one (this might be material for a whole separate post), I reluctantly switched ISPs and now have Comcast. I did, however, keep AT&T for television. Now, U-Verse is an IPTV provider, so I had to put the two services (Internet and IPTV) onto the same wire, which as it turned out was not as easy as it sounds.

tl;dr: This is a "war story" more than a crisp tutorial. If you really just want to see the ebtables rules I ended up using, scroll all the way to the end.

IPTV uses IP Multicast, a technology that allows a single data stream to be sent to a number of devices at the same time. If your AT&T-provided router is the centerpiece of your network, this works well: The router is intelligent enough to determine which one or more receivers (and on what LAN port) want to receive the data stream, and it only sends data to that device (and on that wire).

Multicast, the way it is supposed to work: The source server (red) sending the same stream to multiple, but not all, receivers (green).

Turns out, my dd-wrt-powered Cisco E2000 router is--out of the box--not that intelligent and, like most consumer devices, will turn such multicast packets simply into broadcast packets. That means, it takes the incoming data stream and delivers it to all attached ports and devices. On a wired network, that's sad, but not too big a deal: Other computers and devices will see these packets, determine they are not addressed to them, and drop the packets automatically.

Once your wifi becomes involved, this is a much bigger problem: The IPTV stream's unwanted packets easily satisfy the wifi capacity and keep any wifi device from doing its job, while it is busy discarding packets. This goes so far as to making it entirely impossible to even connect to the wireless network anymore. Besides: Massive, bogus wireless traffic empties device batteries and fills up the (limited and shared) frequency spectrum for no useful reason.

Suddenly, everyone gets the (encrypted) data stream. Whoops.

One solution for this is only to install manageable switches that support IGMP Snooping and thus limit multicast traffic to the relevant ports. I wasn't too keen on replacing a bunch of really expensive new hardware though.

In comes ebtables, part of netfilter (the Linux kernel-level firewall package). First I wrote a simple rule intended to keep all multicast packets (no matter their source) from exiting on the wireless device (eth1, in this case).

ebtables -A FORWARD -o eth1 -d Multicast -j DROP

This works in principle, but has some ugly drawbacks:

1. -d Multicast translates into a destination address pattern that also covers (intentional) broadcast packets (i.e., every broadcast packet is a multicast packet, but not vice versa). These things are important and power DHCP, SMB networking, Bonjour, ... . With a rule like this, none of these services will work anymore on the wifi you were trying to protect.
2. -o eth1 keeps us from flooding the wifi, but will do nothing to keep the needless packets sent to wired devices in check. While we're in the business of filtering packets, might as well do that too.

So let's create a new VLAN in the dd-wrt settings that only contains the incoming port (here: W) and the IPTV receiver's port (here: 1). We bridge it to the same network, because the incoming port is not only the source of IPTV, but also our connection to the Internet, so the remaining ports need to be able to connect to it still.

dd-wrt vlan settings

Then we tweak our filters:

ebtables -A FORWARD -d Broadcast -j ACCEPT
ebtables -A FORWARD -p ipv4 --ip-src ! 10.0.0.0/24 -o ! vlan1 -d Multicast -j DROP

This first accepts all broadcast packets (which it would do by default anyway, if it wasn't for our multicast rule), then any other multicast packets are dropped if their output device is not vlan1, and their source IP address is not local.

With this modified rule, we make sure that any internal applications can still function properly, while we tightly restrict where external multicast packets flow.

That was easy, wasn't it!

Some illustrations courtesy of Wikipedia.

Ganeti is a very powerful tool but often times people have to look for spare hardware to try it out easily. I also wanted to have a way to easily test new features of Ganeti Web Manager (GWM) and Ganeti Instance Image without requiring additional hardware. While I do have the convenience of having access to hardware at the OSU Open Source Lab to do my testing, I’d rather not depend on that always. Sometimes I like trying new and crazier things and I’d rather not break a test cluster all the time. So I decided to see if I could use Vagrant as a tool to create a Ganeti test environment on my own workstation and laptop.

This all started last year while I was preparing for my OSCON tutorial on Ganeti and was manually creating VirtualBox VMs to deploy Ganeti nodes for the tutorial. It worked well but soon after I gave the tutorial I discovered Vagrant and decided to adapt my OSCON tutorial with Vagrant. Its a bit like the movie Inception of course, but I was able to successfully get Ganeti working with Ubuntu and KVM (technically just qemu) and mostly functional VMs inside of the nodes. I was also able to quickly create a three-node cluster to test failover with GWM and many facets of the webapp.

The vagrant setup I have has two parts:

1. Ganeti Tutorial Puppet Module
2. Ganeti Vagrant configs

The puppet module I wrote is very basic and isn’t really intended for production use. I plan to re-factor it in the coming months into a completely modular production ready set of modules. The node boxes are currently running Ubuntu 11.10 (I’ve been having some minor issues getting 12.04 to work), and the internal VMs you can deploy are based on the CirrOS Tiny OS. I also created several branches in the vagrant-ganeti repo for testing various versions of Ganeti which has helped the GWM team implement better support for 2.5 in the upcoming release.

To get started using Ganeti with Vagrant, you can do the following:

git clone git://github.com/ramereth/vagrant-ganeti.git
git submodule update --init
gem install vagrant
vagrant up node1
vagrant ssh node1
gnt-cluster verify

Moving forward I plan to implement the following:

• Update tutorial documentation
• Support for Xen and LXC
• Support for CentOS and Debian as the node OS

Please check out the README for more instructions on how to use the Vagrant+Ganeti setup. If you have any feature requests please don’t hesitate to create an issue on the github repo.

A family of tourists, getting ready to watch the sun set on the Pacific coast. I love silhouette photos like this: It's fun to see the different characters with their body shapes and postures.

We’ve just release version 0.7 of Ganeti Web Manager. Ganeti Web Manager is a Django based web application that allows administrators and clients access to their ganeti clusters. It includes a permissions and quota system that allows administrators to grant access to both clusters and virtual machines. It also includes user groups for structuring access to organizations.

This is the fourth release of Ganeti Web Manager and it contains numerous new features.  It also includes various bug fixes and speed optimizations.  Here is the full CHANGELOG, or read on for the highlights.

Xen Support

Ganeti Web Manager now have full Xen support.  Prior versions could display Xen instances, but now you can create and edit them too.  This as an important addition because Xen is a widely used and mature project.  Now with full hardware virtualization in Linux 3.0, Xen will continue to be an important technology for virtualization.  This was our most often requested feature and we’re glad to have fulfilled it.

Internationalization

Thanks to a large community contribution, internationalization support was added for nearly all aspects of the interface.  Users can switch between their default language and any other.  Currently only a Greek translation is available, but we’d like to see many more languages. If you can read and write another language this is a great opportunity for you to get involved. We’re using Transifex to coordinate people who want to help translate.

Search & Improved Navigation

Administrators of larger cluster can now find objects easier with our search interface.  It includes an Ajax auto-complete feature, along with detailed results.

We’ve also added contextual links wherever we could.  This included ensuring breadcrumbs were properly formatted on each page.  Object Permissions and Object Log were updated to ensure navigating between those screens and Ganeti Web Manager is seamless.

Import Tools

There are now import tools for Nodes.  These work the same as for instances.  The cache updater has also been reworked to support both Nodes and Instances.  It’s now a twisted plugin with modest speed improvements due to Ganeti requests happening asynchronously.

Speed, Scalability, and Bugs

We’ve sought out places where we performed extra and or inefficient database queries.  We identified numerous places where database interaction could be reduced, and pages returned faster.  This is an ongoing process.  We’ll continue to optimize and improve the responsiveness as we find areas of the project we can improve.

Numerous bugs were fixed in both the user interface and the backend.  Notably, the instance creation interface has had several bugs corrected.

Module Releases

We’re building several modules along with Ganeti Web Manager.  The following projects have new releases coinciding with Ganeti Web Manager 0.7:

Django Object Permissions 1.4

• improved user selection widget
• speed improvements

Object Log 0.6

• our first public release
• speed, scalability, and flexibility improvements

Twisted VNC Auth Proxy

• our first public release
• added support for hixie 07 and latest noVNC version.

Want to learn more?

Lance Albertson and I will be speaking about Ganeti & Ganeti Web Manager at several conferences this summer.  Catch us at the following events:

• Open Source Bridge – Hands-on Virtualization with Ganeti
• OSCON – Ganeti Web Manager: Cluster Management Made Simple
• OSCON – Hands-on Virtualization with Ganet (tutorial)

Five OSUOSL co-workers and I recently finished a road trip to Google I/O 2011.  We took two cars on an 11 hour drive through scenic southern Oregon and northern California.  We learned more about Android and other technologies shaping the web.  It was also a great opportunity to spend time with each other outside the office.

Monday night we joined about 30 Google Summer of Code mentors for dinner and drinks hosted by the Google Open Source Programs Office.  We’re always grateful for events that bring together friends old and new.  One developer nervously sat down at our table, professing that he didn’t know anyone.  We might not work on the same project, but we’re all part of the open source community.

The highlight of the conference was the double announcement of Android Open Accessory program and Android @ Home.  Both open up Android to integration with third party devices.  These features coupled with near field communications (NFC) stand to dramatically change how we use our mobiles devices to interact with the world around us.  This is not a new idea.  X10 home automation has existed since 1975.  Zigbee and Z-wave are more modern protocols, but also available for years.  The difference here is 100 million Android users and a half million Arduino hackers.

As Phillip Torrone wrote on the Makezine Blog, “There really isn’t an easier way to get analog sensor data or control a motor easier and faster than with an Arduino — and that’s a biggie, especially if you’re a phone and want to do this.”

It won’t be a short road.  We still have obstacles such as higher costs.  A representative from Lighting Science I spoke to at their I/O booth quoted Android@Home enabled LED lights at $30 per bulb.  Android and Arduino might be the right combination of market penetration, eager hackers, and solid platforms for a more integrated environment.

My favorite session was How To NFC.   NFC (near field communication) is similar to RFID except it only works within a few centimeters.  Newer android phones can send and receive NFC messages any time except when the phone is sleeping.  NFC chips can also be embedded in paper, like the stickers that came in our I/O Badges.  An NFC enabled app can share data such as a url, or launch a multiplayer game with your friend.  It makes complex tasks as simple as “touch the phone here”.  Android is even smart enough to launch an app required for an NFC message, or send you to the market to install the app you need.  Only the Nexus-S supports NFC now, but this feature is so compelling that others will support it soon too.

The other technical sessions were very useful too, whether you were interested in Android, Chrome, or other Google technologies.  The speakers were knowledgeable on the subject areas they spoke on.  I attended mostly Android talks, and it was great hearing from the people who wrote the APIs we’re trying to use.  The sessions were all filmed and are worth watching online.

One of the best features of Ganeti is its ability to grow linearly by adding new servers easily. We recently purchased a new server to expand our ever growing production cluster and needed to rebalance cluster. Adding and expanding the cluster consisted of the following steps:

1. Installing the base OS on the new node
2. Adding the node to your configuration management of choice and/or installing ganeti
3. Add the node to the cluster with gnt-node add
4. Check Ganeti using the verification action
5. Use htools to rebalance the cluster

For simplicity sake I’ll cover the last three steps.

Adding the node

Assuming you’re using a secondary network, this is how you would add your node:

gnt-node add -s <secondary ip> newnode

Now lets check and make sure ganeti is happy:

gnt-cluster verify

If all is well, continue on otherwise try and resolve any issue that ganeti is complaining about.

Using htools

Make sure you install ganeti-htools on all your nodes before continuing. It requires haskell so just be aware of that requirement. Lets see what htools wants to do first:

hbal -m ganeti.example.org
Loaded 5 nodes, 73 instances
Group size 5 nodes, 73 instances
Selected node group: default
Initial check done: 0 bad nodes, 0 bad instances.
Initial score: 41.00076094
Trying to minimize the CV...
 1. openmrs.osuosl.org             g1.osuosl.bak:g2.osuosl.bak => g5.osuosl.bak:g1.osuosl.bak 38.85990831 a=r:g5.osuosl.bak f
 2. stagingvm.drupal.org           g3.osuosl.bak:g1.osuosl.bak => g5.osuosl.bak:g3.osuosl.bak 36.69303985 a=r:g5.osuosl.bak f
 3. scratchvm.drupal.org           g2.osuosl.bak:g4.osuosl.bak => g5.osuosl.bak:g2.osuosl.bak 34.61266967 a=r:g5.osuosl.bak f

<snip>

 28. crisiscommons1.osuosl.org      g3.osuosl.bak:g1.osuosl.bak => g3.osuosl.bak:g5.osuosl.bak 4.93089388 a=r:g5.osuosl.bak
 29. crisiscommons-web.osuosl.org   g2.osuosl.bak:g1.osuosl.bak => g1.osuosl.bak:g5.osuosl.bak 4.57788814 a=f r:g5.osuosl.bak
 30. aqsis2.osuosl.org              g1.osuosl.bak:g3.osuosl.bak => g1.osuosl.bak:g5.osuosl.bak 4.57312216 a=r:g5.osuosl.bak
Cluster score improved from 41.00076094 to 4.57312216
Solution length=30

I’ve shortened the actual output for the sake of this blog post. Htools automatically calculates which virtual machines to move and how using the least amount of operations. In most these moves, the VMs may simply be migrated, migrated & secondary storage replaced, or migrated, secondary storage replaced, migrated. In our environment we needed to move 30 VMs around out of the total 70 VMs that are hosted on the cluster.

Now lets see what commands we actually would need to run:

hbal -C -m ganeti.example.org
Commands to run to reach the above solution:

 echo jobset 1, 1 jobs
 echo job 1/1
 gnt-instance replace-disks -n g5.osuosl.bak openmrs.osuosl.org
 gnt-instance migrate -f openmrs.osuosl.org

 echo jobset 2, 1 jobs
 echo job 2/1
 gnt-instance replace-disks -n g5.osuosl.bak stagingvm.drupal.org
 gnt-instance migrate -f stagingvm.drupal.org

 echo jobset 3, 1 jobs
 echo job 3/1
 gnt-instance replace-disks -n g5.osuosl.bak scratchvm.drupal.org
 gnt-instance migrate -f scratchvm.drupal.org

<snip>

 echo jobset 28, 1 jobs
 echo job 28/1
 gnt-instance replace-disks -n g5.osuosl.bak crisiscommons1.osuosl.org

 echo jobset 29, 1 jobs
 echo job 29/1
 gnt-instance migrate -f crisiscommons-web.osuosl.org
 gnt-instance replace-disks -n g5.osuosl.bak crisiscommons-web.osuosl.org

 echo jobset 30, 1 jobs
 echo job 30/1
 gnt-instance replace-disks -n g5.osuosl.bak aqsis2.osuosl.org

Here you can see the commands it wants  you to execute. Now you can either put these all in a script and run them, split them up, or just run them one by one. In our case I ran them one by one just to be sure we didn’t run into any issues. I had a couple of VMs not migration properly but those were exactly fixed. I split this up into a three day migration running ten jobs a day.

The length of time that it takes to move each VM depends on the following factors:

1. How fast your secondary network is
2. How busy the nodes are
3. How fast your disks are

Most of our VMs ranged in size from 10G to 40G in size and on average took around 10-15 minutes to complete each move. Addtionally, make sure you read the man page for hbal to see all the various features and options you can tweak. For example, you could tell hbal to just run all the commands for you which might be handy for automated rebalancing.

Conclusion

Overall the rebalancing of our cluster went without a hitch outside of a few minor issues. Ganeti made it really easy to expand our cluster with minimal to zero downtime for our hosted projects.

Along with the rest of the OSU Open Source Lab crew (including students), I was invited to the grand opening of Facebook’s new datacenter yesterday in Prineville, Oregon. We were lucky enough to get a private tour by Facebook’s Senior Open Source Manager, David Recordon. I was very impressed with the facility on many levels.

I was glad I was able to get a close look at their Open Compute servers and racks in person. They were quite impressive. One triplet rack can hold ninty 1.5U servers which can add up quickly. We’re hoping to get one or two of these racks at the OSL. I hope they fit as those triplet racks were rather tall!

Here’s a look at a bank of their web & memcached servers. You can find the memcached servers with the large banks of RAM in the front of them (72Gs in each server). The web servers were running the Intel open compute boards while the memcached servers were using AMD. The blue LED’s on the servers cost Facebook an extra $0.05 per unit compared to green LED’s.

The hot aisle is shown here and was amazing quiet. Actually, the whole room was fairly quiet which is strange compared to our datacenter. Its because of the design of the open compute servers and the fact that they are using negative/positive airflow in the whole facility to push cold/hot air.

They had a lot of generators behind the building each a size of a bus easily. You can see their substation in the background. Also note the camera in the foreground, they were everywhere not to mention security because of Green Peace.

The whole trip was amazing and was just blown away by the sheer scale. Facebook is planning on building another facility next to this one within the next year. I was really happy that all of the OSL students were able to attend the trip as well as they rarely get a chance to see something like this.

We missed seeing Mark Zuckerburg by minutes unfortunately. We had a three hour drive back and it was around 8:10PM when we left and he showed up at 8:15PM. Damnit!

If you would like to see more of the pictures I took, please check out my album below.

Thanks David for inviting us!

I just returned from PyCon 2011, the largest annual gathering of Python users and contributors.  The conference was full of energy and I came home with my head stuffed full of new ideas and Python skills.  Hillary Mason best described my feelings about PyCon in her opening keynote, “I’m glad I’m in a room where list comprehensions receive spontaneous applause”.

The Talks

The talks came in many flavors: hands-on tutorials, sessions, a poster session, and open space discussions.  Topics included dev-ops, deployment, scalability, concurrency, large scale data processing, science, and much much more.  There was a great deal to learn for both novice and experienced programmers alike.  Most sessions taught useful skills like:

• Advanced Python I & II
• Through the Side Channel: Timing and Implementation Attacks in Python
• Hidden Treasures in the Standard Library

But some sessions were just fun, mind blowing examples of what you could do with Python:

• Using Python 3 to Build a Cloud Computing Service for my Superboard II
• How to kill a patent with Python

It was difficult to choose which talks to see during most time-slots.  There were just too many great topics to choose from, so it’s fortunate that the session videos are already online.  Many thanks to the PyCon team for being so prompt.

The Jobs

It’s an exciting time for Python developers whether you are just entering the workforce, or looking for something new and exciting.  Part of the exhibit hall was dedicated to startups looking for new employees, but every other exhibitor was looking for employees, too.  There is definitely an employer out there to match your individual passions, and I’m glad to know that my students will have many great choices after graduation.

The Hallway Track

This was my first PyCon and did not know many other attendees, so I planned to spend a good deal of time meeting other people.  Among the 1380 attendees walking the halls and attending sessions were Python Core Developers, authors of your favorite libraries, keynote speakers, and even Guido.  While this could seem intimidating, we all came to PyCon to learn from each other and collaborate. Everyone was welcoming and happy to share knowledge and great conversations.

Six days of talking to random people resulted in many awesome “a ha” moments.  Topics spanned programming, technology, science, and art.  The ideas I shared in these talks were as valuable as the formal sessions I attended.  The best part was making so many new friends.  Looking forward to a great PyCon with you all next year in Santa Clara!

We’ve released Ganeti Web Manager 0.6.  Ganeti Web Manager is a Django based web application that allows administrators and clients access to their ganeti clusters. It includes a permissions and quota system that allows administrators to grant access to both clusters and virtual machines. It also includes user groups for structuring access to organizations.

This release comes after a short development cycle, with the goal of fixing critical bugs and providing important core features.  Check out the full change log, or read on for some highlights:

Virtual Machines

Ganeti Web Manager 0.6 includes multiple improvements to the virtual machine detail view.  We’ve added the complete list of virtual machine properties.  The layout has been updated to group properties into relevant sections, as well as make it more readable.

The following new controls were added for virtual machines:

• Edit a virtual machine’s settings.
• Rename a virtual machine.
• Migrate a virtual machine to it’s secondary storage.

Ganeti Web Manager 0.6 also features improvements to the virtual machine deployment process.  It now detects and recovers from Ganeti errors better than before.  If a create job fails without ganeti deploying the virtual machine, you can edit the settings and re-submit the job.  All other failures will let you continue to the virtual machine detail view where you can use the provided admin tools to repair the virtual machine.

Nodes

Nodes are now cached by Ganeti Web Manager.  This allows views using node data to be displayed faster. We now also provide Node views that allow an admin to issue commands on a node such as migration and changing the node role.  The node detail view also provides information from the perspective of a node including used resources and which virtual machines are deployed on it.

Logs

Ganeti Web Manager 0.6 now provides a log of actions performed on every object.  This will allow admins to see the history of every action taken on a VirtualMachine, Node, and Clusters.  It also shows every action a user account has performed.  The log is intended to aid auditing and troubleshooting.

Logging is provided by the newly branded Django Object Log app.  It is a reusable app that can log generic messages.  Each message can define it’s own rich format, including contextual links to the related objects.  Object Log will be developed in parallel with Ganeti Web Manager and future projects by the OSUOSL.

offlineimap is really fast now that I turned off fsync but this looks like an interesting alternative.

Nicely put together collection of console apps. I want to check out hnb.

Tutorial giving an overview of the simple and well designed mysql API.

Semi automated tool for testing web applications.