Planet OSUOSL

July 13, 2014

Piotr Banaszkiewicz

Peer instruction: update 6

I missed one post last week and this one is already delayed by two days (compared to my typical writings that appeared on Saturdays).

Fortunately, I have good excuses.

First, I wanted to get something done. Second, I was buried under real-life super exhausting stuff like post-exams leisure. ;)

Progress

Okay, good news: Pitt has new interface:

Pitt's fresh interface

Pitt’s fresh interface.

And I’m in the middle of rewriting JavaScript code (again…). Now the code uses good JavaScript patterns (and I don’t consider myself a JS newbie anymore).

I separated application logic code from application appearance code. Overall, the code should be easier to understand for newcomers and a lot easier to maintain.

The list of issues is updated and available here.

I’m also starting to think about Software Carpentry Sprint that takes place on July 22-23. I’ll post some update soon…

by Piotr Banaszkiewicz (piotr@banaszkiewicz.org) at July 13, 2014 10:00 PM

June 28, 2014

Piotr Banaszkiewicz

Peer instruction: update 5

This is gonna be a short one. Because of exams, I have to study a lot. And because there was not much progress on Pitt during this week.

Current release

Current version is v0.3.1, and it’s tagged in git repository. I’m incorporating git-flow model.

What I wanted to get in was two major features:

  1. countdown before switch from group split to broadcasting mode,
  2. small groups of students of variable-defined size.

I can tell that I was successful and these features are now part of v0.3.1 release.

Bugs

While testing with Greg and Mike at their office and me here in Kraków, I couldn’t connect to them and vice versa. Connections between Mike and Greg worked as intended.

This was strange, because I didn’t change the code so much that it’d break. More so: last week while testing with Greg alone it worked!

So I conducted few more tests:

  1. Pitt: Kraków ↔ Kraków
  2. Pitt: Kraków ↔ Toronto Mozilla Office
  3. Pitt older version: Kraków ↔ Toronto Mozilla Office
  4. Other WebRTC application: Kraków ↔ Toronto Mozilla Office

In the previous week, I’m guessing Greg worked from his home. We wanted to try Pitt in this configuration (Kraków ↔ Greg’s home), but due to holidays in Canada we were not able to.

From what I can tell, the issue lies in the client IP addresses resolution. Here’s a lengthy article about how hard it is to do WebRTC properly: http://www.html5rocks.com/en/tutorials/webrtc/basics/#toc-signaling

Anyway, I hopefully will be able to easily overcome this issue.

by Piotr Banaszkiewicz (piotr@banaszkiewicz.org) at June 28, 2014 10:00 PM

June 21, 2014

Piotr Banaszkiewicz

Peer instruction: update 4

This week me and Greg Wilson, my mentor, we decided to test run Pitt. It was a success and, while revealing some annoying bugs, it made Greg really excited.

Current release

Current version is v0.3, altough I might have tagged it wrong in git. Fortunately it seems that tag was not published to the GitHub repository.

Current features

This is a quick recap on what Pitt is capable of right now.

  1. Users coming in and quitting: thanks to signalling, as soon as someone appears online, they’re being logged in and everyone can see them.
  2. Quick switch mode: the instructor can easily and very quickly switch modes from broadcasting (to all students) to work in small groups (students are split into pairs or groups of 3).

The code is decent. I presume there are some bugs with saved state of the application. For example, what to do if the instructor is broadcasting and someone joins in? What if another instructor joins in? What if there are two instructors and one starts broadcasting, should other instructors see their stream too?

These kind of questions will be addressed for the v0.4 release. Right now I’m working to get v0.3.1 release by Friday, 27.06.2014.

Exposed bugs

During my test run with Greg we quickly found out one super annoying bug.

There’s a very interesting human behavior: people slow down their talk when they hear themselves with small delay. It’s really hard to overcome this basic reflex.

Back to the bug: we found out that local stream video (ie. the one where you can see yourself, because browser plays the video from your webcam) is not muted by default. Add the playback delay to the mix and as a result Greg was struggling to speak.

This bug was really easy to fix (in HTML5: <video src=”…” muted=”muted”>) and the fix is included in the recent code.

There was also issue with the state of the application - it somehow broke for Greg (when he was a student). I have yet to discover the actual bug.

Missing features

Greg wants me to add these two new features to the upcoming release:

  1. Countdown. The instructor presses “back to broadcast mode” button and the switch doesn’t happen immediately. First there’s 30 seconds countdown so that no students are interrupted in the middle of the sentence.
  2. Variable group size. The instructor can specify if the students are split into pairs or groups of 3, or groups of 6. This task is more challenging than it seems: I have to ensure that no students are left alone (what is often the case when there’s odd number of students).

Next release

I’m planning to release next version, v0.3.1, on Friday 27.06.2014 or earlier.

I have 3 exams before Friday and 2 after it. Ouch…

by Piotr Banaszkiewicz (piotr@banaszkiewicz.org) at June 21, 2014 10:00 PM

June 20, 2014

Ben Kero

Mozilla’s “try” repository

At Mozilla we use Mercurial for Firefox development. We have several repositories/trees that are used depending on where the code should be. If a developer wishes to test the code they have been developing, they can submit it to a Mercurial repository called ‘try‘, since running our entire test suite is not feasible on developer machines.

We have quite a bit of infrastructure around this including Tinderbox Pushlog (TBPL) and  more. This post deals with the infrastructure and problem we face while trying to scale the ‘try’ repository.

A few statistics:

  • The try repository currently has 17943 heads. These heads are never removed.
  • The try repository is about 3.6 GB in size.
  • Due to Mercurial’s on-wire HTTP protocol, this number of heads causes HTTP cloning to fail
  • There are roughly 81000 HTTP requests to try per day
  • To fix problems (mentioned below), the try repository is deleted and re-cloned from mozilla-central every few months

There are a number of problems associated with such a repository. One particularly nasty one has been present through several years of Mercurial development, and has been tricky in that it is seemingly unreproducible. The scenario is something like:

  • User ‘hg push’es some changes to a new head onto try
  • The push process takes a long time (sometimes between 10 minutes and hours)
  • A developer could issue an interrupt signal (ctrl+C) which causes the client to gracefully hang up and exit (his typically has no effect on the server
  • Subsequent pushes will hang with something similar to ‘remote: waiting for lock on repository /repo/hg/mozilla/try/ held by ‘hgssh1.dmz.scl3.mozilla.com:23974′
  • When this happens a hg process is running on the server has the following characteristics:
    • A ‘hg serve’ process runs single-threaded using 100% CPU
    • strace-ing and ltrace-ing reveal that the process is not making any system calls or external library calls
    • perf reveals that the process is spending all of its time inside some ambiguous python function
    • pdb yields that the process is spending all of its time in a function that (along some point in the stack trace) is going through ancestor calculations
    • The process will eventually exit cleanly
  • As operators there is nothing we can do that to alleviate the situation once the repository gets in this state. We simply inform developers and monitor the situation.

There have been several ideas on ways to alleviate the problem:

  • Periodically reset ‘try’. This is considered bad because 1) it loses history, and 2) it is disruptive to developers, who might have to re-submit try jobs again
  • Reset try on the SSH servers, but keep old try repositories on the HTTP servers. This has the potential to create unforeseen problems of growing these repositories even further on the HTTP servers. If reset (staggered from SSH server resets) this will remove unforeseen problem potential, but still lose history.
  • Creating bundle files out of pushes to ‘try’, then hosting these in an accessible location (S3, http webroot, etc). I will detail this method in a future blog post.

As of now though, try will periodically need to be reset as a countermeasure to the hangs mentioned in this post. Getting a reproducible test case might allow us to track down a bug or inefficiency in Mercurial to fix this problem after all. If you’d like to help us with this, please ping fubar or me (bkero) on irc.mozilla.org.

by bkero at June 20, 2014 10:22 PM

June 13, 2014

Piotr Banaszkiewicz

Peer instruction: update 3

The decision has been made, and I switched to a different codebase.

I had one exam this week (Aparatura Automatyzacji, eng. Automation Systems) and five more in next 2-3 weeks, so it is quite challenging for me to find time for peer instruction right now.

I did, however, spend most of this weekend trying to make Pitt broadcasting mode for instructors.

It works! Proof below:

Me as instructor and two students.

Having dual monitor setup helps getting this kind of screenshots.

As you can see, the layout and design in general is very rough. And so is the code.

This code is mostly based upon multiple events being propagated through sockets.

But I really don’t like the design of these events, and, frankly speaking, I much more prefer a “pub/sub” (publication - subscription) architecture.

At this moment, when the instructor triggers an event, the server has to repropagate it to every student.

In “pub/sub” arch, however, there’s no repropagation in between.

Some people use redis for pub/sub, but I found some really cool protocol: WAMP (don’t confuse it with Windows, Apache, MySQL, PHP stack!).

WAMP authors clearly explain why it is good so please go there and read. Let me just point out a few cool features of WAMP:

  • WAMP is not a NodeJS module, it’s a protocol
  • it can be (and indeed is!) implemented in various languages
  • therefore it’s a solid foundation for a quickly changing service.

What I dislike about WAMP is that at the moment there’s only one leading implementation: Autobahn. And I’m not yet sure if I want to drop NodeJS backend in the future, but if so, then there’s already fast Autobahn|Python.

by Piotr Banaszkiewicz (piotr@banaszkiewicz.org) at June 13, 2014 10:00 PM

June 10, 2014

Lars Lohn

Crontabber and Postgres


This essay is about Postgres and Crontabber, the we-need-something-more-robust-than-cron job runner that Mozilla uses in Socorro, the crash reporting system.

Sloppy database programming in an environment where autocommit is turned off leads to very sad DBAs. There are a lot of programmers out there that cut their teeth in databases that either had autocommit on by default or didn't even implement transactions.  Programmers that are used to working with relational databases in autocommit mode actually miss out on one of the most powerful features of relational databases. However, bringing the cavalier attitude of autocommit into a transactional world will lead to pain.  

In autocommit mode, every statement given to the database is committed as soon as it is done. That isn't always the best way to interact with a database, especially if there are multiple steps to a database related task.

For example say we've got database tables representing monetary accounts. To move money from one account to another requires two steps, deduct from the first account and add to the other. If using autocommit mode, there is a danger that the accounts could get out of sync if some disaster happens between the two steps.

To counter that, transactions allow the two steps to be linked together. If something goes wrong during the two steps, we can rollback any changes and not let the accounts get out of sync. However, having manual transaction requires the programmer to be more careful and make sure that there is no execution path out of the database code that doesn't pass through either a commit or rollback. Failing to do so may end up leaving connections idle in transactions. The risk is critical consumption of resources and impending deadlocks.

Crontabber provides a feature to help make sure that database transactions get closed properly and still allow the programmer to be lazy.

When writing a Crontabber application that accesses a database, there are a number of helpers. Let's jump directly to the one that guarantees proper transactional behavior.

# sets up postgres
@using_postgres()
# tells crontabber control transactions
@as_single_postgres_transaction()
def run(self, connection):
    # connection is a standard psycopg2 connection instance.
    # use it to do the two steps:
    cursor = connection.cursor()
    cursor.execute(
        'update accounts set total = total - 10' where acc_num = '1'
    )
    do_something_dangerous_that_could_cause_an_exception()
    cursor.execute(
        'update accounts set total = total +10' where acc_num = '2'
    )

In this contrived example, the method decorator gave the crontabber job the a connection to the database and will ensure that that if the job runs to completion, the transaction will be commited. It also guarantees that if the the 'run' method exits abnormally (an exception), the transaction will be rolled back.

Using this class decorator is declaring that this Crontabber job represents a single database transaction.  Needless to say, if the job takes twenty minutes to run, you may not want it to be a single transaction.  

Say you have a collection of periodic database related scripts that have evolved over years by Python programmers long gone. Some of the old crustier ones from the murky past are really bad about leaving database connections “idle in transaction”. In porting it to crontabber, call that ill behaved function from within the context of a construct like that previous example. Crontabber will take on the responsibility of transactions for that function with these simple rules:

  • If the method ends normally, crontabber will issue the commit on the connection.
  • If an exception escapes from the scope of the function, crontabber will rollback the database connection.

Crontabber provides three dedicated class decorators to assist in handling periodic Postgres tasks. Their documentation can be found here: Read The Docs: Postgres Decorators.  The @with_postgres_connection_as_argument decorator will pass the connection the run method, but does not handle commit and/or rollback.  Use that decorator if you'd like to manage transactions manually within the Crontabber job. 

Transactional behavior contributes in making Crontabber robust.  Crontabber is also robust because of self healing behaviors. If a given job fails, dependent jobs will not be run. The next time the periodic job's time to execute comes around, the 'backfill' mechanism will make sure that it makes up for the previous failure. See Read The Docs: Backfillfor more details.

The transactional system can also contribute to self healing by retrying failed transactions, if those failures were caused by transient issues. Temporary network glitches can cause failure. If your periodic job runs only once every 24 hours, maybe you'd rather your app retry a few times before giving up and waiting for the next scheduled run time.

Through configuration, the transactional behavior of Postrges, embodied by Crontabber's TransactionExecutor class, can do a “backing off retry”. Here's the log of an example of backoff retry, my commentary is in green:

# we cannot seem to connect to Postgres
2014-06-08 03:23:53,101 CRITICAL - MainThread - ... transaction error eligible for retry
OperationalError: ERROR: pgbouncer cannot connect to server
# the TransactorExector backs off, retrying in 10 seconds
2014-06-08 03:23:53,102 DEBUG - MainThread - retry in 10 seconds
2014-06-08 03:23:53,102 DEBUG - MainThread - waiting for retry ...: 0sec of 10sec
# it fails again, this time scheduling a retry in 30 seconds;
2014-06-08 03:24:03,159 CRITICAL - MainThread - ... transaction error eligible for retry
OperationalError: ERROR: pgbouncer cannot connect to server
2014-06-08 03:24:03,160 DEBUG - MainThread - retry in 30 seconds
2014-06-08 03:24:03,160 DEBUG - MainThread - waiting for retry ...: 0sec of 30sec
2014-06-08 03:24:13,211 DEBUG - MainThread - waiting for retry ...: 10sec of 30sec
2014-06-08 03:24:23,262 DEBUG - MainThread - waiting for retry ...: 20sec of 30sec
# it fails a third time, now opting to wait for a minute before retrying
2014-06-08 03:24:33,319 CRITICAL - MainThread - ... transaction error eligible for retry
2014-06-08 03:24:33,320 DEBUG - MainThread - retry in 60 seconds
2014-06-08 03:24:33,320 DEBUG - MainThread - waiting for retry ...: 0sec of 60sec
...
2014-06-08 03:25:23,576 DEBUG - MainThread - waiting for retry ...: 50sec of 60sec
2014-06-08 03:25:33,633 CRITICAL - MainThread - ... transaction error eligible for retry
2014-06-08 03:25:33,634 DEBUG - MainThread - retry in 120 seconds
2014-06-08 03:25:33,634 DEBUG - MainThread - waiting for retry ...: 0sec of 120sec
...
2014-06-08 03:27:24,205 DEBUG - MainThread - waiting for retry ...: 110sec of 120sec
# finally it works and the app goes on its way
2014-06-08 03:27:34,989 INFO  - Thread-2 - starting job: 065ade70-d84e-4e5e-9c65-0e9ec2140606
2014-06-08 03:27:35,009 INFO  - Thread-5 - starting job: 800f6100-c097-440d-b9d9-802842140606
2014-06-08 03:27:35,035 INFO  - Thread-1 - starting job: a91870cf-4d66-4a24-a5c2-02d7b2140606
2014-06-08 03:27:35,045 INFO  - Thread-9 - starting job: a9bfe628-9f2e-4d95-8745-887b42140606
2014-06-08 03:27:35,050 INFO  - Thread-7 - starting job: 07c55898-9c64-421f-b1b3-c18b32140606
The TransactionExecutor can be set to retry as many times as you'd like with retries at whatever interval is desired.  The default is to try only once.  If you'd like the backing off retry behavior, change TransactionExecutor in the Crontabber config file to TransactionExecutorWithLimitedBackOff or TransactionExecutorWithInifiteBackOff

While Crontabber supports Postgres by default, Socorro, the Mozilla Crash Reporter, extends the support of the TransactionExecutor to HBase, RabbitMQ, and Ceph.  It would not be hard to get it to work for MySQL or,  really, any connection based resource.

The TransactionExecutor, Coupled with Crontabber's Backfilling capabilities, nobody has to get out of bed at 3am because the crons have failed again.  They can take care of themselves.

On Tuesday, June 10, Peter Bengtsson of Mozilla will give a presentation about Crontabber to the SFPUG.  The presentation will be broadcast on AirMozilla.

SFPUG June: Crontabber manages ALL the tasks








by K Lars Lohn (noreply@blogger.com) at June 10, 2014 08:51 PM

May 26, 2014

Pranjal Mittal

Setting up Rsync in daemon mode on an AWS EC2 instance

I was trying to exlore and understand rsync in detail for a very cool project that I am planning to work on. The project is related to FTP Mirror Syncing about which I will write in detail next time. Rsync is a great tool for efficient syncing of directories. It transfers only the differences in files saving time and bandwidth. In this succint post I will quickly walk through the steps I perfomed to be able to setup rsync between 2 Amazon EC2 instances. I will particularly focus on using rsync in daemon mode as opposed to using rsync over ssh which you could explore easily without any problems.

Key to the steps described ahead:

(1) To edit default config file used by rsync daemon
(2) To start rsync daemon
(3) To kill rsync daemon
(4) Command to sync (push) contents in current directory to the server which is runnign the rsync daemon.
(5) To create several demo files for testing rsync


Steps performed in detail:

(Refer to corresponding key number)

(1) sudo nano /etc/rsyncd.conf


rsyncd.conf (contents)


lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
port = 873

# Defines an rsync module.
[modulepranjal]
    path = <absolute_path_this_module_maps_to>
    comment = The syncthis directory shall be synced.
    uid = ubuntu
    gid = ubuntu
    read only = no
    list = yes
    hosts allow = 0.0.0.0/0
    auth users = *
    secrets file = /etc/rsyncd.secrets

# Can define more modules if you want that map to a different path.
...


rsyncd.secrets (contents)


rsync_client_user: keepanypassword


Note: Make sure you change access permissions of your rsyncd.secrets file to 600 if you want your rsync daemon to actually accept your secrets file.

    $ sudo chmod 600 /etc/rsyncd.conf

(2) sudo rsync --daemon

Caveat: Make sure connections to port 873 are allowed on your instance. I spent about 5-6 days trying to figure out why my rsync daemon is not working correctly when I try to rsync to it from some other instance and later figured out that AWS firewall had blocked all conections to port 873 since there was no rule allowing access to port 873.





(3) sudo kill `cat /var/run/rsyncd.pid`

(4) rsync -rv . ubuntu@10.252.164.249::modulepranjal/

Run this command on any other instance (without an rsync daemon) to push all contents in the current directory to the rsync module path on the instance running the rsync daemon.
 -r stands for recursively transferring all the contents of the directory.

Note: Double colon (::) means that rsync protocol will be used rather than ssh. If only a single colon (:) is provided then rsync tries to sync over ssh.


(5) for i in `seq 1 100`; do touch testfile$i; done

This simple bash command will generate 100 testfiles like testfile1, testfile2, etc.. which will be useful in case you wish to explore how a sync with several files involved looks like.

Quick Tip:

Syncing using rsync in daemon mode is much faster than using rsync in ssh mode. The daemon mode turns pretty useful for syncing public content where privacy is not much of a concern. The ssh mode takes more time as some time is spending on encrypting / decrypting the rsync transfer data.

by Pranjal Mittal (noreply@blogger.com) at May 26, 2014 04:51 PM

May 11, 2014

Pranjal Mittal

Google Summer of Code 2014 Onboarding - Mirror Syncing with OSL


I was accepted for Google Summer of Code 2014 and am excited to be working yet again with the OSU Open Source Lab on a project which on completion should have a big impact on the performance and ease of use of mirror syncing systems. The OSL helps serve thousands of terabytes of open source data to the world every year. OSL is responsible for hosting several communities like Debian, Fedora which rely on mirrors for serving package data. Serving package data to millions of users across the globe cannot be possible without a scalable FTP architecture; which calls for mirroring. Further even mirroring should be done effectively and should be easy to manage. The focus of my project is to write specifications for improving performance and convenience (in using) a mirror syncing system and also to implement those specifications. The project though primarily intended to be developed keeping the needs of OSL in mind, is expected to be useful to several other organizations or companies intending to set up a mirroring service to host several projects with a big amount of data involved.


A reference mirror syncing architecture


There are 3 main components of the reference mirror syncing architecture:
  • Upstream Source
  • Master
  • Slave or FTP Host
The reference architecture I will discuss ahead is a 3 layer architecture with the first layer belonging to the main community providing the package (Debian, Fedora) and the other 2 layers belong to the mirror service provider (OSL). A rough example of the reference architecture in use at OSL.






Beginners Questions and FAQ’s


Q) Why not just have a single upstream server to serve all the packages?
A) A single low-end server cannot handle several connections in parallel. An FTP connection is computationally more intensive than simple HTTP connections that are stateless and last for a short time. Thus a way is needed to delegate the serving of data, which calls for multiple layers.

Q) Why don't the upstream providers just directly delegate serving to FTP hosts managed by them, rather than bringing in a 3rd party mirroring service.
A) Several organizations focus on just building packages and do not want to worry so much about making the packages reach end users (even though they want them to be easily available to all).


Q) Why does the mirroring service provider in the reference architecture use a Master?
A) Introducing additional layers adds to delays, as data has to be synced across the whole tree. Then why do we introduce a master layer? The answer is that it makes things more convenient to manage. The master is responsible for syncing data from the upstream source in a well defined manner and then instructs each of the FTP host to sync from it. Further it is easier for upstream sources to grant access to a single node via Access Control Lists. The master helps act as a control point which is useful to instruct all the FTP hosts at once.


Q) Do syncs from upstream need to be periodically performed?
A) Yes, of course. If you are someone who is insistent about having the latest available content at hand, then you will understand that it is necessary to perform sync's from the upstream source periodically. Typically major upstream sources provide ideal sync times and sync frequency.


Q) Why is rsync algorithm a good option for mirroring?
A) rsync uses delta transfer algorithm to increase transfer speeds, having a huge impact on speed where the directories being synced are more or less the same. (small delta's). After every build the package data more or less remains the same with small changes. It would be silly to transfer the whole package over again deleting the package at the receiving end. Its much faster to just trasfer the differences (called delta's) which updates the package data at the destination (called base)

Some Q&A that you should be aware of before reading ahead


Q) How do the FTP hosts (slaves) sync from the master in the reference architecture.
A) The master runs an rsync daemon to allow slaves having rsync clients to sync from it. So basically both upstream sources and master of mirroring service have rsync daemon running on then. In some cases rsync over ssh is used, but not preferred since rsync with ssh encryption is slower and encryption is not a necessity when you want to serve public content. There might be more reasons to it. (maybe)

Q) What are the mirroring parameters associated with an upstream source project?
A) source, destination, rsync_server_password, schedule_details.
schedule_details specifies the interval at which the mirroring service should sync from the upstream source in order to stay updated. Standard times for syncing each day are defined by the upstream sources in many cases.

Problems encountered while setting up a mirroring service:


Performance: High input/output load between upstream and ftp hosts (solved if we use an intermediate high end master server). The reference architecture takes care of this problem

Convenience: For every upstream project added there are lot of parameters like source, destination, rsync_server_password, schedule_details Most of these parameters are defined inside a bash file for that project (one for each project) that performs an rsync internally to fetch data from the upstream source. However the schedule_details are specified separately in cron files. Editing files located at 2 different places is not convenient and further having a new bash file for each project makes things even more messy. Something more simple is needed where all these details can be specified at once.

Performance / Convenience: Setting up an rsync daemon on every ftp host so that the master may be able to push to it, is not a good idea as it again makes things messy with too much configuration involved. Configuration should be only defined at master and easily propagated to slaves. The architecture deployed at OSL close to the reference architecture already understood this issue and made use of a trigger mechanism to tell FTP hosts that they need to update. In this method the slaves did a periodic poling to check if new content was available at the master. This method not only wastes computational resources but it is difficult to manage too, as settings have to be done on each ftp host.

Errors: Collision of rsync calls. Cases where the next scheduled rsync collides with the previously ongoing rsync. This happens rarely because the schedule interval is generally large and sufficient for an rsync call to complete. However if there is a huge delta to be transferred rsync process can take time and collide with the next rsync. Care should be taken to avoid such collisions.

Errors: The slaves sync from master and the master syncs from upstream. Now what happens when the slaves start syncing from master when the master is in process of syncing from upstream. There is a mess! This is again not very common to observe, but causes small errors which needs to be avoided in a strong architecture.


Solving the above problems

Note: Here I'll describe my approach so far at solving these problems after discussion with my mentors. Of course, there can be better ways and I would love to hear about them from you. Feel free to email me at: <lastname>.<firstname>@gmail.com Till now I have primarily worked on solving the problem of mirroring between uptream--&--master by writing code. I am also working on a specifications for syncing between master--&--slaves.

Problem of too many bash/config files

Whoa! Lets just make a single API to add references to all upstream sources along with associated parameters. So instead of adding a new bash file for each project and defining scheduling parameters in cron, all the parameters and references can just be specified at a single location.

The api is implemented in the form of a REST API which takes input via post requests. For. Example to add a new upstream source project the mirroring service administrator makes a POST call to /addproject/ along with additional POST parameters which define the details of the upstream source. See example.

In reality the user need not issue a POST call since REST API's can easily be wrapped over to make cli tools and web applications to make the interactivity even easier.

Technologies used: Flask, APScheduler, subprocess, rsync

Avoiding the installation of rsync daemon on slaves (or FTP Hosts)

Note: I haven't worked on any code for master-slave syncing so this is just theoretical at the moment.

To sync the content from the master to the slaves on the mirroring service one option was to install rsync daemon over each of the slaves and then push content to them from the master node but @ramereth, didn't seem to like the idea of having to run rsync daemons on all the slaves to solve the problem and I am not sure if I remember all the reasons he suggested so but I'll mention one.. Having to run so many rsync daemons (on all slaves) calls for several rsyncd.conf files and it becomes difficult to control the configuration on all the slaves sitting at the master. (without using configuration management tools).


The idea is to have a light app daemon running on all the slaves that acts as an entry point for controlling the FTP hosts (slaves). They do not even need to run an rsync daemon this way, just need to have the rsync client, since the master can simply ask them to initiate a sync when suitable.


The image below shows an image of an architecture with improvements, showing components involved. The architecture specifications are described in more detail at: http://bit.ly/1mKKGW8







Architecture Details: http://bit.ly/1mKKGW8


Let's talk code

Here I'll discuss the code I have written so far as a proof of concept to the new architecture specifications.


The experimental web api daemon is being written in Flask which is an excellent micro framework based in Python. All of the Flask specific code currently resides in a single file ie. main.py. The main purpose of using flask is to provide an api to map HTTP calls to python functions.

APScheduler is a robust and efficient python based scheduler that allows scheduling with the precision of seconds (instead of minutes in cron). It also provides advantages like using a job store of choice which stores the scheduled tasks and their parameters on disk and uses them when needed. (Link: docs)

Subprocess is used to write useful wrappers over rsync to make it easily callable from python code.

1. Syncing upstream projects to master node

1.1 Main api code and function responsible for calling rsync internally (File: main.py)

def sync_project_from_upstream(project, host, source, dest, password)

project: Name of the unix user of the upstream project server
host: IP or hostname of the upstream rsync provider
source: Path on source that tells the location of rsync module.
dest: Path on the master node where the upstream project is to be synced to.
password: rsync password of the rsync daemon running on the source.

This function is instantiated as a lambda function (first class object) using the lambda keyword so that it may be conveniently passed to the scheduler instance. See code in main.py file as reference.

job_function = lambda: sync_project_from_upstream(project,
project_obj["host"],
project_obj["source"],
project_obj["dest"],
project_obj["rsync_password"])

1.2 Calling rsync internally. File: sync_utilities.py (code)

subprocess is used to create a wrapper over the rsync bash client.

# This is how rsync can be called via subprocess as used in the code.
# Using shell=True is not advisable, and I will be changing this in future.

cmd = 'rsync -avH --delete --progress ' + full_source + ' ' + dest
rsync_process = subprocess.Popen(cmd,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,)


I'll am still working on completing this section. So please continue to poll this blog post as and when convenient for you.


P.S:
I am working on this project, under the guidance of my mentor @ramereth (Lance Albertson, Director, OSL) whom I mentioned about above briefly. I am very thankful to him for his patience and answering all my questions which was crucial to the understanding of the current FTP mirror syncing architecture deployed at OSL. Its great to discuss ideas and solutions with members of the OSL community on irc and they are very helpful. I am quite happy to be working on this project which is challenging yet very interesting for me at the same time. Looking forward to an excellent summer, coding (with OSL)!

by Pranjal Mittal (noreply@blogger.com) at May 11, 2014 02:28 PM

May 05, 2014

Lars Lohn

Crouching Argparse, Hidden Configman

I've discovered that people that persist in being programmers over age fifty do not die.  Wrapped in blankets woven from their own abstractions, they're immune to the forces of the outside world. This is the first posting in a series about a pet hacking project of mine so deep in abstractions that not even light can escape.

I've written about Configman several times over the last couple of years as it applies to the Mozilla Socorro Crash Stats project.  It is unified configuration.  Configman strives to wrap all the different ways that configuration information can be injected into a program.  In doing so, it handily passes the event threshold and becomes a configuration manager, a dependency injection framework, a dessert topping and a floor wax.

In my experimental branch of Configman, I've finally added support for argparse.  That's the canonical Python module for parsing the command line into key/value pairs, presumably as configuration.  It includes its own data definition language in the form of calls to a function called add_argument.  Through this method, you define what information you'll accept from the command line.

argparse only deals with command lines.  It won't help you with environment variables, ini files, json files, etc.  There are other libraries that handle those things.  Unfortunately, they don't integrate at all with argparse and may include their own data definition system or none at all.

Integrating Configman with argparse was tough.  argparse doesn't play well in extending it in the manner that I want.  Configman employs argparse but resorts to deception to get the work done.  Take a look at this classic first example from the argparse documentation.

from configman import ArgumentParser

parser = ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
                    help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
                    const=sum, default=max,
                    help='sum the integers (default: find the max)')

args = parser.parse_args()
print(args.accumulate(args.integers))

Instead of importing argparse from its own module, I import it from Configman.  That just means that we're going to use my subclass of the argparse parser class.  Otherwise it looks, acts and tastes just like argparse: I don't emulate it or try to reimplement anything that it does, I use it to do what it does best.  Only at the command line, running the 'help' option, is the inner Configman revealed.

$ ./x1.py 0 0 --help
    usage: x1.py [-h] [--sum] [--admin.print_conf ADMIN.PRINT_CONF]
                 [--admin.dump_conf ADMIN.DUMP_CONF] [--admin.strict]
                 [--admin.conf ADMIN.CONF]
                 N [N ...]

    Process some integers.

    positional arguments:
      N                     an integer for the accumulator

    optional arguments:
      -h, --help            show this help message and exit
      --sum                 sum the integers (default: find the max)
      --admin.print_conf ADMIN.PRINT_CONF
                            write current config to stdout (json, py, ini, conf)
      --admin.dump_conf ADMIN.DUMP_CONF
                            a file system pathname for new config file (types:
                            json, py, ini, conf)
      --admin.strict        mismatched options generate exceptions rather than
                            just warnings
      --admin.conf ADMIN.CONF
                            the pathname of the config file (path/filename)

There's a bunch of options with "admin" in them.  Suddenly, argparse supports all the different configuration libraries that Configman understands: that brings a rainbow of configuration files to the argparse world.  While this little toy program hardly needs them, wouldn't it be nice to have a complete system of "ini" or "json" files with no more work than your original argparse argument definitions? 

using argparse through Configman means getting ConfigObj for free

Let's make our example write out its own ini file:

    $ ./x1.py --admin.dump_conf=x1.ini
    $ cat x1.ini
    # sum the integers (default: find the max)
    #accumulate=max
    # an integer for the accumulator
    #integers=
Then we'll edit that file and make it automatically use the sum function instead of the max function.  Uncomment the "accumulate" line and replace the "max" with "sum".  Configman will associate an ini file with the same base name as a program file to trigger automatic loading.  From that point on, invoking the program means loading the ini file.  That means the command line arguments aren't necessary.  Rather not have a secret automatically loaded config file? Give it a different name.

    $ ./x1.py 1 2 3
    6
    $ ./x1.py 4 5 6
    15
I can even make the integer arguments get loaded from the ini file.  Revert the "sum" line change and instead change the "integers" line to be a list of numbers of your own choice.

    $ cat x1.ini
    # sum the integers (default: find the max)
    #accumulate=max
    # an integer for the accumulator
    integers=1 2 3 4 5 6
    $ ./x1.py
    6
    $ ./x1.py --sum
    21

By the way, making argparse not have a complete conniption fit over the missing command line arguments was quite the engineering effort.  I didn't change it, I fooled it into thinking that the command line arguments are there.


Ini files are supported in Configman by ConfigObj.  Want json files instead of ini files?  Configman figures out what you want by the file extension and searches for an appropriate handler.  Specify that you want a "py" file and Configman will write a Python module of values.  Maybe I'll write an XML reader/writer next time I'm depressed.

Configman does environment variables, too:
    $ export accumulate=sum
    $ ./x1.py 1 2 3
    6
    $ ./x1.py 1 2 3 4 5 6
    21

There is a hierarchy to all this.  Think of it as layers: at the bottom you have the defaults expressed or implied by the arguments defined for argparse.  Then next layer up is the environment.  Anything that appears in the environment will override the defaults.  The next layer up is the config file.  Values found there will override both the defaults and the environment.  Finally, the arguments supplied on the command line override everything else.

This hierarchy is configurable, you can make it any order that you want.  In fact, you can put anything that conforms to the collections.Mapping api into that hierarchy.  However, for this example, as a drop-in augmentation of argparse, the api to adjust the "values source list" is not exposed.

In the next installment, I'll show a more interesting example where I play around with the type in the definition of the argparse arguments.  By putting a function there that will dynamically load a class, we suddenly have a poor man's dependency injection framework.  That idea is used extensively in Mozilla's Socorro to allow us to switch out storage schemes on the fly.

If you want to play around with this, you can pip install Configman.  However, what I've talked about here today with argparse is not part  of the current release.  You can get this version of configman from my github repo: Configman pretends to be argparse - source from github.  Remember, this branch is not production code.  It is an interesting exercise in wrapping myself in yet another layer of abstraction. 

My somewhat outdated previous postings on this topic begin with Configuration, Part 1

by K Lars Lohn (noreply@blogger.com) at May 05, 2014 03:17 PM

May 02, 2014

Beaver BarCamp

A Succesful Beaver BarCamp 14

/theme/img/jdugger.jpg

Over 115 students, educators and community members joined together on Saturday, April 12, in the Kelley Engineering Center at Oregon State University in order to attend the Open Source Lab’s Beaver BarCamp 14.

Continuing on with the spirit of previous BarCamps, Beaver BarCamp 14 had a diverse group of tech based and non-tech based sessions; this year topics ranged from Heartbleed to Vagrant to How to Podcast to Magic the Gathering (with free sample decks).

“The secret I’ve found to getting people to show up to your talks is having lots of props to get their attention,” advises Evan Marshall, who hosted a session on helicopters. Marshall followed his own advice and brought a helicopter flight simulator to accompany his session.

Everyone is welcome to present at barcamp, regardless of their experience level. This open format provides the opportunity to hear from a wide variety of speakers from many different backgrounds and interest areas.

First time presenter Daniel Reichert ran a session on Theoretical Cryptography. “I wanted to get more experience speaking in public,” Reichert states. This was a consensus he shared with many of the other presenters, including Gregory Brewster, who ran a session on Google Glass.

/theme/img/group.jpg

“I decided to come with one of my friends that came last year, and thought I could get experience presenting to people,” Brewster says. “I happen to know about Google Glass, and thought that it could be interesting.”

The interactive element is always strong at the unconference, and Beaver BarCamp 14 was no exception. Those who attended the Google Glass session where given the opportunity to try it out. Some played games, some took photos and some simply explored the different features.

“Google Glass Tennis was absolutely exhilarating,” reports barcamp newcomer Maren Vick. “Your head is the racket and it’s so lifelike. Be careful, though, you need to focus on where you’re walking as well.”

Rackspace sponsored Beaver BarCamp 14, and attendees were able to enjoy a full, free Beaver BarCamp experience along with food, refreshments and t-shirts.

/theme/img/board.jpg

System Message: WARNING/2 (/var/www/beaverbarcamp.org/git/content/beaverbarcamp14.rst, line 40)

Explicit markup ends without a blank line; unexpected unindent.

In the past, Beaver BarCamp has been a biannual event, however the Open Source Lab has decided to switch to an annual format going forward.

“This year we decided that it was best to switch to a once a year format and focus on making Beaver BarCamp better," says Lance Albertson, director of the Open Source Lab. “We also look forward to developing new education programs such as a Hackathon focused on DevOps and FOSS, getting students interested in it early on in their school year. This would also enable us to kickstart DevOps Bootcamp.”

This means that Beaver BarCamp 15 will take place in April 2015. Any updates or details will be posted to the Beaver BarCamp website, so stay tuned!

by OSU Open Source Lab at May 02, 2014 07:36 PM

April 14, 2014

Lars Lohn

Pegboard Tool Storage - circuit tester & grounding adapter

(this post is part 8 of a longer series.  See  

These are the most commonly lost tools that I own.  In fact, I finally found the left grounding adapter on the floor behind the shelf in the garage, no where near an outlet.  That's the problem: make these tools easy to find.


The solution is to make fake electrical outlets to put on the pegboard, of course.  Now all have to do is remember to put them away when I'm done with them.  If I can manage to do that, finding them again will be trivial because they'll be in plain sight.


This simple design is available for download at: pegboard outlet storage

by K Lars Lohn (noreply@blogger.com) at April 14, 2014 03:34 AM

Pegboard Tool Storage - the goat hook

(this post is part 6 of a longer series.  See  

 And why shouldn't pegboard storage systems be whimsical?   This design for a screwdriver holder came as a sudden inspiration while browsing thingiverse.com. I saw the goat head sculpture and decided to scale it down and mix it up with my pegboard blank:



Adding the funnel shaped hole through the top of the head and emerging from the mouth makes for a silly and slightly disturbing tool holder.






This design is available for the 3D printing at: http://www.thingiverse.com/thing:293801

Next in this series, the ever exciting Hex Bits

by K Lars Lohn (noreply@blogger.com) at April 14, 2014 03:32 AM

March 26, 2014

Beaver BarCamp

Upcoming changes to Beaver Barcamp

The Open Source Lab loves Beaver Barcamp, and we know the community does too. The event plays an important role in fostering relationships between the tech and academic communities, something the OSL wants to continue doing. However, we see a need for more hands-on, workshop events. Given the tight academic calendar, as well as the amount of organizing this event entails, the OSL has chosen to substitute the fall Beaver BarCamp with a DevOps or Free and Open Source Software (FOSS) Hackathon event.

“This year we decided that it was best to switch to a once a year format and focus on making Beaver BarCamp better," says Lance Albertson, director of the Open Source Lab. “We also look forward to developing new education programs such as a Hackathon focused on DevOps and FOSS, getting students interested in it early on in their school year. This would also enable us to kickstart DevOps Bootcamp.”

The April 2014 Beaver BarCamp will continue as planned. The lab has decided to call this Beaver BarCamp 14, aligning the number with the year. (And yes, skipping unlucky number 13!)

At Beaver Barcamp 14, the Open Source Lab will offer a feedback session. We hope you’ll join us in discussing how the lab can continue to support the open source and academic communities at Oregon State.

Stay tuned for exciting details about our future events!

http://facebook.com/beaverbarcamp

by OSU Open Source Lab at March 26, 2014 10:15 PM

March 24, 2014

Ben Kero

Measuring the performance improvement of Mercurial (NFS vs local disk)

Earlier this year I was able to move mozilla.org’s Mercurial infrastructure from using NFS-mounted storage to transitioning to local disk. This was done for several reasons.

  1. The Mercurial developers were concerned about race conditions and concurrent write/reads causing service inconsistency between hosts. This became evident when stale file handles started appearing in our apache logs.
  2. An extension we wrote (pushlog) was also being served off of NFS. This is a problem not because we have multiple hosts writing at once, but because the file is kept in memory for the lifetime of the hgweb-serving WSGI process, and we’ve experienced that sometimes requests to the pushlog can be served old information.
  3. During times of peak activity there was non-trivial IOWait which caused clone times to increase.
  4. Netapp licenses aren’t cheap. ;)

This took a lot of effort and coordination with the release engineering team to ensure that downtime was kept minimal and there were no feature or performance regressions along the way.

A large part of the transition was the necessity to rewrite the Puppet module that we use to deploy Mercurial. The module is now available on GitHub for people to comment on and use. Of course, pull requests are appreciated.

Now, on to some stats!

Before:

  • Sample size: 1063259
  • Mean: 16.9414 seconds per Hg operation (?cmd=getbundle)
  • Minimum: 0.000112
  • Maximum: 1353.56
  • 95th percentile: 19.551 seconds per Hg operation

After:

  • Sample size: 536678
  • Mean: 15.7102 seconds per Hg opeation (?cmd=getbundle)
  • Minimum: 0.000123
  • Maximum: 1236.69
  • 95th percentile: 19.5716 seconds per Hg operation

From that data we can’t see much significant performance improvement or degradation. I’m hoping that in the future we’ll be able to measure end-to-end client clone time to see better performance. With the local disks we’ll be able to stream uncompressed copies at line speed, which should result in 180 second clone times.

by bkero at March 24, 2014 08:08 PM

February 25, 2014

Brandon Philips

Slides: etcd at Go PDX

Last week I gave a talk at the PDX Go meetup (Go PDX). The presentation is a refinement on the talk I gave last month at GoSF but contains mostly the same content.

Several people in the audience had some experience with etcd already so it was great to hear their feedback on the project as a whole. The questions included partition tolerance and scaling properties, use cases and general design. It was a smart crowd and it was great to meet so many PDX Gophers.

Resources

etcd:

Raft:

by Brandon Philips at February 25, 2014 12:00 AM

February 16, 2014

Brandon Philips

Getting to Goven

This is the step by step story of how etcd, a project written in Go, arrived at using goven for library dependency management. It went through several evolutionary steps while trying to find a good solution to these basic goals:

  • Reproducible builds: given the same git hash and version of the Go compiler we wanted an identical binary everytime.
  • Zero dependencies: developers should be able to fork on github, make a change, build, test and send a PR without having anything more than a working Go compiler installed.
  • Cross platform: compile and run on OSX, Linux and Windows. Bonus points for cross-compilation.

Checked in GOPATH

Initially, to get reproducible builds and zero dependencies we checked in a copy of the GOPATH to “third_party/src”. Over time we encountered several problems:

  1. “go get github.com/coreos/etcd” was broken since downstream dependencies would change master and “go get” would setup a GOPATH that looked different than our checked in version.
  2. Windows developers had to have a working bash. Soon we had to maintain a copy of our build script written in Powershell.

At the time I felt that “go get” was an invalid use case since etcd was just a project built in Go and “go get” is primarliy useful for easily grabbing libraries when you are hacking on something. However, there was mounting user requests for a “go gettable” version of etcd.

To solve the Windows problem I wrote a script called “third_party.go” which ported the GOPATH management tools and the shell version of the “build” script to Go.

third_party.go

third_party.go worked well for a few weeks and we could remove the duplicate build logic in the Powershell scripts. The basic usage of was simple:

# Bump the raft dependency in the custom GOPATH
go run third_party.go bump github.com/coreos/go-etcd
# Use third_party.go to set GOPATH to third_party/src and build
go run third_party.go build github.com/coreos/etcd

But, there was a fatal flaw with this setup: it broke cross compilation via GOOS and GOARCH.

GOOS=linux go run third_party.go build github.com/coreos/etcd
fork/exec /var/folders/nq/jrsys0j926z9q3cjp1yfbhqr0000gn/T/go-build584136562/command-line-arguments/_obj/exe/third_party: exec format error

The reason is that GOOS and GOARCH get used internally by “go run`. Meaning it literally tries to build “third_party.go” as a Linux binary and runs it. Running a Linux binary on a OSX machine doesn’t work.

This soultion didn’t get us any closer to being “go gettable” either. There were several inquiries per week for this. So, I started looking around for better solutions and eventually settled on goven.

goven and goven-bump

goven achieves all of the desirable traits: reproducible builds, zero dependencies to start developing, cross compilation, and as a bonus “go install github.com/coreos/etcd” works.

The basic theory of operation is it checks all dependencies into subpackages of your project. Instead of importing “code.google.com/p/goprotobuf” you import github.com/coreos/etcd/third_party/code.google.com/p/goprotobuf. It makes the imports uglier but it is automated by goven.

Along the way I wrote some helper tools to assist in bumping dependencies which can be found on Github at philips/goven-bump. The scripts `goven-bump” and “goven-bump-commit” grab the hg revision or git hash of the dependency along with running goven. This makes bumping a dependency and getting a basic commit message as easy as:

cd ${GOPATH}/github.com/coreos/etcd
goven-bump-commit code.google.com/p/goprotobuf
git commit -m 'bump(code.google.com/p/goprotobuf): 074202958b0a25b4d1e194fb8defe5d69c300774'

goven and introduces some additional complexity for the maintainers of the project. But, the simplicity it presents to regular contributors and users used to “go get” make it worth the additional effort.

by Brandon Philips at February 16, 2014 12:00 AM

February 07, 2014

Russell Haering

Ridiculously Fast 'sprintf()' for Node.js

Today I was reminded of one of my neatest Node.js hacks. A few years ago, in the process of optimizing how Rackspace Cloud Monitoring compiles user-supplied alarms (a javascript-like DSL used to implement thresholds) we discovered that we were spending a significant amount of CPU time in a widely used Javascript implemetation of sprintf. This was back in the dark ages of Node.js, before util.format landed.

The CPU time spent in sprintf wasn't enough to be a problem: even compiling a few hundred thousand alarms is pretty fast, as compared to reading them out of a database, serializing the compiled alarms to XML, and loading them into Esper. Nonetheless, in a bout of "not invented here" and with a spirit of adventure in my heart, I did the obvious thing, and took a weekend to write a faster sprintf.

The standard implementation of sprintf takes a format string, followed by any number of positional arguments intended to be injected into the resulting string. It operates by parsing the format string using a series of regular expressions, to generate a parse tree consisting of alternate constant strings and formating placeholders.

For example, consider:

sprintf('The %s ran around the tree', 'dog');

The generated parse tree looks something like:

['The ', '%s', ' ran around the tree']

Then the tree is is iterated, and positional (or named) arguments injected to generate an array that can be joined into the appropriate result:

return ['The ', 'dog', ' ran around the tree'].join('');

As an optimization, the parse tree is cached for each format string, so that repeated calls to sprintf for a given format string need only repeat the actual argument injection.

So how can this be further optimized? We know a few things about V8:

  1. V8 is very good at concatenating strings.
  2. V8 is very good at just-in-time compiling "hot" functions.
  3. At least as of Crankshaft (the latest version of V8 I've used in any seriousness), V8 was unable to optimize code that treated arguments in unusual ways such as iterating it, or mixing its use with named arguments.

I was able to take advantage of these properties by generating a function which applied the format string through a single-line string concatenation, instead of instead of generating a parse tree. Taking the example above, I generate a string such as:

var fnBody = "return 'The ' + arguments[1] + ' jumped over the tree';";

Then compiling that string into a function on the fly:

return Function(fnBody);

By caching the resulting Function object, I was able to cause V8's JIT to optimize calls to sprintf into little more than a dictionary lookup, a function call and a string concatenation.

Security

An obvious risk of this strategy is that an attacker might find a way to cause us to generate arbitrary javascript.

This can be mitigated by never passing user-supplied input as a format string. In fact, because the cache doesn't implement any expiration, you should probably only ever pass literal format strings or you'll end up with a memory leak. This seems to be true of node-sprintf as well, so I don't consider it a serious limitation, just something to be aware of.

Performance

At the time, we saw marked (if not especially necessary) speedups in alarm compilation performance, but I don't have the bencharks on-hand. Instead, on a modern-ish version of Node.js (v0.10.17) running on my Macbook Pro I tested:

  1. My "fast" sprintf
  2. Node's util.format
  3. The widely used sprintf module

The test was:

for (var i = 0; i < 10000000; i++) {
  sprintf_fn('The %s jumped over a tree', i);
}

The results:

Implementation Time
fast sprintf 1504ms
util.format 14761ms
standard sprintf 22964ms

The improved sprintf lacks a lot of the functionality of the other implementations, so the comparison isn't entirely fair. Nonetheless, with a speedup of about 10x over util.format and 15x over sprintf (at least for this benchmark), I think its safe to declare this hack a success.

February 07, 2014 12:00 AM

January 29, 2014

Pranjal Mittal

Into the Vodafone 3G Data plan issues in India: Unjustified speed throttling even after data plan renewal


I recently encountered an issue with Vodafone 3G data plans in India. The issue is somewhat interesting, as it provides a technical insight into how Vodafone 3G Information systems work in India. I am using this plan in UP-East Zone.  Though my problem is not yet resolved and my inferences are just a speculation, this is what I have to say to Vodafone.


Dear Vodafone,

I am very sorry to say that I am extremely disappointed with your 3G services. More specficically, my grievance is with the way 3G data plans are managed by your computer systems. Your technical team fails to identify a glitch in your systems which could be affecting thousands of your customers.

Problem

I am using Vodafone 3G at a location with good Vodafone 3G network and have been using the service for about 6 months, regularly with no issues until I noticed this, strange, unjustified speed throttling that persists even after data plan renewal.

Timeline

31 Jan 2013
I activated a Rs 649 (3G data) plan on my mobile which assures 3 GB data with good speed and post 3 GB data consumption, limits speed to 64 kbps.
I get a message: "Your Unlimited Internet pack is active now valid upto 29.01.2014. ....."

31 Jan 2013 - 11 Jan 2013
3G internet works pretty good, good speed. No issues so far.

12 Jan 2014
I get a message: "You have exhausted the internet pack on your number ***** Continue to enjoy unlimited internet at reduced speed upto 64 kbps.

I notice that the speed has indeed decreased, but I am able to use the internet. Very well, things are as expected.  Your computer systems have done their role in throttilng my 3G speed limits to 64kbs or round about. That's fair enough, since I have exhausted my data plan. (No complaints till here)

14 Jan 2014
I have urgent work which requires good internet connectivity and high speed so I decide to activate a Rs 249 data plan that provides 1 GB (3G data) with 2-8 MBPS speeds.

14:31 IST:  I get a message: "Your Internet pack is active now valid upto 12.02.2014. Bal: 1024.00 MB. For Settings....."

I try to use the new data pack. I notice that there is no increase in speeds. Its still the same. Poor limited speeds like I have seen since 12 Jan 2014, even though I have paid for my new pack which is supposed to give me a good speed. I further notice that my new pack's allowed data balance is also being deducted which means it should be active.

I try my sim on different phones, a different 3G modem, everything that I have; and still get a very poor speed even on 3G. I ask my colleague's for their Vodafone 3G sims, use them on my phone and modem, precisely at the same location. They work pretty well and with a good speed.  Now, Its pretty clear to me that issue is not with my phone or modem.

About 19:00: I call customer care, that something is wrong with my 3G plan. Representative on call tells me my phone make from their records (Vodafone knows how to impress customers) , gives me some instructions, to clear my phone's browser Cache, History & Cookies. I emphasize that the issue is not in my phone browser. He still does not understand. He says that the issue has been registered and will be solved very soon.

I get an sms with my service registration number. (I better understand that as a a compliant number)

About 22:00: (same day), still on with the poor speeds and urgent internet requirements.
I make another call at about 22:30 to Customer care, to make my issue more explicit and describe the problem, and without even properly listening and registerting what I am saying he says my problem will be solved in 24 hours. I emphasize and ask him, "Will it be solved within 24 hours?" He says "yes"

15 Jan 2014
About 22:00 Still the same speed issue. Poor speeds. It has been 24 hours and my issue has not been resolved, contrary to what I had been told on phone. I call customer care, tell them to treat my matter on urgency. I tell them that if they can simply reactivate my internet pack, the issue might be resolved. (I'l explain in the end why!) [1]

The Customer care representantive tells me that my complaint is registered to be solved by 23-Jan-2014. Now this is where I am getting extremely angry. Why does it have to be so late? I explain him that simply deactivating all my 3G packs and activating the current plan again could solve the speed issue. He does not understand again and says that the issue will be solved soon.

(Side Note) Why don't customer care people, listen to "bug reports" more peacefully and take notes as they do so?

16 Jan 2014
19:00: My 3G speed issue is still not resolved. Still getting that poor, throttled 3G speed.
I try to call the customer care. I notice that I am unable to make calls to them on via 198. They have probably disabled my number due to the frequent calls. I don't care. But for sure, this is not how you treat customers.


The glitch I speculate in the Vodafone computer systems or whatever they are called in Telcom jargon.

[1] Somewhere in the Vodafone mobile records their computer systems set a throttle to the 3G speeds correspondong to my number after my Rs 649 data plan 3GB data got exhausted.
The throttle limit would be scheduled upto 29-Jan-2013 till which my plan was supposed to be active,
Activating a new 249 data plan on 14-Jan-2014, changed the data limit in my mobile record from Unlimited to 1024.00 MB, but did NOT reset the throttle limit.

If my speculation is correct, then it also follows that deactivating all 3G packs and reactivating new plan could solve the problem that I am facing which is what I suggested to the customer care, who did not understand it, and did not make any attempts in even trying to reactivate my internet plan.


P.S:
I am keen to know what Vodafone has to say on the issue. I would redact this blog post suitably after I get a deeper understanding on why I face this issue. 

by Pranjal Mittal (noreply@blogger.com) at January 29, 2014 08:58 PM

January 25, 2014

Ben Kero

Solving connectivity problems

This post deals with the technical challenges we’ve encountered while trying to establish reliable communications while staying in Rural Kenya. Some background information is necessary to understand the efforts we’ve gone through to remain connected.

This year the prestigious Hacker Beach event is taking place on the island of Lamu off the eastern coast of Kenya. The island is serviced by a single UMTS tower located above the hospital in the main town of Lamu City. However, our accommodation is on the other side of the island.

The situation

The situation

Our accommodation had a previously installed directional antenna on the roof to provide internet access. Unfortunately the access was very slow, with only 14% signal strength. This was complicated by strong winds blowing against antenna, causing it to be pointed in a wrong direction. This further reduced the cellular reception, sometimes making it disconnect completely.

 

The antenna solution. Photo by Sebastian Kippe (CC BY 2.0)

The antenna solution. Photo by Sebastian Kippe (CC BY 2.0)

An additional problem was that WiFi was served by a single Cradlepoint MBR1000 router in a corner of the fort, making it inaccessible through the impenetrably thick fort walls. This meant we were limited to camping in the upstairs dining hall, which worked well enough due to all the seating, but there was some desire to branch out to work from other areas of the fort, such as the knights-of-the-round-table-esque meeting room.

Our conspiratorium

For a group of 18 hackers, this level of connectivity was unacceptable. Many of us were making excursions into town to work at cafes with better reception. This was a problem because it threatened to undermine the spontaneous collaborative nature of Hacker Beach. The way we saw it there were two problems to fix:

  • Reception of the antenna was abysmal. Was this an inherent problem with the location?
  • WiFi reception was limited to only one corner of the house. Ideally the house should have WiFi everywhere.

We attempted to fix this by purchasing local SIM cards and installing them in portable WiFi Hotspot devices. Oddly enough we were able to receive some 3G reception if the devices were placed in some rather random areas of the fort. Unfortunately the connectivity of these devices wasn’t reliable enough for full-time hacking. So we began efforts in earnest to fix the connectivity problem.

We determined that the most appropriate solution to the WiFi problem was to employ PowerLine Ethernet adapters throughout the Fort to distribute connectivity. Simply repeating wireless signal was not a good option because of the lack of strategic locations to place wireless repeaters. The thick walls meant that the signal would be stopped between floors as well. We took a gamble and assumed that most outlets would be on the same power phase (if circuits are on different phases the PowerLine throughput will be severely limited, or likely not work at all). Since we had some new hackers approaching in a few days shipping was out of the question. Thankfully we were able t source some units in Athens, which (after some begging) our gracious friends were kind enough to pick up and bring for us.

The pairing part was easy, with WiFi SSID/password being copied using WPS. After pairing the devices could be moved anywhere in the fort to increase coverage. We installed two devices which are able to blanket the whole fort with connectivity. Problem solved.

PowerLine Ethernet adapter

Next was a trickier bit that required more calibration and special equipment. While inspecting the old antenna we found that the connectors had been tortured by the elements for several years. This meant that the antenna pigtail connectors were rusted, which was likely causing reception issues. Another problem was that the pigtail was being run through a window, which was then closing on it. We feared this was crushing the cable, which could have easily caused our antenna to become useless.

3G Modem and antenna cable run through window

There were several more hackers arriving from Nairobi in a few days, so we asked them to bring some antenna gear to hopefully help improve our connectivity. In total a questionably-EDGE amplifier, directional antenna, and some cabling was delivered when the hackers arrived early yesterday morning. It didn’t take long for us to tear it all open and start installing it.

Equipped with a laptop, an antenna, and a downstairs accomplice we disconnected the old antenna and threw a new line down to connect to the 3G modem. Next I had opened the router’s modem status page to measure signal strength while another hacker determined the direction the antenna should face to get the best reception. Our best direction was pressed against the old antenna; the people who installed the last one must have known what they were doing.

Unfortunately we were only armed with my multitool which meant that proper mounting was going to be impossible. We tried wrenching the existing nuts that held the antenna in place, but they proved to be well stuck with a decade of rust and generally brutal African elements. Not even cooking oil (our improvised WD-40) would help loosen the offending nuts. Ultimately we ended up doing a bodge job to keep the antenna in place. One of the hackers had brought string with him, which we used to tie the new antenna’s base plate to the old antenna. This worked surprisingly well, although is a horrifyingly temporarily solution. The string will not stand up to more than a few days outside here. Next we plan to source some tools locally and perform a permanent installation of the antenna.

With the old antenna the signal strength would consistently be about 14%, which resulted in throughput of about 200 kbit. After out new antenna was installed and calibrated we were able to see signal strength of up to 80%, which gave us upwards of 1800 kbit of throughput with consistent pings of about 250 ms. Hooray!

After applying liberal traffic shaping on the router we are now able to comfortably surf the internet, download packages, and use IRC.

by bkero at January 25, 2014 12:18 PM

January 18, 2014

Brandon Philips

Video: etcd at GoSF

Last week I gave a talk at the San Francisco Go meetup (GoSF). The event was great and has about 200 Go Gophers in attendance.

Giving the talk was great because it made me realize how much we have accomplished on etcd since my last talk in October. The audience was mostly curious about how it differs from Zookeeper, how master elections work, and how we were testing various failure modes. A great suggestion from Brad Fitz was to use a mock of net.Conn to test various network problems. I hope to start executing on that soon.

by Brandon Philips at January 18, 2014 12:00 AM

January 12, 2014

Justin Dugger

LCA 2014 Videos of Note

Linuxconf 2014 wrapped up last week, and the videos are already online!

I didn't get a chance to review all the video, but here's some of the sessions I thought were interesting:

Rusty Russel discusses virtIO standardization. I thought I knew what virtIO was but his initial explaination leaves me more confused than I started out. Nevertheless, Rusty gives a implementer's view of the standardization process, and shares how virtIO manages forward and backward compatibility between hypervisor, guest OSes, and even hardware.

Elizabeth Krumbach Joseph explains how the OpenStack Core Infra team publishes does their work in the open. We've taken a similar approach, so its nice to see other approaches and bits we might steal =). Storing Jenkins jobs in YAML in config management sounds very nice, and I will have to bring it up at my next meeting.

Bdale Garbee shares his experience losing his home to the Black Forest Fire. As a serial renter / mover, I'm already well prepared to answer the question "What would you take if you had five minutes to clean out your home?" So I would have liked a bit more in the way of disaster recovery / offsite backups / tech stuff, but but I happen to know he rescued his servers from the fire and isn't storing them locally anymore. So perhaps there is no lesson to share yet =)

Michael Still presents a third party CI approach for database migrations in OpenStack. Looks like a combo of gerrit for code reviews, Zuul, and some custom zuul gearman worker. Surprisingly little duplicate content from the other open stack infrastructure talk!

Jim Cheetham asks 'Is it safe to mosh?' The answer appears to be yes, but takes a hands off approach to the underlying cryto.

Lots of exciting talks, and maybe I need to sit down and think about writing my own proposal for LCA 2015.

by Justin Dugger at January 12, 2014 12:00 AM

December 10, 2013

Pranjal Mittal

Interaction between Ganeti Web Manager and Ganeti - Notes



Ok, so this is not a blog post but an extract from some of my notes from the past. I wrote these notes while I was trying to figure out how the Ganeti Web Manager, communicates with Ganeti a cluster-VM management software roughly 6 months back.
GWM is a project initiated at the Open Source Lab and is an amazing application that administrators could use to manage their Ganeti Clusters.

So there you go, my anachronistic blog post or rather “notes of the past” start here:
(Disclaimer: The notes are pretty raw!)


-------------------------------------------------------------------------------------------------------------------------------------------------
I was trying to have a look at the ganeti_webmgr code as an absolute beginner. I had done a fair bit of reading about Ganeti. The Ganeti CLI, setting up a small test cluster of 3 nodes and related tools installed, terminology about Nodes, Clusters, Virtual Machines, etc


Without diving too much into the subtleties, I first wanted to know the bigger picture: of how GWM as a Django Application is able to interact with Ganeti which is a separate tool generally used via a CLI ?

On thinking a bit more,these were the possible ways that came to my mind:


(1) GWM making os.system() calls to ganeti-cli with appropriate parameters and reading results from stdout?


(2) Calls to Ganeti Rest / Remote API via some client? (“Ganeti RAPI Client” being the correct jargon)


(3) Some other Python module / API that provides access to function equivalents of the Ganeti CLI.

As for what I could make out from the code,  RAPI was being used; but I was still not clear on how to use it well.


I asked questions on the mailing list and IRC to finally understand some important concepts:


The interaction between GWM Layer and Ganeti layer is entirely using RAPI.
A RAPI Client has been made available as:


ganeti_webmgr/ganeti_web/utils/client.py


The GWM models do almost all work of dealing with the RAPI.

This is because, GWM uses the idea of caching any objects and response from the RAPI as model objects. These model or database objects are refreshed typically on changes and timeouts.
The idea behind doing this is that frequent calls to RAPI would be costly and cause a lot of delay. Typically, the state of the Cluster, Virtual Machines, Node’s, etc do not change unless some action is taken (a Job is run) Example: Creating a new virtual machine.

This is what Chance (OSL, Student Developer) told me when I was trying to understand the idea:
“The only time we should deal directly with using the RAPI Client is when making model changes (which isn't too often).”

Thats right. models.py is something that you wouldn’t want to change often, for all the migrations you have to run after that..... unless you have something really bad about your planned structure, or a new feature that entirely depends on new / modified models.

Are we simply instantiating the RAPI Client object every time to make use of it for the Ganeti specific calls?

So I got the Ganeti RAPI docs too:

Instantiating the RAPI object


rapi = client.GanetiRapiClient(host, port, user, password, timeout=settings.RAPI_CONNECT_TIMEOUT)

Useful functions inside the python RAPI client.

cat ganeti_webmgr/ganeti_webmgr/utils/client.py | grep def
The results give the prototypes of useful functions that gives a top level idea of what the client provides. The name of the functions are self explanatory and the parameters give me a top level idea of what input the function expects.

def prepare_query(query):
def __init__(self, host, port=GANETI_RAPI_PORT, username=None,

def _SendRequest(self, method, path, query=None, content=None):

def GetVersion(self):

def GetFeatures(self):
def GetOperatingSystems(self):
def GetInfo(self):
def RedistributeConfig(self):
def ModifyCluster(self, **kwargs):
def GetClusterTags(self):
def AddClusterTags(self, tags, dry_run=False):
def DeleteClusterTags(self, tags, dry_run=False):
def GetInstances(self, bulk=False):
def GetInstance(self, instance):
def GetInstanceInfo(self, instance, static=None):
def CreateInstance(self, mode, name, disk_template, disks, nics,
def DeleteInstance(self, instance, dry_run=False):
def ModifyInstance(self, instance, **kwargs):
def ActivateInstanceDisks(self, instance, ignore_size=False):
def DeactivateInstanceDisks(self, instance):
def RecreateInstanceDisks(self, instance, disks=None, nodes=None):
def GrowInstanceDisk(self, instance, disk, amount, wait_for_sync=False):
def GetInstanceTags(self, instance):
def AddInstanceTags(self, instance, tags, dry_run=False):
def DeleteInstanceTags(self, instance, tags, dry_run=False):
def RebootInstance(self, instance, reboot_type=None,
def ShutdownInstance(self, instance, dry_run=False, no_remember=False,
def StartupInstance(self, instance, dry_run=False, no_remember=False):
def ReinstallInstance(self, instance, os=None, no_startup=False,
def ReplaceInstanceDisks(self, instance, disks=None,
def PrepareExport(self, instance, mode):
def ExportInstance(self, instance, mode, destination, shutdown=None,
def MigrateInstance(self, instance, mode=None, cleanup=None):
def FailoverInstance(self, instance, iallocator=None,
def RenameInstance(self, instance, new_name, ip_check,
def GetInstanceConsole(self, instance):
def GetJobs(self):
def GetJobStatus(self, job_id):
def WaitForJobChange(self, job_id, fields, prev_job_info, prev_log_serial):
def CancelJob(self, job_id, dry_run=False):
def GetNodes(self, bulk=False):
def GetNode(self, node):
def EvacuateNode(self, node, iallocator=None, remote_node=None,
def MigrateNode(self, node, mode=None, dry_run=False, iallocator=None,
def GetNodeRole(self, node):
def SetNodeRole(self, node, role, force=False, auto_promote=False):
def PowercycleNode(self, node, force=False):
def ModifyNode(self, node, **kwargs):
def GetNodeStorageUnits(self, node, storage_type, output_fields):
def ModifyNodeStorageUnits(self, node, storage_type, name,
def RepairNodeStorageUnits(self, node, storage_type, name):
def GetNodeTags(self, node):
def AddNodeTags(self, node, tags, dry_run=False):
def DeleteNodeTags(self, node, tags, dry_run=False):
def GetGroups(self, bulk=False):
def GetGroup(self, group):
def CreateGroup(self, name, alloc_policy=None, dry_run=False):
def ModifyGroup(self, group, **kwargs):
def DeleteGroup(self, group, dry_run=False):
def RenameGroup(self, group, new_name):
def AssignGroupNodes(self, group, nodes, force=False, dry_run=False):
def GetGroupTags(self, group):
def AddGroupTags(self, group, tags, dry_run=False):
def DeleteGroupTags(self, group, tags, dry_run=False):
def Query(self, what, fields, qfilter=None):
def QueryFields(self, what, fields=None):


by Pranjal Mittal (noreply@blogger.com) at December 10, 2013 06:04 PM

November 21, 2013

Ben Kero

Day 51

I’ve been as terrible about updating this as I expected to be. Nevertheless, I am struck with inspiration (or maybe it’s just energy from coffee), so another post must be written!

For the next week (and the previous week) I’m spending time in Paris. This turned out to be largely a convenient set of circumstances, since I had an excellent experience when I was here two weeks ago, and I wished I could spend more time here.

I was fortunate enough to be here for two concerts I wanted to attend, namely Morcheeba and Parov Stelar. Both were at the excellent venue L’Olympia. The concert halls were different than I’m used to in the US, being much smaller, serving actual beer in pint sizes, and being a much more stream-lined affair in terms of coat checks, getting in, and getting out. The Morcheeba concert had excellent music (Ross Godfrey can play some mean guitar solos), however the overall experience had a certain quality missing that I cannot put into words.

The Parov Stelar concert certainly had that in abundance though. The audience drew a much bigger and younger crowd. The cheering and applause didn’t stop between songs, sometimes for an awkwardly long period of time. The music and improvisation was excellent though, and I would very much appreciate seeing another performance. If you have the chance to attend I would give it my highest recommendation.

My coworkers and friends in Paris have made sure that my time here was not badly spent. From simple (and extremely rewarding) experiences such as sitting at a cafe and sipping espresso while disconnecting and reading a book to heading out on the town and experiencing the best (and sometimes worst) that Paris has to offer in terms of comestibles, craft quaffables, sights, people watching, and even cinema.

Being a non-francophone is obviously a barrier, but having a rudimentary understanding of Latin and Spanish, and a few brief lessons on Duolingo give me enough knowledge for a superficial attempt at speaking French, if nothing else to gain a bit of respect and understanding when my conversation parter and I have to switch to English. Duolingo has been a great method for language learning though, and is a very powerful tool given one actually puts in the effort to retain the information learned in the lessons.

There are several things I want to fill my remaining week with. The Musée d’Orsay is something I simply must see before I leave. It’s something I missed last time I was here, and I’ve been regretting that ever since. It’s a fantastically large gallery with an unparalleled collection of impressionist artworks. Beyond this, the museum is actually built in a beautifully restored train station.

Additionally I want to check out what’s happening at some of Paris’ hackerspaces. There seem to be several active ones in Paris, and many are actually located in squats, which seems sort of novel to me. I associate hackerspaces as techno-artist and creative havens, and their efforts are always unique and interesting to discover. In particular I look forward to connecting with Le Loop and /tmp/lab.

by bkero at November 21, 2013 10:05 AM

October 01, 2013

Brandon Philips

Video: Modern Linux Server with Containers

At LinuxCon 2013 I gave a talk that dissects “Linux Containers” into its component parts in the Kernel: cgroups and namespaces. The talk shows how cgroups act as the “accounting bean counter” and namespaces as the “castle walls” that isolate processes from each other.

If you are already familiar with the basics of namespaces and cgroups I show off some tools like nsenter, docker, and systemd-nspawn. Skip to the end to catch the demos.

The full slides are availble on slide deck and mirrored as a pdf here.

by Brandon Philips at October 01, 2013 12:00 AM

May 01, 2013

Beaver BarCamp

Students and community members learn together at Beaver BarCamp 11

Quotes and select photos from Beaver BarCamp 11 are now available on the OSU Open Source Lab's website.

A gallery of user-contributed photos from the event is also available in a Google Plus photo gallery.

by OSU Open Source Lab at May 01, 2013 07:12 PM

Students and community members learn together at Beaver BarCamp 11

Quotes and select photos from Beaver BarCamp 11 are now available on the OSU Open Source Lab's website.

A gallery of user-contributed photos from the event is also available in a Google Plus photo gallery.

by OSU Open Source Lab at May 01, 2013 07:12 PM

April 27, 2013

GOSCON News

It's All About Community: DC Metro Open Source Community Summit May 10, 2013

Oregon State University Open Source Lab is pleased lend its support to the Open Source Initiative and the first Open Source Community Summit, being held in Washington D.C. on May 10, 2013.

It's a great way to stand up and be counted as part of the DC open source comunity; check it out!more...

by deborah at April 27, 2013 05:53 AM

April 01, 2013

Alex Polvi

"It is not the critic who counts; not the man who points out how the strong man stumbles, or where..."

“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.”

- Theodore Roosevelt

April 01, 2013 03:29 AM

January 05, 2013

Alex Polvi

Talk @ http://svc2baltics.com/ in Vilnus, Lithuania, with a...



Talk @ http://svc2baltics.com/ in Vilnus, Lithuania, with a 1000+ students interested in starting up. 

January 05, 2013 03:56 AM

November 11, 2012

Alex Polvi

Q&A in UAE w/entrepreneurs and @davemcclure at MAKE.





Q&A in UAE w/entrepreneurs and @davemcclure at MAKE.

November 11, 2012 05:00 AM

September 30, 2012

Justin Dugger

PuppetConf 2012

Recovered from the post-con crash a while ago, so it's time to write up some thoughts. Last week I attended PuppetConf with my coworkers at the OSL. The OSL attended PuppetConf primarily as a pre-deployment information gathering exercise. We want to avoid common pitfalls, and be able to plan for things coming down the pipeline. Puppet 3.0 was targetted to be released on Friday and clearly that slipped.

The venue itself was nice, but space partitioned poorly. The two main tracks had surplus space, but the three side tracks nearly always had people turned away for space concerns. Supposedly, the recordings will be available shortly, so it may not be the Worst Thing In The World, but only time will tell.

Content wise, one recurring theme is to start small and simple, and not worry about scale or sharing until they become an issue. Designing a deployment for thousands of nodes when you have perhaps a dozen gives new life to the term "architecture astronaut," and there's a certain amount of benefit to procrastinating on system design while the tools and ecosystem mature. Basically, build one to throw away.

Another problem we've been worrying about at the OSL is updating 3rd party config modules in their various forms. The hope is that by explicitly annotating in your system where things came from, you can automate pulling in updates from original sources. Pretty much the universal recommendation here is a condemnation: avoid git submodules. Submodules sounds like the right strategy, but it's for a different use case. In our experience, it dramatically complicates the workflow. At least one person mentioned librarian-puppet, which as far as I can tell is isn't much different than mr with some syntactic sugar for PuppetForge. This is great, because mr was basically the strategy I was recommending prior to PuppetConf.

The Better Living Through Statistics talk was less advanced than I'd hoped. Anyone who's spent maybe 5 minutes tuning nagios check_disks realizes how inadequate it is, and that the basic nagios framework is to blame. What you really want is an alert when the time to disk outage approaches time to free up more disk, and no static threshold can capture that. While Jamie did provide a vision for the future, I was really hoping for some new statistical insight on the problem. It appears it's up to me to create and provide said insight. Perhaps in another post.

R Tyler Croy gave a useful talk on behavior/test driven infrastructure. I'd looked into Cucumber before, but RSpec was only a word to me before this talk. It's certainly something I'll need to take some time to integrate into the workflow and introduce to students. One concern I had (that someone else aired) was that in the demo, the puppet code and the code to test it was basically identical, such that software could easily translate from code to test and back. Croy insisted this was not the case in more complicated Puppet modules, but I'm reserving judgement until I see said modules.

Overall, I'd definately recommend the conference to people preparing to deploy puppet. There's plenty more sessions I didn't cover in here that are worth your time. You'd probably get the most out of it by starting a trial implementation first, instead of procrastinating until Wednesday night to read the basics like I did. Beyond simply watching lectures, it's useful to get away from the office and sit down to learn about this stuff. Plus, it's useful to build your professional network of people you can direct questions to later.

by Justin Dugger at September 30, 2012 12:00 AM

July 01, 2012

Justin Dugger

Open Source Bridge Wrapup

Friday marked the end of Open Source Bridge. Just about the best introduction to Portland culture as you can find. Vegan lunches, Voodoo Donut catering, lunch truck friday, and rock and roll pipe organists in the Unitarian's sanctuary.

The keynotes were pretty cool. I'd seen Fenwick's presentation from LCA, and was surprised at how much had changed, hopefully since some of his keystone evidence turned out to be bogus; turns out there's strong evidence that the only "priming" effect was in grad students running the study. I'm still not quite clear on what JScott wants people to run vbox for, but he did have a really good idea about bringing your own recording equipment that I wish I had taken to heart.

Probably the most useful talk I attended was Laura Thompson's presentation on Mozilla's Crash Reporting service, powered by Socorro. A few of the projects the OSL hosts are desktop apps and collecting crash data might be a good engineering tool win for them. A lot of embedded hardware talks that would have been interesting, but not directly relevant to the needs of the OSL. Hopefully they'll be up as recordings soon.

The OSL was also well represented as well in the speaker's ranks: we ran five sessions during the main conference, and two during the Friday unconference. I think next year it would be a good idea to encourage our students to participate as volunteers; getting them facetime with speakers and the community at large can only do us a world of good. I gave a first run of a talk on using GNUCash for personal finance; the turnout was pretty good, given how many people were still at the food carts. I should have recorded it to self-critique and improve.

The "after party" on Thursday was nice. Lance won the 2012 Outsanding Open Source Citizen award, which is great, because he deserves recongition for handling the turmoil at the OSL over the past year. But now I've got to figure out my plan meet or beat that for next year. No small task.

Next up is catching up back at the Lab, and then OSCON!

by Justin Dugger at July 01, 2012 12:00 AM

June 13, 2012

Lance Albertson

Ganeti Tutorial PDF guide

As I mentioned in my previous blog post, trying out Ganeti can be cumbersome and I went out and created a platform for testing it out using Vagrant. Now I have a PDF guide that you can use to walk through some of the basics steps of using Ganeti along with even testing a fail-over scenario. Its an updated version of a guide I wrote for OSCON last year. Give it a try and let me know what you think!

by lance at June 13, 2012 01:53 AM

June 11, 2012

Frédéric Wenzel

Fail Pets Research in UX Magazine

I totally forgot blogging about this!

Remember how I curate a collection of fail pets across the Interwebs? Sean Rintel is a researcher at the University of Queensland in Australia and has put some thought into the UX implications of whimsical error messages, published in his article: The Evolution of Fail Pets: Strategic Whimsy and Brand Awareness in Error Messages in UX Magazine.

In his article, Rintel attributes me with coining the term "fail pet".

Attentive readers may also notice that Mozilla's strategy of (rightly) attributing Adobe Flash's crashes with Flash itself by putting a "sad brick" in place worked formidably: Rintel (just like most users, I am sure) assumes this message comes from Adobe, not Mozilla:

Thanks, Sean, for the mention, and I hope you all enjoy his article.

June 11, 2012 07:00 AM

June 08, 2012

Frédéric Wenzel

Let's talk about password storage

Note: This is a cross-post of an article I published on the Mozilla Webdev blog this week.

During the course of this week, a number of high-profile websites (like LinkedIn and last.fm) have disclosed possible password leaks from their databases. The suspected leaks put huge amounts of important, private user data at risk.

What's common to both these cases is the weak security they employed to "safekeep" their users' login credentials. In the case of LinkedIn, it is alleged that an unsalted SHA-1 hash was used, in the case of last.fm, the technology used is, allegedly, an even worse, unsalted MD5 hash.

Neither of the two technologies is following any sort of modern industry standard and, if they were in fact used by these companies in this fashion, exhibit a gross disregard for the protection of user data. Let's take a look at the most obvious mistakes our protagonists made here, and then we'll discuss the password hashing standards that Mozilla web projects routinely apply in order to mitigate these risks.

A trivial no-no: Plain-text passwords

This one's easy: Nobody should store plain-text passwords in a database. If you do, and someone steals the data through any sort of security hole, they've got all your user's plain text passwords. (That a bunch of companies still do that should make you scream and run the other way whenever you encounter it.) Our two protagonists above know that too, so they remembered that they read something about hashing somewhere at some point. "Hey, this makes our passwords look different! I am sure it's secure! Let's do it!"

Poor: Straight hashing

Smart mathematicians came up with something called a hashing function or "one-way function" H: password -> H(password). MD5 and SHA-1 mentioned above are examples of those. The idea is that you give this function an input (the password), and it gives you back a "hash value". It is easy to calculate this hash value when you have the original input, but prohibitively hard to do the opposite. So we create the hash value of all passwords, and only store that. If someone steals the database, they will only have the hashes, not the passwords. And because those are hard or impossible to calculate from the hashes, the stolen data is useless.

"Great!" But wait, there's a catch. For starters, people pick poor passwords. Write this one in stone, as it'll be true as long as passwords exist. So a smart attacker can start with a copy of Merriam-Webster, throw in a few numbers here and there, calculate the hashes for all those words (remember, it's easy and fast) and start comparing those hashes against the database they just stole. Because your password was "cheesecake1", they just guessed it. Whoops! To add insult to injury, they just guessed everyone's password who also used the same phrase, because the hashes for the same password are the same for every user.

Worse yet, you can actually buy(!) precomputed lists of straight hashes (called Rainbow Tables) for alphanumeric passwords up to about 10 characters in length. Thought "FhTsfdl31a" was a safe password? Think again.

This attack is called an offline dictionary attack and is well-known to the security community.

Even passwords taste better with salt

The standard way to deal with this is by adding a per-user salt. That's a long, random string added to the password at hashing time: H: password -> H(password + salt). You then store salt and hash in the database, making the hash different for every user, even if they happen to use the same password. In addition, the smart attacker cannot pre-compute the hashes anymore, because they don't know your salt. So after stealing the data, they'll have to try every possible password for every possible user, using each user's personal salt value.

Great! I mean it, if you use this method, you're already scores better than our protagonists.

The 21st century: Slow hashes

But alas, there's another catch: Generic hash functions like MD5 and SHA-1 are built to be fast. And because computers keep getting faster, millions of hashes can be calculated very very quickly, making a brute-force attack even of salted passwords more and more feasible.

So here's what we do at Mozilla: Our WebApp Security team performed some research and set forth a set of secure coding guidelines (they are public, go check them out, I'll wait). These guidelines suggest the use of HMAC + bcrypt as a reasonably secure password storage method.

The hashing function has two steps. First, the password is hashed with an algorithm called HMAC, together with a local salt: H: password -> HMAC(local_salt + password). The local salt is a random value that is stored only on the server, never in the database. Why is this good? If an attacker steals one of our password databases, they would need to also separately attack one of our web servers to get file access in order to discover this local salt value. If they don't manage to pull off two successful attacks, their stolen data is largely useless.

As a second step, this hashed value (or strengthened password, as some call it) is then hashed again with a slow hashing function called bcrypt. The key point here is slow. Unlike general-purpose hash functions, bcrypt intentionally takes a relatively long time to be calculated. Unless an attacker has millions of years to spend, they won't be able to try out a whole lot of passwords after they steal a password database. Plus, bcrypt hashes are also salted, so no two bcrypt hashes of the same password look the same.

So the whole function looks like: H: password -> bcrypt(HMAC(password, local_salt), bcrypt_salt).

We wrote a reference implementation for this for Django: django-sha2. Like all Mozilla projects, it is open source, and you are more than welcome to study, use, and contribute to it!

What about Mozilla Persona?

Funny you should mention it. Mozilla Persona (née BrowserID) is a new way for people to log in. Persona is the password specialist, and takes the burden/risk away from sites for having to worry about passwords altogether. Read more about Mozilla Persona.

So you think you're cool and can't be cracked? Challenge accepted!

Make no mistake: just like everybody else, we're not invincible at Mozilla. But because we actually take our users' data seriously, we take precautions like this to mitigate the effects of an attack, even in the unfortunate event of a successful security breach in one of our systems.

If you're responsible for user data, so should you.

If you'd like to discuss this post, please leave a comment at the Mozilla Webdev blog. Thanks!

June 08, 2012 07:00 AM

June 03, 2012

Alex Polvi

Silicon Valley comes to Beijing and Hong Kong! Awesome crew.



Silicon Valley comes to Beijing and Hong Kong! Awesome crew. 

June 03, 2012 04:00 AM

May 31, 2012

Greg Lund-Chaix

Large Moodle downloads die prematurely when served through Varnish

Varnish and Moodle, to be blunt, hate each other. So much so that for my Moodle 1.9.x sites, I simply instruct Varnish to return(pass) without even trying to cache anything on a Moodle site. Today, however, I discovered even that is insufficient. Here’s what happened:

A user was reporting that when downloading large files from within Moodle (500mb course zip backups in this case), they’d stop at approximately 200mb. A look at varnishlog showed that Varnish was properly seeing that it’s a Moodle request that had a “Cache-Control: no-cache” header and didn’t even try to cache it before sending the request off to the backend. The backend was behaving exactly as expected and serving up the file. At some point, however, the download simply terminates before completion. No indications in the Varnish or Apache logs, nothing. It just … stops.

Huh.

So I put the following code in my VCL in vcl_recv:

if (req.url ~ "file.php") {
return (pipe);
}

Success!

Note: this must go into the VCL before the line in vcl_recv that checks the Cache-Control header, otherwise it’ll pass before it gets to the pipe:

if (req.url ~ "file.php") {
return (pipe);
}

# Force lookup if the request is a no-cache request from the client
if (req.http.Cache-Control ~ "no-cache") {
return (pass);
}

Share this: Digg del.icio.us Facebook Google Bookmarks Furl Print Reddit Slashdot StumbleUpon Technorati TwitThis Fark LinkedIn Ma.gnolia NewsVine Ping.fm Pownce Tumblr

by Greg at May 31, 2012 02:42 AM

May 30, 2012

Frédéric Wenzel

Fun with ebtables: Routing IPTV packets on a home network

In my home network, I use IPv4 addresses out of the 10.x.y.z/8 private IP block. After AT&T U-Verse contacted me multiple times to make me reconfigure my network so they can establish a large-scale NAT and give me a private IP address rather than a public one (this might be material for a whole separate post), I reluctantly switched ISPs and now have Comcast. I did, however, keep AT&T for television. Now, U-Verse is an IPTV provider, so I had to put the two services (Internet and IPTV) onto the same wire, which as it turned out was not as easy as it sounds.

tl;dr: This is a "war story" more than a crisp tutorial. If you really just want to see the ebtables rules I ended up using, scroll all the way to the end.

IPTV uses IP Multicast, a technology that allows a single data stream to be sent to a number of devices at the same time. If your AT&T-provided router is the centerpiece of your network, this works well: The router is intelligent enough to determine which one or more receivers (and on what LAN port) want to receive the data stream, and it only sends data to that device (and on that wire).

Multicast, the way it is supposed to work: The source server (red) sending the same stream to multiple, but not all, receivers (green).

Turns out, my dd-wrt-powered Cisco E2000 router is--out of the box--not that intelligent and, like most consumer devices, will turn such multicast packets simply into broadcast packets. That means, it takes the incoming data stream and delivers it to all attached ports and devices. On a wired network, that's sad, but not too big a deal: Other computers and devices will see these packets, determine they are not addressed to them, and drop the packets automatically.

Once your wifi becomes involved, this is a much bigger problem: The IPTV stream's unwanted packets easily satisfy the wifi capacity and keep any wifi device from doing its job, while it is busy discarding packets. This goes so far as to making it entirely impossible to even connect to the wireless network anymore. Besides: Massive, bogus wireless traffic empties device batteries and fills up the (limited and shared) frequency spectrum for no useful reason.

Suddenly, everyone gets the (encrypted) data stream. Whoops.

One solution for this is only to install manageable switches that support IGMP Snooping and thus limit multicast traffic to the relevant ports. I wasn't too keen on replacing a bunch of really expensive new hardware though.

In comes ebtables, part of netfilter (the Linux kernel-level firewall package). First I wrote a simple rule intended to keep all multicast packets (no matter their source) from exiting on the wireless device (eth1, in this case).

ebtables -A FORWARD -o eth1 -d Multicast -j DROP

This works in principle, but has some ugly drawbacks:

  1. -d Multicast translates into a destination address pattern that also covers (intentional) broadcast packets (i.e., every broadcast packet is a multicast packet, but not vice versa). These things are important and power DHCP, SMB networking, Bonjour, ... . With a rule like this, none of these services will work anymore on the wifi you were trying to protect.
  2. -o eth1 keeps us from flooding the wifi, but will do nothing to keep the needless packets sent to wired devices in check. While we're in the business of filtering packets, might as well do that too.

So let's create a new VLAN in the dd-wrt settings that only contains the incoming port (here: W) and the IPTV receiver's port (here: 1). We bridge it to the same network, because the incoming port is not only the source of IPTV, but also our connection to the Internet, so the remaining ports need to be able to connect to it still.

dd-wrt vlan settings

Then we tweak our filters:

ebtables -A FORWARD -d Broadcast -j ACCEPT
ebtables -A FORWARD -p ipv4 --ip-src ! 10.0.0.0/24 -o ! vlan1 -d Multicast -j DROP

This first accepts all broadcast packets (which it would do by default anyway, if it wasn't for our multicast rule), then any other multicast packets are dropped if their output device is not vlan1, and their source IP address is not local.

With this modified rule, we make sure that any internal applications can still function properly, while we tightly restrict where external multicast packets flow.

That was easy, wasn't it!

Some illustrations courtesy of Wikipedia.

May 30, 2012 07:00 AM

May 21, 2012

Lance Albertson

Trying out Ganeti with Vagrant

Ganeti is a very powerful tool but often times people have to look for spare hardware to try it out easily. I also wanted to have a way to easily test new features of Ganeti Web Manager (GWM) and Ganeti Instance Image without requiring additional hardware. While I do have the convenience of having access to hardware at the OSU Open Source Lab to do my testing, I'd rather not depend on that always. Sometimes I like trying new and crazier things and I'd rather not break a test cluster all the time. So I decided to see if I could use Vagrant as a tool to create a Ganeti test environment on my own workstation and laptop.

This all started last year while I was preparing for my OSCON tutorial on Ganeti and was manually creating VirtualBox VMs to deploy Ganeti nodes for the tutorial. It worked well but soon after I gave the tutorial I discovered Vagrant and decided to adapt my OSCON tutorial with Vagrant. Its a bit like the movie Inception of course, but I was able to successfully get Ganeti working with Ubuntu and KVM (technically just qemu) and mostly functional VMs inside of the nodes. I was also able to quickly create a three-node cluster to test failover with GWM and many facets of the webapp.

The vagrant setup I have has two parts:

  1. Ganeti Tutorial Puppet Module
  2. Ganeti Vagrant configs

The puppet module I wrote is very basic and isn't really intended for production use. I plan to re-factor it in the coming months into a completely modular production ready set of modules. The node boxes are currently running Ubuntu 11.10 (I've been having some minor issues getting 12.04 to work), and the internal VMs you can deploy are based on the CirrOS Tiny OS. I also created several branches in the vagrant-ganeti repo for testing various versions of Ganeti which has helped the GWM team implement better support for 2.5 in the upcoming release.

To get started using Ganeti with Vagrant, you can do the following:

git clone git://github.com/ramereth/vagrant-ganeti.git
git submodule update --init
gem install vagrant
vagrant up node1
vagrant ssh node1
gnt-cluster verify

Moving forward I plan to implement the following:

  • Update tutorial documentation
  • Support for Xen and LXC
  • Support for CentOS and Debian as the node OS

Please check out the README for more instructions on how to use the Vagrant+Ganeti setup. If you have any feature requests please don't hesitate to create an issue on the github repo.

by lance at May 21, 2012 06:09 AM

April 26, 2012

Jeff Sheltren

Memcached and PECL memcache on CentOS and Fedora

At Tag1 Consulting we do a lot of work on increasing web site performance, especially around Drupal sites. One of the common tools we use is memcached combined with the Drupal Memcache module. In Drupal, there are a number of different caches which are stored in the (typically MySQL) database by default. This is good for performance as it cuts down on potentially large/slow SQL queries and PHP execution needed to display content on a site. The Drupal Memcache module allows you to configure some or all of those caches to be stored in memcached instead of MySQL, typically these cache gets/puts in memcache are much faster than they would be in MySQL, and at the same time it decreases work load on the database server. This is all great for performance, but it involves setting up an additional service (memcached) as well as adding a PHP extension in order to communicate with memcached. I've seen a number of guides on how to install these things on Fedora or CentOS, but so many of them are out-dated or give instructions which I wouldn't suggest such as building things from source, installing with the 'pecl' command (not great on a package based system), or using various external yum repositories (some of which don't mix well with the standard repos). What follows is my suggested method for installing these needed dependencies in order to use memcached with Drupal, though the same process should be valid for any other PHP script using memcache.

PECL Packages

For the Drupal Memcache module, either the PECL memcache or PECL memcached (note the 'd'!) extensions can be used. While PECL memcached is newer and has some additional features, PECL memcache (no 'd'!) tends to be better tested and supported, at least for the Drupal Memcache module. Yes, the PECL extension names are HORRIBLE and very confusing to newcomers! I almost always use the PECL memcache extension because I've had some strange behavior in the past using the memcached extension; likely those problems are fixed now, but it's become a habit and personal preference to use the memcache extension.

Installing and Configuring memcached

The first step is to get memcached installed and configured. CentOS 5 and 6 both include memcached in the base package repo, as do all recent Fedora releases. To install memcached is simply a matter of:
# yum install memcached

Generally, unless you really know what you're doing, the only configuration option you'll need to change is the amount of memory to allocate to memcached. The default is 64MB. That may be enough for small sites, but for larger sites you will likely be using multiple gigabytes. It's hard to recommend a standard size to use as it will vary by a large amount based on the site. If you have a "big" site, I'd say start at 512MB or 1GB; if you have a smaller site you might leave the default, or just bump it to 512MB anyway if you have plenty of RAM on the server. Once it's running, you can watch the memory usage and look for evictions (removal of a cache item once the cache is full) to see if you might want to increase the memory allocation.

On all Fedora / CentOS memcached packages, the configuration file is stored in /etc/sysconfig/memcached. By default, it looks like this:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS=""

To increase the memory allocation, adjust the CACHESIZE setting to the number of MB you want memcached to use.

If you are running memcached locally on your web server (and only have one web server), then I strongly recommend you also add an option for memcached to listen only on your loopback interface (localhost). Whether or not you make that change, please consider locking down the memcached port(s) with a firewall. In order to listen only on the 127.0.0.1 interface, you can change the OPTIONS line to the following:

OPTIONS="-l 127.0.0.1"

See the memcached man page for more info on that or any other settings.

Once you have installed memcached and updated the configuration, you can start it up and configure it to start on boot:

# service memcached start
# chkconfig memcached on

CentOS / RHEL PECL Module Install

Fedora

If you are on Fedora and using PHP from the base repo in the distribution, then installation of the PECL extension is easy. Just use yum to install whichever PECL extension you choose:

# yum install php-pecl-memcache

Or

# yum install php-pecl-memcached

CentOS 5 / RHEL 5

CentOS and RHEL can be a bit more complicated, especially on EL5 which ships with PHP 5.1.x, which is too old for most people. Here are the options I'd suggest for EL5:

  • If you are OK using the PHP provided with EL5, then you can get the PECL extensions from EPEL. Once you've enabled the EPEL repository (instructions), you can install either PECL extension by using the same yum commands outlined above in the Fedora section.
  • If you want to use PHP 5.2 or PHP 5.3 with EL5, I suggest using the IUS repositories (IUS repo instructions). Note that IUS provides the PECL memcache extension, but not the PECL memcached extension. Based on which PHP version you decide to use, you can install the PECL memcache extension with either:

    # yum install php52-pecl-memcache

    Or

    # yum install php53u-pecl-memcache

CentOS 6 / RHEL 6

EL6 ships with PHP 5.3, though it is an older version than is available for EL6 at IUS. If you are using the OS-provided PHP package, then you can install the PECL memcache extension from the base OS repo. If you want the PECL memcached extension, it is not in the base OS repo, but is available in EPEL. See the instructions linked from the CentOS 5 section above if you need to enable the EPEL repo.

# yum install php-pecl-memcache

Or, enable EPEL and then run:

# yum install php-pecl-memcached

As with EL5, some people running EL6 will also want the latest PHP packages and can get them from the IUS repositories. If you are running PHP from IUS under EL6, then you can install the PECL memcache extension with:

# yum install php53u-pecl-memcache

Similar to EL5, the IUS repo for EL6 does not include the PECL memcached module.

PECL Memcache Configuration

If you are using PECL memcache extension and will be using the clustering option of the Drupal Memcache module which utilizes multiple memcached instances, then it is important to set the hash strategy to "consistent" in the memcache extension configuration. Edit /etc/php.d/memcache.ini and set (or un-comment) the following line:

memcache.hash_strategy=consistent

If you are using the PECL memcached module, this configuration is done at the application level (e.g. in your Drupal settings.php).

Once you've installed the PECL memcache (or memcached) extension, you will need to reload httpd in order for PHP to see the new extension. You'll also need to reload httpd whenever you change the memcache.ini configuration file.

# service httpd reload

SELinux

If you have SELinux enabled (you should!), I have an older blog post with instructions on configuring SELinux for Drupal.

That's it, you're now good to go with PHP and memcache!

by jeff at April 26, 2012 06:02 PM

cfengine 3.3.0 packages for Fedora / CentOS / RHEL

As I've used cfengine less and less recently the packages in Fedora and EPEL have been a bit neglected. At one point someone stepped up to update them, but then nothing ever came of it. I've finally updated the packages to the latest upstream version as of this writing (3.3.0) in Fedora 16, Fedora 17, Fedora Devel, and EPEL 6. They should be pushed to the updates-testing repos for each of those releases soon if not already there. There are some package changes since the last 3.x release, so any testing people can do would be appreciated.

I've uploaded EL6 and F17 RPMs here for reference: http://sheltren.com/downloads/cfengine/testing/

Note that these are quite different from the upstream-provided RPMs which simply dump everything in /var/cfengine. The good news here is I've actually provided a source RPM for those that need to tweak the build. Also, I hit some configure errors when attempting to build on EL5 which I haven't worked out yet -- looks like an upstream bug with the configure script to me, so there are no EL5 packages at the moment.

If anyone is willing to co-maintain these in Fedora and/or EPEL with me, please let me know.

by jeff at April 26, 2012 04:06 PM

December 21, 2011

Jeff Sheltren

Stop Disabling SELinux!

I see a lot of people coming by #centos and similar channels asking for help when they’re experiencing a problem with their Linux system. It amazes me how many people describe their problem, and then say something along the lines of, “and I disabled SELinux...”. Most of the time SELinux has nothing to do with the problem, and if SELinux is the cause of the problem, why would you throw out the extra security by disabling it completely rather than configuring it to work with your application? This may have made sense in the Fedora 3 days when selinux settings and tools weren’t quite as fleshed out, but the tools and the default SELinux policy have come a long way since then, and it’s very worthwhile to spend a little time to understand how to configure SELinux instead of reflexively disabling it. In this post, I’m going to describe some useful tools for SELinux and walk through how to configure SELinux to work when setting up a Drupal web site using a local memcached server and a remote MySQL database server -- a pretty common setup for sites which receive a fair amount of traffic.

This is by no means a comprehensive guide to SELinux; there are many of those already!
http://wiki.centos.org/HowTos/SELinux
http://fedoraproject.org/wiki/SELinux/Understanding
http://fedoraproject.org/wiki/SELinux/Troubleshooting

Too Long; Didn’t Read Version

If you’re in a hurry to figure out how to configure SELinux for this particular type of setup, on CentOS 6, you should be able to use the following two commands to get things working with SELinux:
# setsebool -P httpd_can_network_connect_db 1
# setsebool -P httpd_can_network_memcache 1

Note that if you have files existing somewhere on your server and you move them to the webroot rather than untar them there directly, you may end up with SELinux file contexts set incorrectly on them which will likely deny access to apache to read those files. If you are having a related problem, you’ll see something like this in your /var/log/audit/audit.log:
type=AVC msg=audit(1324359816.779:66): avc: denied { getattr } for pid=3872 comm="httpd" path="/var/www/html/index.php" dev=dm-0 ino=549169 scontext=root:system_r:httpd_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file

You can solve this by resetting the webroot to its default file context using the restorecon command:
# restorecon -rv /var/www/html

Server Overview

I’m going to start with a CentOS 6 system configured with SELinux in targeted mode, which is the default configuration. I’m going to be using httpd, memcached, and PHP from the CentOS base repos, though the configuration wouldn’t change if you were to use the IUS PHP packages. MySQL will be running on a remote server which gives improved performance, but means a bit of additional SELinux configuration to allow httpd to talk to a remote MySQL server. I’ll be using Drupal 7 in this example, though this should apply to Drupal 6 as well without any changes.

Initial Setup

Here we will setup some prerequisites for the website. If you already have a website setup you can skip this section.

We will be using tools such as audit2allow which is part of the policycoreutils-python package. I believe this is typically installed by default, but if you did a minimal install you may not have it.
# yum install policycoreutils-python

Install the needed apache httpd, php, and memcached packages:
# yum install php php-pecl-apc php-mbstring php-mysql php-pecl-memcache php-gd php-xml httpd memcached

Startup memcached. The CentOS 6 default configuration for memcached only listens on 127.0.0.1, this is great for our testing purposes. The default of 64M of RAM may not be enough for a production server, but for this test it will be plenty. We’ll just start up the service without changing any configuration values:
# service memcached start

Startup httpd. You may have already configured apache for your needs, if not, the default config should be enough for the site we’ll be testing.
# service httpd start

If you are using a firewall, then you need to allow at least port 80 through so that you can access the website -- I won’t get into that configuration here.

Install Drupal. I’ll be using the latest Drupal 7 version (7.9 as of this writing). Direct link: http://ftp.drupal.org/files/projects/drupal-7.9.tar.gz
Download the tarball, and expand it to the apache web root. I also use the --strip-components=1 argument to strip off the top level directory, otherwise it would expand into /var/www/html/drupal-7.9/
# tar zxf drupal-7.9.tar.gz -C /var/www/html --strip-components=1

Also, we need to get the Drupal site ready for install by creating a settings.php file writable by apache, and also create a default files directory which apache can write to.
# cd /var/www/html/sites/default/
# cp default.settings.php settings.php
# chgrp apache settings.php && chmod 660 settings.php
# install -d -m 775 -g apache files

Setup a database and database user on your MySQL server for Drupal. This would be something like this:
mysql> CREATE DATABASE drupal;
mysql> GRANT ALL ON drupal.* TO drupal_rw@web-server-ip-here IDENTIFIED BY 'somepassword';

Test this out by using the mysql command line tool on the web host.
# mysql -u drupal_rw -p -h drupal

That should connect you to the remote MySQL server. Be sure that is working before you proceed.

Now for the Fun Stuff

If you visit your new Drupal site at http://your-hostname-here, you’ll be presented with the Drupal installation page. Click ahead a few times, setup your DB info on the Database Configuration page -- you need to expand “Advanced Options” to get to the hostname field since it assumes localhost. When you click the button to proceed, you’ll probably get an unexpected error that it can’t connect to your database -- this is SELinux doing its best to protect you!

Allowing httpd to Connect to a Remote Database

So what just happened? We know the database was setup properly to allow access from the remote web host, but Drupal is complaining that it can’t connect. First, you can look in /var/log/audit/audit.log which is where SELinux will log access denials. If you grep for ‘httpd’ in the log, you’ll see something like the following:
# grep httpd /var/log/audit/audit.log
type=AVC msg=audit(1322708342.967:16804): avc: denied { name_connect } for pid=2724 comm="httpd" dest=3306 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket

That is telling you, in SELinux giberish language, that the httpd process was denied access to connect to a remote MySQL port. For a better explanation of the denial and some potential fixes, we can use the ‘audit2why’ utility:
# grep httpd /var/log/audit/audit.log | audit2why
type=AVC msg=audit(1322708342.967:16804): avc: denied { name_connect } for pid=2724 comm="httpd" dest=3306 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket

Was caused by:
One of the following booleans was set incorrectly.
Description:
Allow HTTPD scripts and modules to connect to the network using TCP.

Allow access by executing:
# setsebool -P httpd_can_network_connect 1
Description:
Allow HTTPD scripts and modules to connect to databases over the network.

Allow access by executing:
# setsebool -P httpd_can_network_connect_db 1

audit2why will analyze the denial message you give it and potentially explain ways to correct it if it is something you would like to allow. In this case, there are two built in SELinux boolean settings that could be enabled for this to work. One of them, httpd_can_network_connect, will allow httpd to connect to anything on the network. This might be useful in some cases, but is not very specific. The better option in this case is to enable httpd_can_network_connect_db which limits httpd generated network connections to only database traffic. Run the following command to enable that setting:
# setsebool -P httpd_can_network_connect_db 1

It will take a few seconds and not output anything. Once that completes, go back to the Drupal install page, verify the database connection info, and click on the button to continue. Now it should connect to the database successfully and proceed through the installation. Once it finishes, you can disable apache write access to the settings.php file:
# chmod 640 /var/www/html/sites/default/settings.php

Then fill out the rest of the information to complete the installation.

Allowing httpd to connect to a memcached server

Now we want to setup Drupal to use memcached instead of storing cache information in MySQL. You’ll need to download and install the Drupal memcache module available here: http://drupal.org/project/memcache
Install that into your Drupal installation, and add the appropriate entries into settings.php. For this site, I did that with the following:
# mkdir /var/www/html/sites/default/modules
# tar zxf memcache-7.x-1.0-rc2.tar.gz -C /var/www/html/sites/default/modules

Then edit settings.php and add the following two lines:
$conf['cache_backends'][] = 'sites/default/modules/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';

Now if you reload your site in your web browser, you’ll likely see a bunch of memcache errors -- just what you wanted! I bet it’s SELinux at it again! Check out /var/log/audit/audit.log again and you’ll see something like:
type=AVC msg=audit(1322710172.987:16882): avc: denied { name_connect } for pid=2721 comm="httpd" dest=11211 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket

That’s very similar to the last message, but this one is for a memcache port. What does audit2why have to say?
# grep -m 1 memcache /var/log/audit/audit.log | audit2why
type=AVC msg=audit(1322710172.796:16830): avc: denied { name_connect } for pid=2721 comm="httpd" dest=11211 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket

Was caused by:
One of the following booleans was set incorrectly.
Description:
Allow httpd to act as a relay

Allow access by executing:
# setsebool -P httpd_can_network_relay 1
Description:
Allow httpd to connect to memcache server

Allow access by executing:
# setsebool -P httpd_can_network_memcache 1
Description:
Allow HTTPD scripts and modules to connect to the network using TCP.

Allow access by executing:
# setsebool -P httpd_can_network_connect 1

Again, audit2why gives us a number of options to fix this. The best bet is to go with the smallest and most presice change for our needs. In this case there’s another perfect fit: httpd_can_network_memcache. Enable that boolean with the following command:
# setsebool -P httpd_can_network_memcache 1

Success! Now httpd can talk to memcache. Reload your site a couple of times and you should no longer see any memcache errors. You can be sure that Drupal is caching in memcache by connecting to the memcache CLI (telnet localhost 11211) and typing ‘stats’. You should see some number greater than 0 for ‘get_hits’ and for ‘bytes’.

What are all these booleans anyway?

Now we’ve used a couple SELinux booleans to allow httpd to connect to memcached and MySQL. You can see a full list of booleans which you can control by using the command ‘getsebool -a’. They are basically a preset way for you to allow/deny certain pre-defined access controls.

Restoring default file contexts

As I mentioned briefly in the ‘TL;DR’ section, another common problem people experience is with file contexts. If you follow my instructions exactly, you won’t have this problem because we untar the Drupal files directly into the webroot, so they will inherit the default file context for /var/www/html. If, however, you were to untar the files in your home directory, and then use ‘mv’ or ‘cp’ to place them in /var/www/html, they will maintain the user_home_t context which apache won’t be able to read by default. If this is happening to you, you will see the file denials logged in /var/log/audit/audit.log -- something like this:
type=AVC msg=audit(1324359816.779:66): avc: denied { getattr } for pid=3872 comm="httpd" path="/var/www/html/index.php" dev=dm-0 ino=549169 scontext=root:system_r:httpd_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file

The solution in this case is to use restorecon to reset the file contexts back to normal:
# restorecon -rv /var/www/html

Update: It was noted that I should also mention another tool for debugging audit messages, 'sealert'. This is provided in the setroubleshoot-server package and will also read in the audit log, similar to what I described with audit2why.
# sealert -a /var/log/audit/audit.log

Tags:

by jeff at December 21, 2011 11:36 PM

November 25, 2011

Frédéric Wenzel

Day 329 - Ready for the Sunset

Day 329 - Ready for the Sunset

A family of tourists, getting ready to watch the sun set on the Pacific coast. I love silhouette photos like this: It's fun to see the different characters with their body shapes and postures.

November 25, 2011 08:00 AM

November 08, 2011

Jeff Sheltren

CentOS Continuous Release

The CentOS Continuous Release repository (“CR”) was first introduced for CentOS 5.6, and currently exists for both CentOS 5 and CentOS 6. The CR repo is intended to provide package updates which have been released for the next point release upstream (from RHEL) which has not yet been officially released by CentOS yet due to delays around building, testing, and seeding mirrors for a new point release. For example, this means that once RedHat releases RHEL 5.8, CentOS will include package updates from 5.8 base and updates in CentOS 5.7 CR repo until the time that CentOS is able to complete the release of CentOS 5.8. For admins, this means less time without important security updates and the ability to be on the latest packages released in the latest RHEL point release.

Details on the CR Repo

What’s included in CR and how might it affect your current CentOS installs? At this point, the CR repo is used only for package updates which are part of the next upstream point release. For example, for CentOS 5.7, once Red Hat releases RHEL 5.8, the CR repo will contain updates from upstream base and updates repos. When a new update for RHEL 5.8 is released, it will be built in the CentOS build system, go through a relatively minimal amount of QA by the CentOS QA team, and then will be pushed to the CentOS 5.7 CR repo. This process will continue until the time that CentOS releases its own 5.8 release. Once CentOS releases 5.8, the CR repo will be cleared out until the time that RedHat releases the next (5.9) point release.

The CR repo is not enabled by default, so it is up to a system administrator to enable it if desired. That means, by default, you won’t see packages added to the CR repo. Installing the repo is very easy as it’s now part of the CentOS extras repository which is enabled by default. To enable CR, you simply have to:

yum install centos-release-cr

If you don’t have CentOS Extras enabled, you can browse into the extras/ directory for the release of CentOS you’re currently running and download and install the centos-release-cr package by hand, or manually create a centos-cr.repo in /etc/yum.repos.d/

In my opinion, unless you have an internal process for testing/pushing updates, you should absolutely be using the CR repo. Even if you do have your own local processes for updates, I would consider the CR repo to be part of CentOS updates for all intents and purposes, and pull your updates from there for testing/release. The packages in the CR repo can fix known security issues which without the CR repo you won’t have access to until the next CentOS point release -- and that can sometimes take longer than we’d like!

A New Proposal: Include CR by Default

In a recent post to the CentOS Developers list, Karanbir Singh proposed moving the CR repo into the main release for 6.x. What this would mean is for CentOS 6.x and onward, we would see the base OS and ISO directories be updated for each point release, but in general, updates would be pushed to a central 6/ directory, basically incorporating CR into what is currently considered updates/.

This proposal is different from the current CR setup in that it incorporates CR into the release by default, and puts less reliance on the old point release model. This will help ensure that people are always running the latest security updates as well as take a bit of pressure off of CentOS developers and QA team when trying to build, test, and release the next point release. If the package updates are already released and in use, point releases become less important (though still useful for new installs).

Incorporating CR more into the main release doesn’t mean that point releases will go away completely. They will still include updated base packages and ISO images, typically with installer bug fixes and/or new and updated drivers. In general, I see this as a good move: it means more people will be getting security updates by default instead of waiting during the time lapse between upstream RHEL releases and the time it takes for CentOS to rebuild, test, and release that point release. Having those packages available by default is great, especially for those admins who don’t pay close attention and wouldn’t otherwise enable the CR repo. It should be noted that at this point, the incorporation of CR into the main release is only being discussed for CentOS 6.x onward and won’t change anything in the 5.x releases where people will still need to manually opt-in to the CR packages.

References:
http://wiki.centos.org/AdditionalResources/Repositories/CR
http://lists.centos.org/mailman/listinfo/centos-cr-announce
http://lists.centos.org/pipermail/centos-devel/2011-November/008268.html

Tags:

by jeff at November 08, 2011 04:03 PM

August 14, 2011

Justin Dugger

Solving the Sunday Puzzle with coreutils & grep

Not too long ago, a puzzler admitted to solving the NPR Sunday Puzzle with a computer. Since I hate word puzzles quite a bit, I'd taken similar steps in the past. For example, this recent challenge:

Fill in the 4x4 square crossword with common uncapitalized words, using no letter more than once:

NAGS
E
W
T

My secret weapon? GNU coreutils. Regex are a great tool, but I rarely have to use some of the more obscure features, which hurts on the occasions where they're called for. So the NPR puzzle can be a good way to practice and learn!

Edit: Commenter hggdh points out that the heavy worker here is grep, which is not part of coreutils. If your OS vendor doesn't provide grep, GNU grep sounds like a suitable replacement.

  1. I'm using the American English dictionary provided by Ubuntu /usr/share/dict/words. The format of this file is one word per line. Every form of a word, including contractions and possessives, gets its own line. We use | (pipe) to chain the output of one command as the input of the next. Cat simply outputs a file, and wc -l counts the lines in it.

    laptop:~$ cat /usr/share/dict/words | wc -l

    98569

  2. I assume no apostrophes are in the puzzle. Grep reads input and outputs only those lines that match a regular expression (regex). Using the -v option to grep changes it to output only lines that don't match our pattern.

    laptop:~$ cat /usr/share/dict/words | grep -v "'" | wc -l

    74059

  3. That's a lot of words to fuddle around with, so lets winnow this down. Firstly, we only care about 4 letter words. We can use grep to give us only these words, using the regular expression "^....$". Caret (^) represents the start of a line, and $ represents the end of one. Each period is a single free choice character for grep, matching exactly one character in the input.

    laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^....$" | wc -l

    3174

  4. Having cut the search space by 96 percent, we now turn to the clues for... clues. Fortunately, nags and newts define which letters every word can start with. Grep treats symbols within [] as alternatives, meaning any any one symbol within can match the input. Below alters the regex from 3 to only match words starting with a, g, s, e, w or t.

    laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^[agsewt]...$" | wc -l

    777

  5. Rules say no two letters repeat in the puzzle, so we'll exclude all words with the letters from nags and newts anywhere other than the first letter. As an alternative to -v, we can use carets inside brackets to indicate "not".

    laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^[agsewt][^nagsewt][^nagsewt][^nagsewt]$" | wc -l

    131

  6. Next, we can rule out words with repeated letters, like solo and wool. To do this quickly, we'll need to use backreferences. Backreferences can be slow, but since our dataset is so tiny, it will be fine to add it to the end of the pipeline.

    cat /usr/share/dict/words | grep -v "'" | grep "^[agsewt][^nagsewt][^nagsewt][^nagsewt]$" | grep -vE "([a-z]).*(\1)" | wc -l

    106

  7. Starting to get close! From here on out, this plays a lot like sudoku. Our goal is now to start constructing regex for each word. We replace the leading alternative for a specific letter. To start off, we've only got 7 options for 2 across:

    laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^e[^nagsewt][^nagsewt][^nagsewt]$" | grep -vE "([a-z]).*(\1)"

    echo

    ecru

    emir

    epic

    euro

    evil

    expo

We now write a different regex without negations to get the same list.

`laptop:~$ cat /usr/share/dict/words | grep "^e[cmpuvx][hipr][cloru]$" | grep -vE "([a-z]).*(\1)" | wc -l`

`7`

Now we build a similar regex for 2 down. Adding in what we know about it's intersection with 2 across (cmpuvx) is the sudoku like step:

`laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^a[cmpuvx][^nagsewt][^nagsewt]$" | grep -vE "([a-z]).*(\1)"`

achy

acid

amid

amok

avid

We rewrite this one as

laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^a[cmv][hio][dky]$" | grep -vE "([a-z]).*(\1)" | wc -l

5

Applying the same logic to 3 down yields "^g[ir][lriu][bdlmp]$", and 4 down yields "^s[lu][cilmoru][bdfhkmopr]$".

  1. The last positions in each down regex constructs a new regex for 4 across:

cat /usr/share/dict/words | grep -v "'" | grep "^t[dky][bdlmp][bdfhkmopr]$" | grep -vE "([a-z]).*(\1)"

typo

A unique solution to 4 across!

  1. Revisiting 2 down with this new fact also yields a unique answer. I leave it from here as an exercise to the reader to solve the entire puzzle.

by Justin Dugger at August 14, 2011 12:00 AM

August 09, 2011

GOSCON News

New Speaker Announced: Dr. David A. Wheeler

We've added our final speaker to the GOSCON Cost Take Out Panel: David A. Wheeler. Dr. Wheeler is a Research Staff Member at the Institute for Defense Analyses and is an expert on developing secure software and the use of open source software in the security space. He is the author of several well known works in this space, including Secure Programming for Linux and Unix HOWTO, Why Open Source Software / Free Software (OSS/FS)?, Look at the Numbers!, and How to Evaluate OSS/FS Programs. more...

by Leslie at August 09, 2011 08:54 PM

Wayne Moses Burke

Executive Director
Open Forum Foundation

Mr. Moses Burke will be moderating the Building Outside the Box Panel during GOSCON DC 2011 at the Innovation Nation Forum.more...

by Leslie at August 09, 2011 08:48 PM

Alexander B. Howard

Government 2.0 Correspondent

O’Reilly Media

Mr. Howard will be moderating the Cost Take Out Panel during GOSCON DC 2011 at the Innovation Nation Forum.more...

by Leslie at August 09, 2011 08:43 PM

June 19, 2011

Peter Krenesky

Ganeti Web Manager 0.7

Ganeti Web ManagerWe’ve just release version 0.7 of Ganeti Web Manager. Ganeti Web Manager is a Django based web application that allows administrators and clients access to their ganeti clusters. It includes a permissions and quota system that allows administrators to grant access to both clusters and virtual machines. It also includes user groups for structuring access to organizations.

This is the fourth release of Ganeti Web Manager and it contains numerous new features.  It also includes various bug fixes and speed optimizations.  Here is the full CHANGELOG, or read on for the highlights.

Xen Support

Ganeti Web Manager now have full Xen support.  Prior versions could display Xen instances, but now you can create and edit them too.  This as an important addition because Xen is a widely used and mature project.  Now with full hardware virtualization in Linux 3.0, Xen will continue to be an important technology for virtualization.  This was our most often requested feature and we’re glad to have fulfilled it.

Internationalization

Thanks to a large community contribution, internationalization support was added for nearly all aspects of the interface.  Users can switch between their default language and any other.  Currently only a Greek translation is available, but we’d like to see many more languages. If you can read and write another language this is a great opportunity for you to get involved. We’re using Transifex to coordinate people who want to help translate.

Search & Improved Navigation

Administrators of larger cluster can now find objects easier with our search interface.  It includes an Ajax auto-complete feature, along with detailed results.

We’ve also added contextual links wherever we could.  This included ensuring breadcrumbs were properly formatted on each page.  Object Permissions and Object Log were updated to ensure navigating between those screens and Ganeti Web Manager is seamless.

Import Tools

There are now import tools for Nodes.  These work the same as for instances.  The cache updater has also been reworked to support both Nodes and Instances.  It’s now a twisted plugin with modest speed improvements due to Ganeti requests happening asynchronously.

Speed, Scalability, and Bugs

We’ve sought out places where we performed extra and or inefficient database queries.  We identified numerous places where database interaction could be reduced, and pages returned faster.  This is an ongoing process.  We’ll continue to optimize and improve the responsiveness as we find areas of the project we can improve.

Numerous bugs were fixed in both the user interface and the backend.  Notably, the instance creation interface has had several bugs corrected.

Module Releases

We’re building several modules along with Ganeti Web Manager.  The following projects have new releases coinciding with Ganeti Web Manager 0.7:

Django Object Permissions 1.4

  • improved user selection widget
  • speed improvements

Object Log 0.6

  • our first public release
  • speed, scalability, and flexibility improvements

Twisted VNC Auth Proxy

  • our first public release
  • added support for hixie 07 and latest noVNC version.

Want to learn more?

Lance Albertson and I will be speaking about Ganeti & Ganeti Web Manager at several conferences this summer.  Catch us at the following events:

by peter at June 19, 2011 03:49 AM

May 18, 2011

Peter Krenesky

Google I/O 2011

Google I/O LogoFive OSUOSL co-workers and I recently finished a road trip to Google I/O 2011.  We took two cars on an 11 hour drive through scenic southern Oregon and northern California.  We learned more about Android and other technologies shaping the web.  It was also a great opportunity to spend time with each other outside the office.

Monday night we joined about 30 Google Summer of Code mentors for dinner and drinks hosted by the Google Open Source Programs Office.  We’re always grateful for events that bring together friends old and new.  One developer nervously sat down at our table, professing that he didn’t know anyone.  We might not work on the same project, but we’re all part of the open source community.

The highlight of the conference was the double announcement of Android Open Accessory program and Android @ Home.  Both open up Android to integration with third party devices.  These features coupled with near field communications (NFC) stand to dramatically change how we use our mobiles devices to interact with the world around us.  This is not a new idea.  X10 home automation has existed since 1975.  Zigbee and Z-wave are more modern protocols, but also available for years.  The difference here is 100 million Android users and a half million Arduino hackers.

As Phillip Torrone wrote on the Makezine Blog, “There really isn’t an easier way to get analog sensor data or control a motor easier and faster than with an Arduino — and that’s a biggie, especially if you’re a phone and want to do this.”

It won’t be a short road.  We still have obstacles such as higher costs.  A representative from Lighting Science I spoke to at their I/O booth quoted Android@Home enabled LED lights at $30 per bulb.  Android and Arduino might be the right combination of market penetration, eager hackers, and solid platforms for a more integrated environment.

NFC Sticker

My favorite session was How To NFC.   NFC (near field communication) is similar to RFID except it only works within a few centimeters.  Newer android phones can send and receive NFC messages any time except when the phone is sleeping.  NFC chips can also be embedded in paper, like the stickers that came in our I/O Badges.  An NFC enabled app can share data such as a url, or launch a multiplayer game with your friend.  It makes complex tasks as simple as “touch the phone here”.  Android is even smart enough to launch an app required for an NFC message, or send you to the market to install the app you need.  Only the Nexus-S supports NFC now, but this feature is so compelling that others will support it soon too.

The other technical sessions were very useful too, whether you were interested in Android, Chrome, or other Google technologies.  The speakers were knowledgeable on the subject areas they spoke on.  I attended mostly Android talks, and it was great hearing from the people who wrote the APIs we’re trying to use.  The sessions were all filmed and are worth watching online.

by peter at May 18, 2011 10:46 PM

May 03, 2011

Lance Albertson

Rebalancing Ganeti Clusters

One of the best features of Ganeti is its ability to grow linearly by adding new servers easily. We recently purchased a new server to expand our ever growing production cluster and needed to rebalance cluster. Adding and expanding the cluster consisted of the following steps:

  1. Installing the base OS on the new node
  2. Adding the node to your configuration management of choice and/or installing ganeti
  3. Add the node to the cluster with gnt-node add
  4. Check Ganeti using the verification action
  5. Use htools to rebalance the cluster

For simplicity sake I'll cover the last three steps.

Adding the node

Assuming you're using a secondary network, this is how you would add your node:

gnt-node add -s <secondary ip> newnode

Now lets check and make sure ganeti is happy:

gnt-cluster verify

If all is well, continue on otherwise try and resolve any issue that ganeti is complaining about.

Using htools

Make sure you install ganeti-htools on all your nodes before continuing. It requires haskell so just be aware of that requirement. Lets see what htools wants to do first:

$ hbal -m ganeti.example.org
Loaded 5 nodes, 73 instances
Group size 5 nodes, 73 instances
Selected node group: default
Initial check done: 0 bad nodes, 0 bad instances.
Initial score: 41.00076094
Trying to minimize the CV...
1. openmrs.osuosl.org g1.osuosl.bak:g2.osuosl.bak g5.osuosl.bak:g1.osuosl.bak 38.85990831 a=r:g5.osuosl.bak f
2. stagingvm.drupal.org g3.osuosl.bak:g1.osuosl.bak g5.osuosl.bak:g3.osuosl.bak 36.69303985 a=r:g5.osuosl.bak f
3. scratchvm.drupal.org g2.osuosl.bak:g4.osuosl.bak g5.osuosl.bak:g2.osuosl.bak 34.61266967 a=r:g5.osuosl.bak f

<snip>

28. crisiscommons1.osuosl.org g3.osuosl.bak:g1.osuosl.bak g3.osuosl.bak:g5.osuosl.bak 4.93089388 a=r:g5.osuosl.bak
29. crisiscommons-web.osuosl.org g2.osuosl.bak:g1.osuosl.bak g1.osuosl.bak:g5.osuosl.bak 4.57788814 a=f r:g5.osuosl.bak
30. aqsis2.osuosl.org g1.osuosl.bak:g3.osuosl.bak g1.osuosl.bak:g5.osuosl.bak 4.57312216 a=r:g5.osuosl.bak
Cluster score improved from 41.00076094 to 4.57312216
Solution length=30

I've shortened the actual output for the sake of this blog post. Htools automatically calculates which virtual machines to move and how using the least amount of operations. In most these moves, the VMs may simply be migrated, migrated & secondary storage replaced, or migrated, secondary storage replaced, migrated. In our environment we needed to move 30 VMs around out of the total 70 VMs that are hosted on the cluster.

Now lets see what commands we actually would need to run:

$ hbal -C -m ganeti.example.org

Commands to run to reach the above solution:

echo jobset 1, 1 jobs
echo job 1/1
gnt-instance replace-disks -n g5.osuosl.bak openmrs.osuosl.org
gnt-instance migrate -f openmrs.osuosl.org
echo jobset 2, 1 jobs
echo job 2/1
gnt-instance replace-disks -n g5.osuosl.bak stagingvm.drupal.org
gnt-instance migrate -f stagingvm.drupal.org
echo jobset 3, 1 jobs
echo job 3/1
gnt-instance replace-disks -n g5.osuosl.bak scratchvm.drupal.org
gnt-instance migrate -f scratchvm.drupal.org

<snip\>

echo jobset 28, 1 jobs
echo job 28/1
gnt-instance replace-disks -n g5.osuosl.bak crisiscommons1.osuosl.org
echo jobset 29, 1 jobs
echo job 29/1
gnt-instance migrate -f crisiscommons-web.osuosl.org
gnt-instance replace-disks -n g5.osuosl.bak crisiscommons-web.osuosl.org
echo jobset 30, 1 jobs
echo job 30/1
gnt-instance replace-disks -n g5.osuosl.bak aqsis2.osuosl.org

Here you can see the commands it wants you to execute. Now you can either put these all in a script and run them, split them up, or just run them one by one. In our case I ran them one by one just to be sure we didn't run into any issues. I had a couple of VMs not migration properly but those were exactly fixed. I split this up into a three day migration running ten jobs a day.

The length of time that it takes to move each VM depends on the following factors:

  1. How fast your secondary network is
  2. How busy the nodes are
  3. How fast your disks are

Most of our VMs ranged in size from 10G to 40G in size and on average took around 10-15 minutes to complete each move. Addtionally, make sure you read the man page for hbal to see all the various features and options you can tweak. For example, you could tell hbal to just run all the commands for you which might be handy for automated rebalancing.

Conclusion

Overall the rebalancing of our cluster went without a hitch outside of a few minor issues. Ganeti made it really easy to expand our cluster with minimal to zero downtime for our hosted projects.

by lance at May 03, 2011 05:55 AM

April 25, 2011

Russell Haering

Cast Preview Release

For the last few months I've been working on and off for Cloudkick (now Rackspace) on a project that we are calling Cast. I'm happy to announce that this afternoon we're releasing Cast version 0.1. The source has been on Github all along, but with this release we feel that the project has finally progressed to a point where:

  1. We've implemented the functionality planned for the first iteration.
  2. The afforementioned functionality actually works against the current version of Node.js.
  3. We have a website and documented most of the imporant parts.

Thats Great, So What Is It?

In short, Cast is an open-source deployment and service management system.

At Cloudkick we tend to see users deploying their code in one of three ways:

  1. Services are deployed via a configuration management system such as Puppet or Chef.
  2. Services are deployed by some sort SSH wrapper such as Fabric or Capistrano.
  3. Services are deployed to a "Platform as a Service" such as Heroku.

But none of these are perfect. Respectively:

  1. The high overhead in interacting with configuration management systems is fine when they are managing 'infrastructure' (that is, the systems on which you run your services), but tend to impede a smooth "devops" style workflow with fast iterations and easy deployment and upgrades.
  2. SSH wrappers typically work well enough on small scales, but but they feel like a hack, and don't trivially integrate with in-house systems.
  3. Of all the options, people seem to like these the best. The price speaks for itself - Platforms as a Service (PaaS) are hugely valuable to their users. The problem is that these platforms are closed systems, inflexible and not very "sysadmin friendly". When they go down, you're trapped. When the pricing or terms change, you're trapped. If they don't or can't do what you want, you're trapped.

With this situation in mind, what could we write for our users? An Open Platform (optionally, as a Service).

What Can it Do?

Using Cast you can:

  1. Upload your application to a server.
  2. Create 'instances' of your application. Think 'staging' and 'production'.
  3. Manage (start, stop, restart, etc) services provided by your application.
  4. Deploy new versions of your application.
  5. Do all of this from the command line or from a REST API.

We have a lot more interesting features planned. Hint: think "Cast cluster". But if this sounds like something you're interested in, stay tuned, share your thoughts or consider looking into a job at the new San Francisco Rackspace Office

April 25, 2011 12:00 AM

April 19, 2011

Greg Lund-Chaix

Facebook in Prineville, a slightly different view

On Friday, Facebook’s Senior Open Programs Manager, David Recordon, took a group of us from the OSL on a fantastic behind-the-scenes tour of the new Facebook data center in Prineville, Oregon. It was an amazing experience that prompted me to think about things I haven’t thought about in quite a few years. You see, long before I was ever a server geek I spent my summers and school holidays working as an apprentice in my family’s heating and air conditioning company. As we were walking through the data center looking at the ground-breaking server technology, I found myself thinking about terms and technologies I hadn’t considered much in years – evaporative cooling, plenums, airflow, blowers. The computing technology is fascinating and ground-breaking, but they’ve been covered exhaustively elsewhere. I’d like to spend some time talking about something a bit less sexy but equally important: how Facebook keeps all those servers from melting down from all the heat they generate.

First, though, some scale. They’re still building the data center – only one of the three buildings has been built so far, and it has less than half of its server rooms completed – but even at a fraction of its proposed capacity the data center was reportedly able to handle 100% of Facebook’s US traffic for a while when they tested it last week. The students we brought with us did a bit of back-of-the-envelope calculation: when the facility is fully built out, we suspect it’ll be able to hold on the order of hundreds of thousands of servers. It’s mind-boggling to think how much heat that many servers must generate. It’s hard enough to keep the vastly-smaller OSL data center cool, the idea of scaling it that large is daunting to say the least. As the tour progressed, I found myself more and more fascinated by the airflow and cooling.

The bottom floor of the facility is all data center floor and offices, while the upper floors are essentially giant plenums (the return air directly above the main floor, and the supply above the return). There is no ductwork, just huge holes (10′x10′) in the ceiling of the data center floor bring the cool air down from the “penthouse”, and open ceilings above the “hot” side of the racks to move the hot air out. A lot of the air movement is passive/convective – hot air rises from the hot side of the racks through the ceiling to the second floor and the cooled air drops down from the third floor onto the “cool” side of the server racks, where it’s pulled back though the servers. The air flow is certainly helped along by the fans in the servers and blowers up in the “penthouse”, but it’s clearly designed to take advantage of the fact that hot air rises and cold air sinks. They pull off a bit of the hot air to heat the offices, and split the rest between exhausting it outside and mixing with outside air and recirculating.


(Click to enlarge)

OK, enough with the talking, here are some pictures. Click on the images to enlarge them. Walking through the flow, we start at the “cool” side of the server racks:
  
Notice there are no faceplates to restrict the airflow. The motherboards, power supplies, processor heat sinks, and RAM are all completely exposed.

Then we move on to the “hot” side of the racks:
    
The plastic panels you can see on top of the racks and in the middle image guide the hot air coming out of the servers up through the open ceiling to the floor above. No ductwork needed. There are plastic doors at the ends of the rows to completely seal the hot side from the cold side. It was surprisingly-quiet even here. The fans are larger than standard and low speed. While uncomfortably warm, it was not very loud at all. We could speak normally and be heard easily. Very unlike the almost-deafening roar of a usual data center.

The second “floor” is basically just a big open plenum that connects the exhaust (“hot”) side of the server racks to the top floor in a couple of places (recirculating and/or exhaust, depending on the temperature). It’s a sort of half-floor between the ground floor and the “penthouse” that isn’t walk-able, so we climbed straight up to the top floor – a series of rooms (30′ high and very long) that do several things:

First, outside air is pulled in (the louvers to the right):

The white block/wall on the left is the return air plenum bringing the hot air from the floor below. The louvers above it bring the outside air into the next room.

Mix the outside air with the return air and filter it:

The upper louvers on the right are outside air, lower are return air bringing the hot air up from the servers. The filters (on the left) look like standard disposable air filters. Behind them are much more expensive high-tech filters.

Humidify and cool the air with rows and rows of tiny atomizers (surprisingly little water, and it was weird walking through a building-sized swamp cooler):
    
The left image shows the back of the air filters. The middle image shows the other side of the room with the water jets. The right image is a closer shot of the water jets/atomizers.

Blowers pull the now-cooled air through the sponges (for lack of a better word) in front of the atomizers and pass it on to be sent down to the servers:

They were remarkably quiet. We could easily speak and be heard over them and it was hard to tell how many (if any) were actually running.

Finally the air is dumped back into the data center through giant holes in the floor:
    
The first image shows the back of the blowers (the holes in the floor are to the right). The middle image shows the openings down to the server floor (the blowers are off to the left). The third image is looking down through the opening to the server room floor. The orange devices are smoke detectors.

The last room on the top floor is where the the unused hot return air is exhausted outside:

None of the exhaust fans were actually running, the passive airflow was sufficient without any assistance. The grates in the floor open down to the intermediate floor connecting to the hot side of the racks.

No refrigerant is used at all, just evaporative cooling (and that then only when needed). The only electricity used in the cooling system is for the fans and the water pumps. All of it – the louvers, the water atomizers, and the fans – are automatically controlled to maintain a static temperature/humidity down on the data center floor. When we were there, none of the fans (neither intake nor exhaust) appeared to be running, it was cool enough outside that they were passively exhausting all of the air from the data center and pulling in 100% outside air on the supply. As best I could tell, the only fans that were actually running were the little tiny 12V fans actually mounted on the servers.

This design makes great sense. It’s intuitive – hot air rises, cool air falls – and it obviously efficiently takes advantage of that fact. I kept thinking, “this is so simple! Why haven’t we been doing this all along?”

Share this: Digg del.icio.us Facebook Google Bookmarks Furl Print Reddit Slashdot StumbleUpon Technorati TwitThis Fark LinkedIn Ma.gnolia NewsVine Ping.fm Pownce Tumblr

by Greg at April 19, 2011 09:04 PM

April 17, 2011

Lance Albertson

Facebook Prineville Datacenter

Along with the rest of the OSU Open Source Lab crew (including students), I was invited to the grand opening of Facebook's new datacenter yesterday in Prineville, Oregon. We were lucky enough to get a private tour by Facebook's Senior Open Source Manager, David Recordon. I was very impressed with the facility on many levels.

Triplet racks & UPS

Triplet racks & UPS

I was glad I was able to get a close look at their Open Compute servers and racks in person. They were quite impressive. One triplet rack can hold ninty 1.5U servers which can add up quickly. We're hoping to get one or two of these racks at the OSL. I hope they fit as those triplet racks were rather tall!

Web & memcached servers

Web & memcached servers

Here's a look at a bank of their web & memcached servers. You can find the memcached servers with the large banks of RAM in the front of them (72Gs in each server). The web servers were running the Intel open compute boards while the memcached servers were using AMD. The blue LED's on the servers cost Facebook an extra $0.05 per unit compared to green LED's.

Hot aisle

Hot aisle

The hot aisle is shown here and was amazing quiet. Actually, the whole room was fairly quiet which is strange compared to our datacenter. Its because of the design of the open compute servers and the fact that they are using negative/positive airflow in the whole facility to push cold/hot air.

Generators

Generators

They had a lot of generators behind the building each a size of a bus easily. You can see their substation in the background. Also note the camera in the foreground, they were everywhere not to mention security because of Green Peace.

The whole trip was amazing and was just blown away by the sheer scale. Facebook is planning on building another facility next to this one within the next year. I was really happy that all of the OSL students were able to attend the trip as well as they rarely get a chance to see something like this.

We missed seeing Mark Zuckerburg by minutes unfortunately. We had a three hour drive back and it was around 8:10PM when we left and he showed up at 8:15PM. Damnit!

If you would like to see more of the pictures I took, please check out my album below.

Facebook Prineville Datacenter

Facebook Prineville Datacenter

Thanks David for inviting us!

by lance at April 17, 2011 01:38 AM

March 21, 2011

Greg Lund-Chaix

Google Summer of Code 2011

I am delighted to announce that the OSU Open Source Lab has once again been selected as a mentoring organization for the Google Summer of Code. This year Google has selected 175 organizations in the open source world to act as mentors. We at the OSL are honored to be included in such a prestigious list. The GSoC program is open to college and university students worldwide, providing students with a cash stipend to fund their work over the summer on some great open source projects.

GSoC logo

This year we’re focusing on projects related to Ganeti Web Manager and Django tools (specifically Django Object Permissions and Django Object Log).

Ganeti Web Manager is the web-based virtualization management system that helps power the OSL’s server virtualization infrastructure, including the newly-launched alpha test of the Supercell cluster. We’re looking for student projects that add a number of new features to the project.

Django Object Permissions is an implementation of Object Permissions, a.k.a. row level permissions. Object Permissions allow you to assign a permission to an instance of any Model in a Django project. There are a number of outstanding issues in the project that we’d love to have a student working on this summer.

Django Object Log provides the ability to log user actions on model instances in Django applications. The project is approaching its initial release, so there are multiple opportunities to contribute to the project.

If you’re a college or university student and interested in python, we’d love to see a project proposal from you! Please check out our ideas page for more detailed information.

Share this: Digg del.icio.us Facebook Google Bookmarks Furl Print Reddit Slashdot StumbleUpon Technorati TwitThis Fark LinkedIn Ma.gnolia NewsVine Ping.fm Pownce Tumblr

by Greg at March 21, 2011 11:59 PM

March 16, 2011

Peter Krenesky

PyCon2011: Snakes in a [Mother$#@!in] Brain

I just returned from PyCon 2011, the largest annual gathering of Python users and contributors.  The conference was full of energy and I came home with my head stuffed full of new ideas and Python skills.  Hillary Mason best described my feelings about PyCon in her opening keynote, “I’m glad I’m in a room where list comprehensions receive spontaneous applause”.

The Talks

The talks came in many flavors: hands-on tutorials, sessions, a poster session, and open space discussions.  Topics included dev-ops, deployment, scalability, concurrency, large scale data processing, science, and much much more.  There was a great deal to learn for both novice and experienced programmers alike.  Most sessions taught useful skills like:

But some sessions were just fun, mind blowing examples of what you could do with Python:

It was difficult to choose which talks to see during most time-slots.  There were just too many great topics to choose from, so it’s fortunate that the session videos are already online.  Many thanks to the PyCon team for being so prompt.

The Jobs

It’s an exciting time for Python developers whether you are just entering the workforce, or looking for something new and exciting.  Part of the exhibit hall was dedicated to startups looking for new employees, but every other exhibitor was looking for employees, too.  There is definitely an employer out there to match your individual passions, and I’m glad to know that my students will have many great choices after graduation.

The Hallway Track

This was my first PyCon and did not know many other attendees, so I planned to spend a good deal of time meeting other people.  Among the 1380 attendees walking the halls and attending sessions were Python Core Developers, authors of your favorite libraries, keynote speakers, and even Guido.  While this could seem intimidating, we all came to PyCon to learn from each other and collaborate. Everyone was welcoming and happy to share knowledge and great conversations.

Six days of talking to random people resulted in many awesome “a ha” moments.  Topics spanned programming, technology, science, and art.  The ideas I shared in these talks were as valuable as the formal sessions I attended.  The best part was making so many new friends.  Looking forward to a great PyCon with you all next year in Santa Clara!

by peter at March 16, 2011 05:31 PM

March 13, 2011

Greg Lund-Chaix

DrupalCon Chicago

Whew! What an amazing week! DrupalCon Chicago is – sadly – now past, and we’ve started looking forward to London and Denver. Now that I’ve had time to return home and get some much-needed sleep, it’s time to make note of some observations I’ve made this year.

The Good:

The Drupal community is as energetic (and quirky) as ever.

Whether it’s presenters leaping up onto tabletops (thank you Emma Jane and Morten) or a wedding (congratulations to Melissa and Michael!), DrupalCon has once again proven to be an entertaining and unpredictable experience. The sheer energy of DrupalCon seems to recharge the communal batteries and sends us all home buzzing with ideas and motivation.

Drupal is open for business.

One of the things that really struck me this year was the sheer number of companies present and the volume of money they seem to be moving through the Drupal community. It used to be we could count the big name Drupal development shops on one hand. This year, however, the exhibit hall was packed with more vendors than I’ve ever seen in the five years I’ve been a part of the Drupal community – and almost all of them seemed to be hiring. I was also struck at the good-natured (but very real) rivalry and competition amongst the various firms. Competition is good, and the friendly one-upsmanship I saw between the big players is just the sort of constructive competition that we need for a healthy ecosystem.

I am once again impressed by the civility shown by the members of the Drupal community when we disagree.

Whether it was the debate surrounding “Drupal as a framework or Drupal as a platform” or the “how can we store configuration information” discussion, it’s clear to me that the participants have not forgotten that our similarities are greater than our differences. Arguments tend to be rooted in fact instead of emotion, objective and not personal. As with any community of this size, there are always interpersonal conflicts – but the infighting we often see in other open source communities is not as prevalent. Civility is the norm, and nearly everyone does a good job of remembering that we all want the same thing – a great piece of software supported by a great community. I find that fact very reassuring and take it as a sign that our community is healthy.

The Hallway Track and the BoFs were full of win.

The sessions I attended were all well done and very useful, in a lot of ways the semi-random encounters in the hallways along with the more informal environment of the BoFs were just as valuable. I met some great new people, re-established relationships that had faded since San Francisco, and learned a lot about what my peers were doing. I came home with a fistful of new contacts to follow up with and even more ideas on how I might better serve my users based on what the other attendees are doing.

The Lightbulb Moment returns!

Past DrupalCons have invariably delivered a “Lightbulb Moment” where something suddenly becomes clear and changes how I work with Drupal. This year was no exception. I’d been very loosely keeping an eye on automating deployment of sites and features with tools like AEgir, the Features module, drush make, and profiles since we were all in San Francisco last year. I hadn’t quite seen everything I needed yet, but was definitely interested in seeing where the projects were headed. Dmitri’s session on Wednesday afternoon finally brought it all together. With the addition of the Profiler module – a tool I had not yet had a chance to investigate – the missing pieces fell into place. There’s an excellent chance they’ll completely change how I manage and deploy sites in the future.

The “Meh”:

Chicago is a great city with wonderful food and accommodations, but it’s a bit hard on one’s wallet.

Not all of us are on corporate expense accounts. While it’s relatively cheap to fly into O’Hare, the hotels and restaurants around “Drupal Towers” can put a strain on a budget. It was also surprisingly expensive to get in from the airport for those of us who aren’t familiar with the CTA, and it was a bit of a hike to the nearest ‘L’ station while toting luggage. The DrupalCon site even states that “There is no direct public transportation between either O’Hare or Midway airports and the Sheraton.” While the hotel and its facilities were excellent and the nearby restaurants were very good, it would have been just as good a conference (if a bit less scenic) in a somewhat less expensive neighborhood.

The Field Museum party was well-planned, with one exception.

The location was excellent, the food was tasty (although they ran out of the good beer before I even made it in the door), and the logistics of getting so many people from the Sheraton to the museum were handled magnificently. The only thing that keeps the event from being a roaring success in my book is that the band was probably not the best choice for the location. Putting an amplified band in a space dominated by (quite literally) tons of smooth shiny marble was a poor decision acoustically. Conversation immediately became almost impossible and even though the band seemed like they were quite good, there appeared to be an exodus for the buses as soon as they started playing. When even the 20-somethings were saying it was too loud perhaps the band wasn’t the best fit for the situation.

The Bad:

“Nice Nodes”

One of the vendors this year was giving out t-shirts with the words “nice nodes” emblazoned across the chest. While the vendor claims the sexual connotations were unintentional, I can’t seem to find a way to interpret it as anything but a crude anatomical double entendre. Unfortunately, when I pointed out on Twitter that the shirts were in very poor taste their response was less than satisfying and more than a little bit snarky:

“node” is gender indeterminate. Do you know something we don’t? #drupalcon #nicenodes

When I replied pointing out that “nodes” (plural) across someone’s chest is not ambiguous I received a rather insulting response:

Sorry if you are offended. The t-shirt is simply meant to be silly. Nothing more. An inside joke for those familiar with Drupal

Doubly implying that 1) there must be something wrong with me that I’d be offended (but not sorry that they may have done something offensive?) and 2) I’m not familiar enough with Drupal to know what “nodes” are. Neither are true, and I’m insulted that they responded to my well-intentioned “hey, your shirts are in poor taste” with snark. (Also a bit ironic from an organization that was earlier in the day tweeting about social change and celebrating International Women’s Day.) Given the controversy in past DrupalCons over the card game and Parisian silhouette flamewar, one would think vendors would know better and be more careful. Sadly, apparently not.

The Conclusion:

DrupalCon Chicago rocked.

I caught up with old friends and made some new ones. I laughed. I learned. I didn’t sleep (much). I came home re-evaluating the tools I use to do my job. I had my belief in the greatness of the Drupal community reinforced. Bravo! DrupalCon Chicago team, you did a fantastic job. You set a high bar for your peers in London and Denver. Well-done. I’m looking forward to seeing everyone again next time.

Share this: Digg del.icio.us Facebook Google Bookmarks Furl Print Reddit Slashdot StumbleUpon Technorati TwitThis Fark LinkedIn Ma.gnolia NewsVine Ping.fm Pownce Tumblr

by Greg at March 13, 2011 08:57 AM

March 05, 2011

Peter Krenesky

Ganeti Web Manager 0.6

We’ve released Ganeti Web Manager 0.6.  Ganeti Web Manager is a Django based web application that allows administrators and clients access to their ganeti clusters. It includes a permissions and quota system that allows administrators to grant access to both clusters and virtual machines. It also includes user groups for structuring access to organizations.

This release comes after a short development cycle, with the goal of fixing critical bugs and providing important core features.  Check out the full change log, or read on for some highlights:

Virtual Machines

Ganeti Web Manager 0.6 includes multiple improvements to the virtual machine detail view.  We’ve added the complete list of virtual machine properties.  The layout has been updated to group properties into relevant sections, as well as make it more readable.

The following new controls were added for virtual machines:

  • Edit a virtual machine’s settings.
  • Rename a virtual machine.
  • Migrate a virtual machine to it’s secondary storage.

Ganeti Web Manager 0.6 also features improvements to the virtual machine deployment process.  It now detects and recovers from Ganeti errors better than before.  If a create job fails without ganeti deploying the virtual machine, you can edit the settings and re-submit the job.  All other failures will let you continue to the virtual machine detail view where you can use the provided admin tools to repair the virtual machine.

Nodes

node detail view

Nodes are now cached by Ganeti Web Manager.  This allows views using node data to be displayed faster. We now also provide Node views that allow an admin to issue commands on a node such as migration and changing the node role.  The node detail view also provides information from the perspective of a node including used resources and which virtual machines are deployed on it.

Logs

Ganeti Web Manager 0.6 now provides a log of actions performed on every object.  This will allow admins to see the history of every action taken on a VirtualMachine, Node, and Clusters.  It also shows every action a user account has performed.  The log is intended to aid auditing and troubleshooting.

Logging is provided by the newly branded Django Object Log app.  It is a reusable app that can log generic messages.  Each message can define it’s own rich format, including contextual links to the related objects.  Object Log will be developed in parallel with Ganeti Web Manager and future projects by the OSUOSL.

by peter at March 05, 2011 12:35 AM

February 12, 2011

Russell Haering

Directory-Backed Resources in Node.js

After writing my last post about the flow control techniques we are using in cast I realized that the process of writing the post was probably as valuable to me as the post will be to anyone else (especially since it was on such a well covered topic). It forced me to explicitly state and justify my reasoning, and caused me to rethink a few things in the process. Since then, whenever I've found myself contemplating a design issue I've tried to think of how I would write it up. This post is the logical extension of that plan, that is, an actual writeup of just such an issue.

The Problem

In cast, we maintain very little state in memory between requests. Most state is stored to the filesystem in the form of the directory structures that we use to manage applications, services, etc. As such we have a lot of code dedicated to managing these resources, including the 'Instance' class I mentioned last time (pedants: I know, JavaScript technically has no classes, but thats the pattern we're emulating so thats what I'm going to call them). For a lot of reasons, most of which are outside the scope of this discussion, we've found an Object-Oriented model to be the best way to represent these resources.

There are three main ways that instances of such a class tend to be created:

  1. We create a new resource on the filesystem and return an instace of the class for further operations.
  2. We create a single instance of the class to represent a resource that already exists.
  3. We create a list of class instances to represent the set of all existing resources of this sort on the filesystem.

The problem is that creating a new resource on the filesystem (obviously) requires filesystem operations and when retrieving one or more instances to represent pre-existing resources we almost invariably want to make sure that the resource in question actually exists, again requiring filesystem operations. This means that each of these operations must be done in an asynchronous manner. In short

var foo = new Instance('foo');

is not sufficient to either create a new instance or retrieve one with any assurance that it exists.

The Solution

Once again, the solution itself isn't all that revolutionary: we need factories. While many seem to associate the term with unpleasant memories of Java, factories are pretty standard practice in Node, and as a general matter are a very good thing.

There are two general patterns with factories in Node. The first, and the one that is most common in Node's public APIs, is to return objects that implement the EventEmitter (a sort of general purpose promise - but don't tell anyone I said that) interface, then let the user register callbacks for various events on that object. For example, to create a TCP connection:

var net = require('net');

// Create a connection to google.com on port 80
var goog = net.createConnection(80, 'google.com');

// Wait for the connection to actually open before using it
goog.on('connect', function() {
  goog.setEncoding('utf8');
  goog.on('data', function(chunk) {
    console.log(chunk);
  });
  goog.write('GET /\r\nhost: google.com\r\n\r\n');
});

The other, which is less common in Node itself, is to pass a callback to the factory function, which is called with the instance once it is ready. One case where this pattern still exists is in the Node API is in fs.stat():

var fs = require('fs');

fs.stat('/tmp', function(err, stats) {
  console.log(stats.isDirectory());
});

Note: the pattern of passing a callback directly to a function is quite common in Node, but fs.stat() is the only case I know of where the second argument (ie, the one that isn't 'err') is something other than a primitive or a list of primitives. On reflection that's probably a bad way to distinguish a 'factory', so maybe the above is a bad, example but you get the idea.

This second pattern is more appropriate for our purposes, as the instances will tend to be short lived and would be unlikely to ever emit any other event.

Sticking to this pattern, what we've ended up with is something like our instance API in cast:

// Create instance 'foo' using fooapp version 1.0
deployment.create_instances('foo', 'fooapp', '1.0', function(err, instance) {
  ...
});

// Retrieve instance 'foo'
deployment.get_instance('foo', function(err, instance) {
  ...
}); 

// Retrieve a list of all instances
deployment.list_instances(function(err, instances) {
  ...
});

Factory Factories

I've spent the last week or so reworking our test framework in cast and updating everything to work with Node 0.4.0, but when I get the chance one thing I may try is factoring out all of the common code into a single module. This would include a single class that all such resources could inherit from, which would likely include an 'exists' method to test for existence and (yeah, go ahead and hate me, I'm going to say it anyway) "factory factories" to generate factory methods for retrieving and listing them.

While "factory factories" are commonly cited as a great reason to hate Java, in my opinion JavaScript's 'functions as first-class objects' model makes quite pretty. In this case they could make generation of factory methods for directory-backed resource FooResource look something like this:

...
exports.get_fooresource = dir_resources.get(FooResource);
exprots.get_fooresource = dir_resources.get_list(FooResource);

February 12, 2011 12:00 AM

February 07, 2011

Russell Haering

Flow Control in Node.js

Late last summer I began working on Cloudkick's new deployment tool, cast which is written in Node.js. At the time I had only minimal experience with JavaScript, having occasionally used it for frontend work on personal projects. After a crash course from Bjorn, Cloudkick's in-house JavaScript wizard, I was able to write mostly working code.

The Problem

As others have found however, writing a Node app that is maintainable is a lot different than just writing one that works.

In cast, we frequently perform long sequences of filesystem operations. For example, for each application deployed in cast we maintain a symlink to the 'current' version. Because symlinks can be atomically replaced with a call to rename the 'current' link will always point to the single current version of the application.

The sequence of operations we need to perform to update the 'current' symlink is:

  1. Determine the path to the new version. We have a method that can do this for us, but it hits the filesystem so calls to it are asynchronous.
  2. Make sure the new version exists. Again, we're hitting the filesystem so this happens asynchronously as well.
  3. Create a new symlink pointing to the specified version. Another filesystem operation.
  4. Swap the new symlink into place using 'rename'.

The naive solution comes out looking something like this:

Instance.prototype.activate_version = function(version, callback) {
  var self = this;
  var new_version_link = path.join(self.root, 'new');
  var cur_version_link = path.join(self.root, 'current');

  self.get_version_path(version, function(err, vp) {
    if (err) {
      return callback(err);
    }
    path.exists(vp, function(exists) {
      if (!exists) {
        return callback(new Error('Cannot activate nonexistent version \'' + version + '\''));
      }
      fs.symlink(path.resolve(vp), new_version_link, function(err) {
        if (err) {
          return callback(err);
        }
        fs.rename(new_version_link, current_version_link, callback);
      });
    });
  });
};

This isn't too bad, but its already getting unwieldy after only four operations (other methods perform ten or more such operations). There are a few problems here:

  1. We quickly end up with very deep nesting which pushes all our code increasingly further to the right and limits the space left on each line for code.
  2. Inserting a new operation early in the chain forces us add another level of indentation to every subsequent operation. This causes commits to contain massive diffs which can hide other accidental (or, theoretically, malicious) changes and obfuscates the actual changes.
  3. Its just plain difficult to read.

The Solution

This isn't exactly revolutionary, but the first step is to use one of the many flow control modules available for Node. For cast we've been using async and I've grown quite attached to it, but there are a lot of other nice ones available as well. Using async, our 'activate_version' method now looks like this:

Instance.prototype.activate_version = function(version, callback) {
  var self = this;
  var new_version_path;
  var new_version_link = path.join(self.root, 'new');
  var current_version_link = path.join(self.root, 'current');

  async.series([
    // Get the path to the specified version
    function(callback) {
      self.get_version_path(version, function(err, vp) {
        new_version_path = vp;
        return callback(err);
      });
    },

    // Make sure the version exists
    function(callback) {
      path.exists(new_version_path, function(exists) {
        var err = null;
        if (!exists) {
          err = new Error('Cannot activate nonexistent version \'' + version + '\'');
        }
        return callback(err);
      });
    },

    // Create the new link
    function(callback) {
      fs.symlink(path.resolve(new_version_path), new_version_link, callback);
    },

    // Atomically move it into place
    async.apply(fs.rename, new_version_link, current_version_link)
  ], callback);
};

I don't want to turn this into an async tutorial, so check out the documentation (or code) if you're curious about the specifics, but in short we get a number of benefits here:

  1. We describe our list of actions by providing a list of functions: its very intuitive.
  2. We avoid both of the problems I mentioned that result from nesting
  3. We provide a single final callback that is always fired, allowing centralized error handling, logging, etc. Alternatively, if the signatures match, we can pass the method's callback argument to async.
  4. We can use async's 'parallel', 'waterfall' or several other methods to change the behavior without significantly altering the pattern.

Once you're using some sort of flow control library, the some things to keep in mind that keep things manageable:

  1. Don't chain multiple asynchronous calls within one function passed to async. If two calls are part of the same logical operation then chances are you should make that operation its own asynchronous function.
  2. If you have a group of logically related calls that could be parallelized, split them into their own function and use async.parallel() (or some equivalent).
  3. If you need to be able to undo your operations in reverse order, you can build a stack of reverse operations as you go, then execute it with async in the final callback.

February 07, 2011 12:00 AM