froztbyte.getBlog() » python

“Tip me that output quickly, would you?”

froztbyte — Fri, 02 Nov 2012 11:43:32 +0000

Jonathan recently wrote Codetip, a pastebin based on Twisted.Web with some neat client-side features and integrating server-side JS as well.

There’s a demo one running at http://pb.vuze.la/

Short of the setup doc in the readme, there’s some further work you get to do when installing this on something like Debian Squeeze. Here’s a quick rundown from my shell history:

 1237  aptitude -t wheezy install python-pip
 1242  aptitude -t wheezy install virtualenvwrapper
 1252  aptitude -t wheezy install python-virtualenv
 1257  aptitude -t wheezy install python-sqlite python-pysqlite2

All of those mostly because of versioning. Node.js was also installed previously (for Brewer.js), so npm was around as well, at the sid package version level.

It was set up in a separate user account, with a virtualenv to avoid crapping all over the system. For those who don’t know, a virtualenv is used to make a little jail for python things to get installed into. This way you can have a different local version of something to what’s installed globally (globally meaning system level). Turns out this makes life a bit more painful:

Drop this into .zshenv (or the appropriate equivalent for your shell):

# virtualenv
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--system-site-packages'
export VIRTUALENVWRAPPER_LOG_DIR=$WORKON_HOME
export PIP_VIRTUALENV_BASE=$WORKON_HOME
export PIP_RESPECT_VIRTUALENV=true

And then run:

git clone https://github.com/jonathanj/Codetip.git
mkvirtualenv Codetip
activate Codetip
pip install Twisted; pip install Epsilon; pip install Axiom; ln -s =nodejs $(VIRTUAL_ENV)/bin/node
npm install brewer
cd Codetip
./node_modules/brewer/bin/brake install
./node_modules/brewer/bin/brake all
twistd -no-web --notracebacks --class=Codetip.resource.RootResource

The virtualenvwrapper package provides ‘mkvirtualenv’ amongst other things, use `source /etc/bash_completion.d/virtualenvwrapper` to get the hooks. This shellscript also works in zsh.

And that’s it, a working Codetip instance.

Should I buy some stock?

froztbyte — Tue, 07 Aug 2012 15:03:21 +0000

thoughts: I still have some stock account balance left..wonder if I should buy some..

In [1]: from random import choice
In [2]: choices = {'buy': 0, 'wait': 0}
In [3]: for i in range(0,5000):
    choices[choice(['buy','wait'])] += 1
   ...:
In [4]: choices
Out[4]: {'buy': 2518, 'wait': 2482}

Guess I’ll buy some stocks.

[ed's note: this method works equally well to decide which stocks you want to buy when you're lazy]

Zenoss – Find transforms

froztbyte — Thu, 25 Aug 2011 11:10:33 +0000

So I was looking around in one of my zenoss installs some time ago to find what EventClasses I’d set up transforms in, but didn’t feel like digging around through the entire tree of EventClasses (a cursory check now reveals that there’s 136 of them in my one installation). At the time, I solved the problem, extracted the data I needed, and then consequently forgot about it.

And then today I needed that info again. \o/ for IRC logs. To do this, connect to the dmd (on my system, which is installed with the debian package, the command for this is su -c “/usr/local/zenoss/zenoss/bin/zendmd” zenoss. Adjust it for your own system), and then run the following code

foo = dmd.Events.getSubEventClasses()
for i in foo:
    if len(i.transform) != 0: print "%s :: \n%s\n\n" % (i.getOrganizerName(), i.transform)

This will give you human-readable list of all your existing transforms, which makes it easy to find and re-use them.

Edit: this is confirmed working on 3.2.1 (and probably works on the rest of 3.x as well, post in the comments if it doesn’t). Thanks to jmp242 from #zenoss for testing.

DNS Platform Migration Fun

froztbyte — Sat, 06 Aug 2011 00:14:17 +0000

This post could go by the alternative title “Screw you, ISC, and thanks for making software that makes me hate DNS even more”. So let’s dive right in, shall we?

(to those who don’t care for the intermediate ranting and DNS explanations, page down for the tech bits)

There are various criticisms of the Domain Name System — the thing which enables anything on the internet to turn “www.google.com” or any other such name into something that is meaningful to a computer (see here) — but for the most part it works reasonably well. You set up some DNS software, perhaps battle with the config for a while, and then it works. But as a quote I’ve seen somewhere (and can’t find the origin of now with a quick search) says, “you can’t truly recommend some software [tool] until you can tell me why it sucks.” And ISC’s BIND is arguably a highly irritating piece of software, which has over the years led to a rise in popularity for various other options. Amongst these you’ll find some general free/opensource implementations, as well as some commercial platforms:

(That’s the nice thing about diversity and openness — in this regard, an open protocol — you always get some choice and you can pick which one best suits your needs.)

Some years ago, long before my time at my current employer, there was a business requirement for some DNS support in our product suite. And BIND was chosen as the platform, since it’s a fairly well-known one. As time progresses, so do the things we do, and one day we found BIND was no longer sufficient to do what we needed to. Amongst others, things like a supermaster (a master from which a slave will accept all domain information, regardless of whether that slave knows of such a domain) and dynamic backend functionality were some of those needs.
Now some options like bind-dlz and friends existed, but none of these really suited us. In the end we decided upon PowerDNS with our own custom software written to handle the dynamic things as business rules would require, and set forth on this path. Some time passes with Rossi writing all the backend code which we’ve then successfully been running in combination with PowerDNS for some time now.

Of course, we still have all those old BIND-based installations to get upgraded, and this is that tale. Thankfully, the latest version of our platform was designed with exactly this sort of scenario in mind, since we have to inter-operate with other AXFR-speaking nameservers. So I think “let’s just use the config interface to add the migration host as a second slave, massage the data as required on there and then port that data over to the new platform” even as a tiny voice in my head says “it’s never that simple and you know it.” About 2 hours later I’m found at my desk swearing violently about all manner of things, which is my out when dealing with frustrating software. This is because I’d ended up trying to find out why BIND wasn’t actually slaving anything to my “new” nameserver, even though all the configs and zonefiles were right. Not just that, it had also at some point stopped slaving everything it should to the secondary nameserver, which at this point isn’t a worry since I’m replacing it anyway.

:: TECH ::

After figuring out the bits of the migration that matter — such as fixing up the SQL output (from the handy

zone2sql

tool from pdns) that had some oddities due to what looked like multiple $ORIGIN statements in one file — had been figured out, it was pretty painless to move. There were some fun points, like handling multiple $INCLUDE statements in a zonefile, and *hattip* to Jonathan Hitchcock (for the pre- and post-insert idea) and Bryn Divey (for googling better than I).

So, sed trick 1, splitting the file into parts:

1
2
3

cat foo.zone | sed -n '1,/match/p' > firstbit
cat foo.zone | sed -n '1,/match/!p' > secondbit
cat firstbit secondbit > newfoo

Sed trick 2, reading in an external file to use it as the replacement text. We have:

1
2
3

# grep INCLUDE 10_in-addr_arpa.zone
$INCLUDE "/var/cache/bind/10_in-addr_arpa.zone.ns";
$INCLUDE "/var/cache/bind/10_in-addr_arpa.zone.mx";

We do:

1
2
3

sed -i '/$INCLUDE.*\.ns.*$/ r 10_in-addr_arpa.zone.ns' 10_in-addr_arpa.zone
sed -i '/$INCLUDE.*\.mx.*$/ r 10_in-addr_arpa.zone.mx' 10_in-addr_arpa.zone

And tada, instant awesome. This reads the 10_in-addr_arpa.zone.mx file for us, and replaces from the appropriate “$INCLUDE” start to end with the contents of said file.

Another issue I ran into was having the generate the appropriate reverse-entry zones for all the public IP netblocks, and with two /21s and a /18 to worry about I wasn’t planning to do myself if I could help it, so I employed a quick hack with ipcalc and dnspython to transform my /18 into its various component /24s, and then generate reverses:

1
2
3
4
5

ipcalc 11.22.33.0/18 /24 | grep 'Network.*/24' | awk '{print $2}' | cut -d"/" -f 1
11.22.0.0
11.22.1.0
11.22.2.0
...

We can then easily manipulate these in python or sed or cut, depending on how hacky we feel, but I went with python since I was already using MySQLdb to insert the records after massaging them into the right form.

1
2
3
4

>>> import dns.reversename
>>> range = "10.22.0.0"
>>> print dns.reversename.from_address(range).to_text().split(".",1)[1]
0.22.10.in-addr.arpa.

And that’s it for somewhat useful little tricks. There was a bit of a discussion had about delimited formats like this, and Piet Delport (see blogroll) hacked up a neat little delimited datatype which you can find over here. Quick usage instructions:

1
2
3
4
5
6

>>> d = delimited('foo.bar.baz', '.'); d.sort(); print d
bar.baz.foo
d[1:] -> 'bar.baz'
d[1:2] = ['x', 'y', 'z']; d -> 'foo.x.y.z.baz'
>>> d = delimited('0.2.1.10.in-addr.arpa', '.'); del d[0]; print d
2.1.10.in-addr.arpa.

And now as the sounds of Mogwai, Flunk and Placebo massage my tired noggin, it’s time for me to go to bed.