froztbyte.getBlog() » hacks

Old tricks, new coins; same problems

froztbyte — Tue, 02 Oct 2012 19:34:33 +0000

In setting up some mirrors recently, I’ve come to learn that rsync’s algorithm by default doesn’t deal well with long-haul TCP going hand in hand with small files. I still need to set aside some time to find a nice set of optimization flags to tweak that up a bit more. And when I say “doesn’t deal well”, I mean in the region of “can tx about 1/3rd a single-session TCP speedtest can do over the same path”. Later’s worry, though.

So, the point of this point:

receiver$ nc -l 1234 | pv | tar xf

sender$ tar cf - moz | pv | nc receiver.domain.tld 1234

And there you have a minimal-CPU-usage streaming file delivery setup. Of course, this doesn’t deal with retransmits or anything else. This just gets the bulk over. After that you can run an rsync with big block lengths over all of it, and get any files fixed using that.

And the “old trick” part of this? As far as I know, this is pretty much how the whole tape drive thing used to work[0]. I’ve just added some internet in the middle.

[0] – I wouldn’t know for certain, before my time

HP iLO(2) tapdance

froztbyte — Tue, 10 Jul 2012 22:28:40 +0000

Most people who run bigger sorts of servers are probably familiar with OOB management systems, but for those who aren’t here’s a short summary: you pay a little bit more when you buy your server, and you get a fantastical tool (vendors, please, get this stuff to fit the modern age. It’s not like we don’t want to use them) to use with your server. Power control, hardware status info, (usually) full IP KVM, etc. HP, Dell, Supermicro, Cisco UCS all have this in their own respective flavours.

That’s just to set the tone for what follows. So let’s pretend you live in .za, and you have crappy upstream bandwidth from your home. This would make things like firing up the HP SmartStart ISO on your hardware pretty painful, because uploading all that data takes forever. So what do we do?

We download it to another box on the same network and load up the image via a “hidden” section of iLO that allows us to mount images from an HTTP source, of course:

hpiLO-> show
status=0
status_tag=COMMAND COMPLETED

/
  Targets
    system1
    map1
  Properties
  Verbs
    cd version exit show

hpiLO-> cd /map1/oemhp_vm/cddr
status=0
status_tag=COMMAND COMPLETED

/map1/oemhp_vm/cddr

hpiLO-> show
status=0
status_tag=COMMAND COMPLETED

/map1/oemhp_vm/cddr
  Targets
  Properties
    oemhp_image=None
    oemhp_connect=No
    oemhp_boot=No_Boot
    oemhp_wp=No
    oemhp_applet_connected=No
  Verbs
    cd version exit show

hpiLO-> set oemhp_image=http://192.0.2.1/helpstuff/
status=0
status_tag=COMMAND COMPLETED

hpiLO-> set oemhp_boot=Connect
status=0
status_tag=COMMAND COMPLETED

hpiLO-> show
status=0
status_tag=COMMAND COMPLETED

/map1/oemhp_vm/cddr
  Targets
  Properties
    oemhp_image=http://192.0.2.1/helpstuff/
    oemhp_connect=Yes
    oemhp_boot=Always
    oemhp_wp=Yes
    oemhp_applet_connected=No
  Verbs
    cd version exit show

So, in summary:

We cd to the path that contains cddr (which is the virtual disc path). A note on this, the vm path might sometimes be oemhp_vm1. Do a show under /map if you can’t find the thing.
Then we set oemhp_image and oemhp_boot to values useful for booting.
Now we reboot.

After you’re done with stuff, just set oemhp_boot to Never, and it’ll disconnect stuff.

I didn’t check whether this worked for iLO3 as well, but I’d guess it’s relatively similar. Been a few months since I even looked at an iLO3 system. Here’s the command ref doc for iLO2 if you want to dig around for some more cool stuff.

Zenoss – Find transforms

froztbyte — Thu, 25 Aug 2011 11:10:33 +0000

So I was looking around in one of my zenoss installs some time ago to find what EventClasses I’d set up transforms in, but didn’t feel like digging around through the entire tree of EventClasses (a cursory check now reveals that there’s 136 of them in my one installation). At the time, I solved the problem, extracted the data I needed, and then consequently forgot about it.

And then today I needed that info again. \o/ for IRC logs. To do this, connect to the dmd (on my system, which is installed with the debian package, the command for this is su -c “/usr/local/zenoss/zenoss/bin/zendmd” zenoss. Adjust it for your own system), and then run the following code

foo = dmd.Events.getSubEventClasses()
for i in foo:
    if len(i.transform) != 0: print "%s :: \n%s\n\n" % (i.getOrganizerName(), i.transform)

This will give you human-readable list of all your existing transforms, which makes it easy to find and re-use them.

Edit: this is confirmed working on 3.2.1 (and probably works on the rest of 3.x as well, post in the comments if it doesn’t). Thanks to jmp242 from #zenoss for testing.

DNS Platform Migration Fun

froztbyte — Sat, 06 Aug 2011 00:14:17 +0000

This post could go by the alternative title “Screw you, ISC, and thanks for making software that makes me hate DNS even more”. So let’s dive right in, shall we?

(to those who don’t care for the intermediate ranting and DNS explanations, page down for the tech bits)

There are various criticisms of the Domain Name System — the thing which enables anything on the internet to turn “www.google.com” or any other such name into something that is meaningful to a computer (see here) — but for the most part it works reasonably well. You set up some DNS software, perhaps battle with the config for a while, and then it works. But as a quote I’ve seen somewhere (and can’t find the origin of now with a quick search) says, “you can’t truly recommend some software [tool] until you can tell me why it sucks.” And ISC’s BIND is arguably a highly irritating piece of software, which has over the years led to a rise in popularity for various other options. Amongst these you’ll find some general free/opensource implementations, as well as some commercial platforms:

(That’s the nice thing about diversity and openness — in this regard, an open protocol — you always get some choice and you can pick which one best suits your needs.)

Some years ago, long before my time at my current employer, there was a business requirement for some DNS support in our product suite. And BIND was chosen as the platform, since it’s a fairly well-known one. As time progresses, so do the things we do, and one day we found BIND was no longer sufficient to do what we needed to. Amongst others, things like a supermaster (a master from which a slave will accept all domain information, regardless of whether that slave knows of such a domain) and dynamic backend functionality were some of those needs.
Now some options like bind-dlz and friends existed, but none of these really suited us. In the end we decided upon PowerDNS with our own custom software written to handle the dynamic things as business rules would require, and set forth on this path. Some time passes with Rossi writing all the backend code which we’ve then successfully been running in combination with PowerDNS for some time now.

Of course, we still have all those old BIND-based installations to get upgraded, and this is that tale. Thankfully, the latest version of our platform was designed with exactly this sort of scenario in mind, since we have to inter-operate with other AXFR-speaking nameservers. So I think “let’s just use the config interface to add the migration host as a second slave, massage the data as required on there and then port that data over to the new platform” even as a tiny voice in my head says “it’s never that simple and you know it.” About 2 hours later I’m found at my desk swearing violently about all manner of things, which is my out when dealing with frustrating software. This is because I’d ended up trying to find out why BIND wasn’t actually slaving anything to my “new” nameserver, even though all the configs and zonefiles were right. Not just that, it had also at some point stopped slaving everything it should to the secondary nameserver, which at this point isn’t a worry since I’m replacing it anyway.

:: TECH ::

After figuring out the bits of the migration that matter — such as fixing up the SQL output (from the handy

zone2sql

tool from pdns) that had some oddities due to what looked like multiple $ORIGIN statements in one file — had been figured out, it was pretty painless to move. There were some fun points, like handling multiple $INCLUDE statements in a zonefile, and *hattip* to Jonathan Hitchcock (for the pre- and post-insert idea) and Bryn Divey (for googling better than I).

So, sed trick 1, splitting the file into parts:

1
2
3

cat foo.zone | sed -n '1,/match/p' > firstbit
cat foo.zone | sed -n '1,/match/!p' > secondbit
cat firstbit secondbit > newfoo

Sed trick 2, reading in an external file to use it as the replacement text. We have:

1
2
3

# grep INCLUDE 10_in-addr_arpa.zone
$INCLUDE "/var/cache/bind/10_in-addr_arpa.zone.ns";
$INCLUDE "/var/cache/bind/10_in-addr_arpa.zone.mx";

We do:

1
2
3

sed -i '/$INCLUDE.*\.ns.*$/ r 10_in-addr_arpa.zone.ns' 10_in-addr_arpa.zone
sed -i '/$INCLUDE.*\.mx.*$/ r 10_in-addr_arpa.zone.mx' 10_in-addr_arpa.zone

And tada, instant awesome. This reads the 10_in-addr_arpa.zone.mx file for us, and replaces from the appropriate “$INCLUDE” start to end with the contents of said file.

Another issue I ran into was having the generate the appropriate reverse-entry zones for all the public IP netblocks, and with two /21s and a /18 to worry about I wasn’t planning to do myself if I could help it, so I employed a quick hack with ipcalc and dnspython to transform my /18 into its various component /24s, and then generate reverses:

1
2
3
4
5

ipcalc 11.22.33.0/18 /24 | grep 'Network.*/24' | awk '{print $2}' | cut -d"/" -f 1
11.22.0.0
11.22.1.0
11.22.2.0
...

We can then easily manipulate these in python or sed or cut, depending on how hacky we feel, but I went with python since I was already using MySQLdb to insert the records after massaging them into the right form.

1
2
3
4

>>> import dns.reversename
>>> range = "10.22.0.0"
>>> print dns.reversename.from_address(range).to_text().split(".",1)[1]
0.22.10.in-addr.arpa.

And that’s it for somewhat useful little tricks. There was a bit of a discussion had about delimited formats like this, and Piet Delport (see blogroll) hacked up a neat little delimited datatype which you can find over here. Quick usage instructions:

1
2
3
4
5
6

>>> d = delimited('foo.bar.baz', '.'); d.sort(); print d
bar.baz.foo
d[1:] -> 'bar.baz'
d[1:2] = ['x', 'y', 'z']; d -> 'foo.x.y.z.baz'
>>> d = delimited('0.2.1.10.in-addr.arpa', '.'); del d[0]; print d
2.1.10.in-addr.arpa.

And now as the sounds of Mogwai, Flunk and Placebo massage my tired noggin, it’s time for me to go to bed.