DHCP, LXC, phy-less (?) bridges, and checksums

tl;dr: if your lxc container on a bridged/veth network is randomly failing to get a dhcp address, you can probably fix it with `ethtool br0 x off  tx off` (br0 being the bridge interface).

With my home DSL acting up quite a bit lately (“lately” == month and a half now. “acting up” == ground fault, and waiting for the telco to fix it..), I’ve been doing a lot of sandboxing work on my microserver at home. But because of its resource scarcity (2GB RAM, and I just haven’t bought more yet), I’ve been giving LXC a go[0] (where I’d normally just do libvirt’d kvm). It’s pretty easy to get started (check this post for pretty much info you need), but I did deviate from the norm slightly.

I like my eth interfaces a server like this to be non-bridged, mostly because of a lack of ipmi (but also being brctl is a clown sometimes). So my setup for this at home looks like so:

auto lxc0
iface lxc0 inet static
  address 192.168.2.1
  broadcast 192.168.2.255
  netmask 255.255.255.0
  bridge_stp on
  pre-up /sbin/brctl addbr lxc0
  post-down /sbin/brctl delbr lxc0

I’ve also got dnsmasq listening on everything, although I hadn’t had it doing dhcp on that interface yet. Today I decided to change that, which is what led me to discovering this: (afaict) if you don’t have a phy interface attached to your bridge, checksum offloading behaviour on that bridge appears to be fucked-by-default.

How this manifested in my case was that lxc containers couldn’t succesfully DHCP (mostly silent failure), but doing a pcap or dhcpdump on the inside interface would show responses actually getting to your container. After some various derpery with dhclient’s options and applying enough patience, I finally managed to see a message: ’5 bad udp checksums in 5 packets’. Some quick searching revealed people advising doing `iptables -A POSTROUTING -t mangle -p udp –dport bootpc -j CHECKSUM –checksum-fill` to fix this. Not just liking magic patches, I did check into why this is the case, and as mentioned earlier, it seems that this happens when you don’t have a phy attached to the bridge[0].

My network config for that interface now has a `post-up /sbin/ethtool tor0 rx off tx off` in it, and things seem dandy.

[0] – Mostly works pretty well. On debian wheezy you need some backported stuff for the various cgroups support. I think I got them from sid.
[1] – “seems”, because a) I don’t feel like testing this by attach a real phy to the bridge now, and b) I haven’t run into this before when using various other things (KVM usually) that were running on phy-attached bridges so I can only guess this is what happens. Feel free to test and let me know.

puppetvpn

Just a quick post to announce puppetvpn. It’s a puppet module for easily setting up openvpn links (p2p and hub-and-spoke), with the main point/attraction/difference being that it lets puppet take care of the key management effort.

Based on initial work by Tristan, so all credit as due goes there.

It’s not very smart right now (presumes a whole bunch of things). Patches accepted.

The worst day I’ve had in a while

Otherwise known as the narrative of 15 Feb 2014. I don’t really know who’ll give a crap, but I just need to get this out of my system.

Having lived in South Africa all my life (thus far), the thing about crime has often come up. And, for the most part, I’ve been pretty damned fortunate to not have had to deal with it at all. Until last night, that is. Which is the worst time it could possibly have hit me at. I’ve been in a bit of a dark spell the last couple of weeks, due to a variety of factors, but things were finally starting to feel a bit better last night. I actually went as far as to comment to someone “I think I’m finally through it all!” around 10~11pm. But hey…apparently it wasn’t quite time yet.

Around 2am (which I guess is technically Sunday?), I left a friend’s house, going homewards via Andiccio 24 (which is a regular habit when I’ve been up all night). I’d been sick since last weekend, too, and was feeling pretty damn drained by this point. After getting my pizza and starting to drive home, somewhere along the route my scooter just failed on me. I don’t know why yet, haven’t exactly had the headspace to look. Sounds electrical-ish, but w/e. And to start the bad, my pizza had fallen onto the ground. This annoyed me, but was just a slight taste of what was to come. At this point, I phone the AA, who give me an estimate of 60 to 90 minutes as pickup time. Now I only live a couple of blocks from where I broke down (albeit uphill), so while I wait for their confirmatory call I start trying to push it as far as I can go. Which isn’t very far, because I’m exhausted as hell. As I got a little bit up the one hill, a white car with two occupants comes driving past, as best as I can tell seeming to be some kind of neighbourhood patrol. And as I mention my difficulties and issues there, the guy claims that he might be able to help me with getting the AA to hurry up. And, hell, I’m tired (had been awake since 05h40 on the Saturday), hungry, stuck in the middle of uselessville at 3am, and I decide to trust the guy. Why not? Not like I ever really do trust strangers. I can’t tell you why I did it.

But I phone the AA up, mention to the callcentre person that this other individual wants to speak to them, and hand my phone over to the guy in the car. After they speak for about 10~15 seconds, I hear the guy mention that he’s with SAPS (which I think is probably illegal to impersonate), and at this point they start driving off, go around the corner, and gone. Now I’ve got no comms. No ETA on getting home. No ability to confirm the pickup. I’m tired, I’m hungry, I’m stressed, and I snap. The last few weeks’ worth of bad just hit me like a ton of bricks dislodged from a cargo carrier, and I am /fucked/. I try to ring a few doorbells, don’t really get anywhere. Finally I see an actual Beagle Security (local patrol crowd) driving past, and after I literally begged the driver, I managed to get home. From where I then started trying to get Prey and such installed on my phone. Which doesn’t help, you need to do it in advance. So I try android remote wipe. Nope, that needs to be set up in advance too. And just a couple of days ago I’d turned off location reporting on my tablet, and forgotten to check if it was account wide. So I couldn’t trace the handset either.

I start pulling out whatever tricks I can try manage. I even try to get my FNB Connect voice thing back up. But I couldn’t find the details saved, and thus couldn’t get any calls done. I need to phone the police. I didn’t even have a plate number for the car. But fuckit, what do you do instead? I post on facebook, asking for help, which is something I do so rarely I don’t even know how to do it. It’s at this point that I ask around, see if anyone has some voip I could use. It’s at this point where some guys over at sonic.net helped me out, and I am extremely thankful for that. The teams who run that place I could solidly count as A+ people. Technically, and for reasons such as this. I manage to phone the police, I get as far as I can manage. I start changing account passwords, dissociating things from my phone, making it as useless as I could possibly hope (ie. no posts showing up when notifications come through, etc). It didn’t have much battery life left, either, so it would’ve died on the people soon afterwards anyway.

But at this point I’ve hit a low. Being without comms is a major hit, for multiple reasons, and being unable to contact (because of this) the one or two people who could’ve actually helped me through it…well, yeah. I hit a low I haven’t hit about since the time I realized I need to cut myself off from my family in totality. I’m not ashamed to say that I cried a bit. I was feeling so utterly helpless in the face of everything, and I just didn’t know what to do.

Positive things! A couple:

  • thank you very much to the person who helped me with the ability to call (I’m finding out if I can name them here)
  • thanks to Tristan for helping me get around in the morning, breakfast (to deal with the fact that I’m starved and getting a migraine), helping get to police, and sitting patiently through multiple stages of trying to get my handset blacklisted (the IMEI shown on the network HLR doesn’t match the IMEI on the box I have)
  • there were a fair handful of people who responded to my FB post (each of you have been noted, and I will make sure to pay you back whenever I can one day)
  • thank you to the people on Shadowfire who let me rage and rant and vent
  • fortunately I had handset insurance from my provider, so this should not make much of a knock on my finances (which was a related fear)

Speculation: maybe…just maaaaybe…the guys in the car were legit. As I said, the battery would’ve died soon. But I don’t feel so. I ran down to the traffic light after they bolted, and I couldn’t see the car by the time I’d gotten there (about 40~50m down the road).

And now it’s Sunday evening. It took a day to get some stuff sorted (prepaid SIM, ability to phone, arranged transport to get to my Monday meeting), and I had the maximum indicated dose of Migril to try counteract my migraine. I still haven’t eaten properly, and don’t feel I can easily. re:hidrat it is, then, for getting myself feeling a slight bit better. My cat is sleeping next to me (she annexed a whole pillow on my couch), there’s good music playing, and I’m clutching my way back to feeling better. Slowly, bit by bit. But this one will hurt for a while.

edit: I know many of these things sound tiny. With perspective, they are. But they added up in just the right set of flavours. Here’s to getting some perspective again, though!

edit 2: did eventually get the scooter back home. The original call with AA had been closed, apparently the guys claimed that they’d done a pickup. I hope to open some kind of case and maybe get my hands on that recording. Or something. I don’t know. But did get home, and after a minor freakout for a while (of not being able to find the right keys to get the scooter movable (steering lock)), all roughly ended up “okay”

Sonic.net == made of awesome

This is a short post, for now. But. I am I .za. Sonic.net is not. I have had some major shit go down right now. And a friend at Sonic.net has helped me. More than I can quantify right now. If you can *in any fucking way possible* buy their services, do so. Srsly. I am not even fucking kidding or exaggerating. Support these people. They are the ultimate awesome.

And another one

Seriously loving my cellphone company right now.

vodacom billing engine excellence

PRERATE ALL THE THINGS

So, to whichever unfortunate person ends up with my support ticket for this, I wish you luck.

P.S. I was told there’s more context needed, so: look at the bundle type in the top bar, and then at the used/avaiable counts. What happened (my guess, but I’m pretty sure) is that when I bought it, their rating system preburned all the data I’ve already done this month (under previous rollover bundle).

Yet another bit of Vodacom portal fun

On my normal account. That used to work perfectly fine.

stupidity

stupidity

Rocket-solid .za DSL setup

After a couple of SOHO router iterations (not all my own, sometimes seen via other people), recurring problems:

  • overheating devices due to bad design
  • crappy stock firmware
  • bugs (that often never get fixed, or updates are never applied)
  • other assorted silly issues, like a 1000-connection session table, or small arp tables, or or or …

Much like the rest of the local linux community, I’d long just done DSL bridging and let a Linux box do the work of Real Internet Connection(tm), but that’s not the easiest pitch for Joe Average Home User. So I found an alternative set of things that works fairly well, and is solid enough for you to fire rockets at it. The recipe is as follows

  • 1x DSL bridge of some kind. Some Broadcom chipsets are excellent
    • I’ve had a hell of a lot of lifetime out of the DSL-2500U
    • if you want cheaper, this Tenda unit
  • 1x Non-shit router
    • A RB750 is pretty good for this. Gets your packets going without too much fancy. Has other benefits too
  • 1x Extra switch, because cheap gigabit is good
    • again, a Tenda unit
    • there are some cheap tp-links and HPs around too that I’ve had before

Cheap, effective, and just slightly annoying on amount of power sockets used. The power used is fairly low, too (we’ve had to test it on another project before, and it’s something like R20/month).

I’ll update the post a bit later with the basic tik config to apply.

Indulgence

This is not a technical post. Nor anything of any consequential substance. I just felt like telling a story.

As mentioned before, I run a little project called Earnoms, which was started to find a place for people to share music, but (if I had to blurb it) in ways that are more social than the “social” music sites. Tonight was a mellow music night. Ended up with over 100 new URLs in the LinkDB just between a couple of people. Not bad, all things considered.

There’s an Afrikaans song called Lisa Se Klavier. I’m not sure if it was Koos Kombuis or Laurika Rauch who wrote it, and I’m sure a quick trip over via Google could answer the question, but I don’t actually care all that much right now. The best online version that I know of is this one. I should note that the statement is somewhat subjective, since I don’t like it when people go full-on drumkit with songs. Almost always it spoils the feeling. If that’s your thing, there’s another version on grooveshark that’ll suit you. There were a couple of other ones mentioned too, such as this track by aKING, and the version of Hallelujah by Karen Zoid which I discovered tonight.

But my story is more of history. Of a couple of years ago when I used to be in choir, at a time when choir was one of the few things of my life that I don’t want to rewrite (or at least, don’t consider wasted, unlike much else of the time then). Of a time when I knew three girls, all whose names started with L. All who could play piano (to varying extents), all who were in choir with me. All who were friends with each other, and loved to song. And I mean loved to sing. I don’t think I ever saw any of them sing without a smile on their faces. It was them who introduced me to some of my first tastes of slightly-specific musical appreciation, even way back then, entirely unintentionally. Unintentionally, because they were just singing to sing. But all things considered, it’s their version of Lisa Se Klavier that is my all-time favourite. Because between the three of them they covered soprano, alto, and contralto. Their version had them picking up the rises and falls from each other, toying with the sound, making it all the more playful (which, if you look at the lyrics, are oddly fitting in some ways). All the while still following on perfectly with each other, all the while smiling. That’s probably one of the memories I’ll carry with me for a very long time. Because it stands out brightly, lightly, and oh so very pleasantly.

All that waffled, I’ll leave you with this: Ella Fitzgerald and Louis Armstrong singing Summertime.

My IRC setup

Potentially a somewhat bland topic, but I find myself referring to this often enough that I wanted to write it up; saves me the effort of explaining it in future.

First off, a couple of requirements:

  • low latency on the user input side
  • deal with my somewhat ridiculous volume of IRC usage
  • accessible from any reasonable platform (which I roughly classify as “anything with a keyboard for input, and has internet”)

So when I say “somewhat ridiculous”, that means:

  • 12 IRC networks
  • 65 channels (of varying volume)
  • varying numbers of query windows, usually about 30+ open

I flatten my jabber/gtalk to IRC as well, by using bitlbee. It counts among the 12.

Historically, I had this set up as irssi with irssi-proxy in the USA, then another irssi+irssi-proxy at my home (to join a network only accessible via the WUG at that point), and then I would connect my machine-local client to that server (which was on dyndns). A couple of pain points with this included the fact that dyndns sucks donkey balls, and syncing of logs (which I did with rsync at the time) was crappy. At the time I also had less IRC volume than I do now.

Aside from those pain points, and the occasional power outage at home (which just made me link up to my parent client), this worked well. Quassel’s backlog fetching is shiny, though. Very shiny. I wanted it. So I redid my IRC setup. Now there’s only one master server (currently in Germany), with a quassel core connecting to it. The irssi proxy config looks like so:

20:08:39 -!- Irssi: Module proxy/proxy already loaded
20:08:44 [irssiproxy]
20:08:44 irssiproxy_bind = 127.0.0.1
20:08:44 irssiproxy_password = passwordhere
20:08:44 irssiproxy_ports = freenode=6001 shadowfire=6002 oftc=6003 bitlbee=6004 ...

So basically:

  • /load proxy
  • /set irssiproxy_bind ip
  • /set irssiproxy_password ircpassword
  • /set irssiproxy_ports network=port network2=port2

Connect the quassel core up to the proxy, and that’s it. Infinite scrollback for any of my devices with quassel, and I can just ssh from some random server and connect up to screen as well (which has saved my bacon in DCs a few times).

And yes, I know this is (a bit) crazy.

Vodacom supply chain management adventure

Or “two week’s later and I still don’t know when I’m getting my new phone”.

On the 8th of this month (May 2013), Vodacom announced they had the HTC One up for order, which I’ve been awaiting fairly eagerly. So I go through the order one, have the upgrade process initiated, and ….. well, I’d like to tell you what’s happening next, but apparently I’m in customer support limbo:

Time to wait some more, I guess.