From my bin

Since about 1996, there's a directory in my home called mybin. It's in my path and full of little (or sometime a bit larger) scripts and hacks I need or needed at some point.

I'm fully aware that mostly, there are better programs for doing some of these things or that the stuff is something of a waste of time for other reasons as well. Therefore, and because most of it isn't really written beautifully, I've not been going through the trouble of actually polishing up the software for release (which is roughly two to ten times the work of writing it).

Still, maybe some of these things solve your problem -- if you stumbled over this page coming from some random search engine, this is not unlikely. I'd be happy if someone asked me to fix this or that up, add docs and whatnot. Just talk to me. Seriously

Note that much of this stuff probably is not portable. I haven't really reviewed it for my-machine-isms, linuxisms and unixisms. Again, I'll do what I can on request.

Wikipedia in a box

This one is actually used by others, and thus it's got a page of its own.

Pining for the Fjords

My timekeeping stuff also has found some users and has moved to a page of its own.

Fancyconfig

fancyconfig.py tries to do for configuration processing what optparse did for command line option processing: A declarative way of handling the main chores. Having used it in quite a few projects now, I'm close to putting it up on PyPI. One piece of positive feedback will do it.

There's minimal documentation in the module docstring, examples in the embedded unit tests, and more examples in pftf, pysmap, spitmonkey, and onthers. I'll provide more docs on request.

This stuff is available through svn as well: http://svn.tfiu.de/pftf/.

Get twitter updates

I'm not into microblogging, and if I were, I'd use my identi.ca account rather than twitter. Still, when great or lesser things happen, Twitter's usually the place to go for gossip, and we all love gossip. However, I certainly do not love the CPU-guzzling javascript mess twitter delivers as a user interface, and thus I wrote twitupdate.py. You pass some search term (say, a hash tag -- use quotes for those) on the command line and get new tweets matching your term on your console every 100 seconds or so.

Annotator

I figured I needed to organize the large body of photos I have accumulated over time. This being the second half of the naughties, it had to be with tags. Of course, the gazillion programs that are already out there for doing just this were, eh, not invented here, so I wrote something myself. The result, ianno-0.3.tar.gz, is now based on an SQLite database and certainly not superior to most of the other photo handling code. There are, however, Tkinter-based timeline and tag cloud widgets in there that might be useful to other people. If you are interested, let me know.

There's not README, so here's the rules: When annotating, run ianno *.jpg or similar. On the right side of the window, you can add and select tags. The rest of the functionality is not very discoverable, so I'll give you the key bindings:

To make the thing run, you'll need PIL (python-imaging in debian)

Once you have annotated your files, you'll want to view them. To do this, call ``iaview``. The key bindings there are:

VIM starters

Trivial capturing portal

I'm a bit of a zealot on many issues. One thing I'm conviced of is that the right to share is a quite fundamental one. So, despite all the FUD I'm keeping my access point open. Since my access point actually is an old subnotebook, I decided to make it clear that it's open on purpose so people don't feel bad about using it. Basically, I wanted to tell people "You're welcome, but behave yourself."

The solutions for capturing portals out there were all much too fat for my taste. So, I wrote my own: openFw. It's all in one file.

To install it, make yourself a directory (say, /portal), and put openFw in there. Go there and adapt the configuration at the top of the file to your liking. You should also edit the embedded HTML pages for the portal itself, then say python openFw make. This will build a helper that will enter a MAC address into the iptables. It has to run as root, and you will be asked to set it to suid root. Do that -- the alternative would be to let the apache user run iptables, and that's not a good idea even if you're not paranoid.

After building the thing, openFw suggests a stanza for inclusion in your apache's config. Use it or something else, openFw itself is just a CGI.

Finally, openFw also serves as init.d-style "script" to setup or tear down the iptables rules; you should run openFw start as root at some point during system boot. You could make a link from /etc/rc2.d/S80capportal to openFw if you want, have a SYSV init, etc.

Note that I haven't worried about isolating wired and wireless, there are no DMZs and so on. All machines in my net are at some point are out on the internet, so I don't bother giving them additional protection from people using my WLAN. If you rely on firewalls for your network's security, this thing isn't for you, at least not without modification.

Offline Mapnik rendering

I personally believe OpenStreetMap is a project with a potential comparable to wikipedia. So, I started contributing. And I wanted to use it offline. I didn't like tilecache and so I started writing something to control mapnik rendering and displaying tile caches without having to go through web interfaces and the like.

When I had this, I realized I wanted to crop GPX traces before uploading them to osm, and so I added support for doing that. While I was about that, I decided I wanted some privacy enhancement by slightly modifying the time stamps on the traces.

The result is pysmap-0.2a.tar.gz. You can find a subversion repository for this at http://svn.tfiu.de/pysmap/.

Asynchronous radio listening

When I got my MP3 player back in 2000 (thanks again, ADS folks, though the device itself has retired in 2007), I immediately had my computer record radio broadcasts that I could listen to when I had time to do so rather than when they were broadcast. I consider myself a pioneer of podfetching...

Anyway, around this grew a large number of hacks, e.g. oggsnarf.py to fetch ogg streams, mp3stream2mp3 to fetch and recode mp3 streams (when I still did this; fortunately, I'm no longer tied to mp3, and the times of the 32 Meg cards of the old iJam player are long gone), getpods.py as a simple CLI podcatcher, or mpcardman.py which I used back in the day to conveniently transfer and chop up the files to the cards (this last one contains so many dependencies on my system that it's unlikely to be any use to you).

You'll need Timeparser.py for some of those, and SizeSelector.py for mpcardman, plus some modules you'll find in your distribution's repository. Talking about the SizeSelector, this one is also used in my incredibly hacky CD-burning program selAndBurn.py. Let's ignore that, shall we? It's really only interesting if you don't mind hacking around in the source.

These days, I run my own player software on a Zaurus, which get its data from autofeeder.py, a HTTP server that exposes the currently recorded broadcasts as an RSS feed. After a download, the client sends another request that causes the downloaded file to be moved into an archive (or removed) -- I usually burn those archives to CD now and then. It needs feedgenerator.py that I lifted off the net but can't remember where.

The client as it runs on the Zaurus is feedpull.py. However, it depends on some modules from my player (which carries the uninspired name "autoplayer"). That, in turn, is a mad experiment with asynchronous interprocess communication with frontends for Tkinter (which I use to let my computer tell me stories while I doze off), pygame (which I used on the gp2x before I realized I shouldn't have left the Zaurus) and pyqt (or rather qpe, primarily on the Zaurus). It would be real work to package that thing up, so I'll definitely only do that if there's credible interest.

I'm working on a twisted-based rewrite of the thing off and on, but the stuff works wonderfully for me, so there's little incentive right now apart from the fact that the old system is technically unfulfilling (although I kind of like the asynchronous automaton system that's in there). Ah well.

Knapsack problem in media collections

Back when 20 Gig disks were considered luxurious, I burnt off my media collections to CD. I wanted folders to stay together and get CD-sized chunks. So I wrote the autoburner. With it, you define "heads", like ~/media/books and ~/media/audio. For each head, you run autoburner.py now and then. If enough stuff for a CD has accumulated, the thing builds a hierarchy of links (by default /usr/media/cdimgN, but there's the -r switch to change that, or just edit the default in _makeOptionParser; you see, the program even has a working -h switch, thanks to the great optparse module) that you can then burn using something like

mkisofs -v -f -l -R -J -r -o cd.img cdimg?
cdrecord -v -tao -dev 2,0,0 cd.img

With today's huge disks, I've stopped going into all that trouble, but I kind of enjoyed the program anyhow. To use it, you need

Sawfish dock

The window manager sawfish doesn't have a built-in dock. There's the relatively fancy sawdock to remedy this, but for reasons I don't remember I didn't quite like it. Instead I realized that a dock, in the end, is nothing but a fixed-position window manager that can raise and lower the windows it manages in unison.

So I wrote (without actually bothering to understand what this "command" thing of sawfish's is and in general not actually delving too deeply into rep) mydock.jl, which you can drop in your ~/.sawfish/lisp directory. After that, you can put things like these in your .sawfishrc:

(require 'mydock)
(mydock-add-to-dock '(0 . -1) nil "pager")
(mydock-add-to-dock '(0 . 0) "sunclock" "sunclock / clock")
(mydock-add-to-dock '(0 . 58) "wmbubble" "wmbubble")
(mydock-add-to-dock '(58 . 58) "wmacpimon" "wmacpimon")
...

The idea is that the cons cell gives the position, the next argument is the executable to run, and the last argument is the window title. All windows with this title will be forced to this position, which I find useful if I want to restart a docked applet. The source above binds the "toggle front/back" function to the select key (which I've mapped on the windows key on my keyboards).

Three more hints:

One line CLI

onelinecli.py is a bit of a personal "classic"; I've written this about ten years ago and have always wanted to rewrite it in something else to cut the overhead of python/Tkinter or replace it by some other program of its kind, but it never happened.

Anyway, this is one of these "run"-like dialogues, but I mainly use it to select a URL, call it up (it's bound to the "Menu" key on my keyboard) and hit return to open that URL, or a similar thing for search terms; there's a history and tab completion, but I rarely use that (well, the history, sometimes).

The one and single selling point is the simplicity you can add handlers; just add a function handle_something(command) -> int; inspect command (which may be a unicode string), and if you decide to do anything with it, return True, otherwise return False. The machinery will pick up your handler automatically.

Python bindings for the tremor library

My media player had to run on some platforms that don't have an FPU, and so I did a very rough wrap of the fixed-point vorbis decoder tremor (that you'll need before this stuff builds; use the make file for cross-compiling, since cross-compiling with distutils sucks): pytremor-0.1.tar.gz.

Suspending and resuming USB devices

I used to have a DVB-T receiver that drew obscene amounts of power even when idle (judging from how warm it got, I reckon it's quite a bit more than 1 W). However, I wanted to use it as a digital video recorder, and thus, plugging it in before use and out after use isn't much of an option -- the machine has to be able to turn it on.

That's why I wrote a hackish script usbsleep that takes a USB id and either "on" or "off" on the command line. If one or more devices with the given id are connected, they are either woken up or suspended.

Note that the thing sudoes itself if it's not root. If you want to work without user intervention, you'll have to add it into a NOPASSWD clause in sudoers. Don't do this for users that can't sudo root anyway, since even if I gave absolute paths, shell scripts are terribly hard to get safe in a limited sudo environment. I guess I should do a C implementation...

netshare

I frequently find myself in a situation in which my machine should act as a router, e.g., for my zaurus, when there's wireless, but others can only do wired etherenet or, conversely, I have a wire connection and others for some reason or other can only go wireless.

To spare myself the intellectual strain of having to figure out the iptables incantations, I have set up a couple of "demo" interfaces in my network/interfaces (you see, I'm not only a Linux bigot, I'm, worse, a Debian bigot, though the script itself is not distribution specific), wrote some definitions for the dhcpd and came up with netshare.

The usage is rather simple: You say, e.g., netshare -u usb0 eth0 to do DHCP and NAT on usb0, forwaring the packets to eth0 (think "up") and, when done netshare -d usb0 eth0 (think "down"). When you give no option, you the the "interactive mode", where the script sets up the forward and waits for your keyboard input until tearing it down again.

Note that the program needs root access to do all that magic. It'll try to sudo itself if necessary. You'll probably want to review this script before giving the password, don't you?

Scan the wireless situation

Now, given that most lawmakers are paranoid technophobe morons (I'll give you that the moron part may be debatable), it may be that the possession and/or use of wlanview.py is illegal in your jurisdiction, but I happen to think it's a nice toy anyway. It shows the WLANs in your neighbourhood (green is open, red is encrypted, and from the fact that I don't differentiate between WEP and WPA you can see that I'm really a good guy) with funny dancing bars that get pushed when a beacon comes in and then decay slowly.

It's quite likely that this only works for ipw2100 or similar chipsets.

Unrotate images

This one is in the department "old, horrible sources that may do something useful". Imagine you have a directory full of g3 or g4 tiff scans. Many of them are somehow skewed. To fix them, I've written imgcleanup.py. It gives you a slightly funky interface to specify how to rotate (basically, you follow the outlines of the unrotated pages -- in the GIMP, this is known as "corrective" rather than "traditional") and then comes up with a script that can run unattended to fix the images.

You'll need unrotate.py on your path to do the actual work; it in turn uses ghostscript (thanks to Alberto Accomazzi for that brilliant idea).

Alarm clock

There are times when even I need an alarm clock. For those times, I've written wecker.py. You'll need to give yourself access to /proc/acpi/alarm for this to work, but then you can turn off your machine and it'll come up when the time is right. Now, of course the noise of the machine coming up will probably wake you, but if not, there's music fading in do finish the job.

You'll probably need to adapt the sleepCommand at the top of the script or kill the functionality that the machine goes to sleep after having played its tune. I think it's a bad idea to kill it, though, since it's quite conceivable that machine is in a tight spot when waing up, and you wouldn't want to have it in there when some CPU-intensive task comes up from the depths of cron.

Resolving MACs to vendors

I once was at an astronomer's conference that had some hare-brained access scheme for the wireless network that involved a page dumping the MACs the thing had seen (I really have no idea what rode the implementors to do that). This prompted me to write matchoui.py. For this little hack, you'll have to select the function you want by commenting in and out the toplevel function calls at the bottom of the script.

Also, you need the public OUI listing in your directory (or change the path in the default argument of loadoui). You can then either resolve one OUI from the commandline, resolve all from a a file (which was what I wrote the thing for) or, quite on the verge of becoming unethical, pipe the output of tcpdump -e into the script and see what vendors are on your net in buffered real time (so you'll have to have some patience on networks with low traffic.

By the way, it turned out back then astronomers love Apple notebooks; you can to cheap statistics using the usual unix tools: python matchoui.py silly_dump_from_wlan_control.txt | sort | uniq -c | sort -n.

Symmetrizing people

Once upon a time there was an evening when the topic of symmetric or non-symmetric faces came up, so I wrote symmetrizer.py. Give it the name of an image on the command line, set the line of a supposed axis of symmetry with the mouse, press 'd' and it'll show you an image each from combining the left half with its mirror ("links") and the right half with its mirror ("rechts"). Hit 's' to save the two images to right.jpeg and left.jpeg.

It's surprisingly hard to obtain good photos of faces for that purpose, if only because people's heads always seem to be a bit askew. I've always wanted to develop this program a bit further; I guess marking the nose and the two eyes would give the program enough data to straighten the image a bit and then get more convincing symmetrized faces. Any interest will push my motivation...

El-cheapo accounting

Even a real cool joint like the bicycle self-help repair workshop URRmEL needs a little bit of accounting. To do that, I started off doing some awk hacking but quickly returned to python. This resulted in makereport.py. It's basically a little program do generate reports from receipts coming in.

Check the top of the script for an explanation on how this is supposed to work.

Actually, I just lied. I wrote the thing to manage a large hardware purchase with 5 institutes buying maybe 250 different things. It even worked fine for this. I won't try to convert you from awk, but it beats "office"-type spreadsheets ("Excel", for those who call cream cheese "Philadelphia" and Samba "Nutella") for doing such things any day. And you can enter your data in vi.

EBNF from pyparsing

pyparsingToEBNF.py is a (somewhat incomplete) quick hack at generating a more or less readable variant of EBNF from pyparsing grammars. There are some usage guidelines in the module docstring.

Format pyparsing debug output

The script pyparseFmt is useful when you debug pyparsing grammars. It adds indentation to the Matching/Matched pairs. This is particularly handy when you load the result into vim and say set foldmethod=indent -- you can then fold in and out the various experiments of pyparsing.

Screenscraping

Well, when the web was still relatively young and wild (like, 1999) I wrote a horrible screenscraping library to harvest bibliographic information. After looking at BeautifulSoup I had to realize I did it wrong. BeautifulSoup (plus maybe the getWithCache function contained in dlctmag.py) is the way to go if you are in the unfortunate situation of have to screen scrape (there's a package python-beautifulsoup in Debian, since python 2.5 BeautifulSoup is part of the standard library).

If you don't know Beautiful Soup, check out grabarte.py. This is for downloading videos from ARTE's video site (called ARTE+7). It's the usual flash mess, which makes viewing the stuff extremely unpleasant. grabarte just takes the URL of the embedding page and uses BeautifulSoup and rtmpdump. And ARTE folks, if you read this, pleaseplease stop this sillyness of making it deliberately hard to view your videos.

Well, and while we're at screen scraping: Nasa TV is yet another of those plugin-infested video pages that lead you through lots of funny XML-based redirects. Mplayer gets confused, and that's when I hate XML for a few moments. And when I hated XML, I wrote my RE-based revenge: nasatv.py; it should start an mplayer on the live stream of the press feed. Ha! Yes, I know, you shouldn't do it like this, and I only post those few lines here out of contempt for the current state of video distribution on the web.

Spitmonkey

This is almost screenscraping, but this one I've polished a bit more, and actually, it's more about analyzing document structure. I probably shouldn't have done this anyway, since people could take this to be an eBay sniper. It's not, really, though it would be almost trivial to build one base on this.

It's basically a command line interface to bidding on eBay. I wrote it after I had tried to get a spare JVC machine but either was the victim of snipers or forgot about the auction on a couple of occasions. So, I came up with spitmonkey.py to leave the actual bidding to someone more reliable that I am. The accuracy of unix' at(1) is more than enough unless you're trying something evil. There's some docs at the top of the file. If you use it at all, do so ethically, please.

Fixup script for pages like this

I know there are more HTML preprocessors than people using them. Here's mine, and it's used for this page and my JVC page: fixup.py. It uses myxml.py.

By the way, the whole thing gets nifty with a Makefile like this:

.DELETE_ON_ERROR:

%.d: %.phtml
	@echo -n ${<:.phtml=.html}: > $@
	@fixup.py --deps < $< >> $@

%.html: %.phtml
	fixup.py < $< > $@

SOURCES = index.phtml

index.html:

include $(SOURCES:.phtml=.d)

Almost looks like perl, eh? This way, you'll get automatic dependencies for all those files included. Of course, you'll need to adapt SOURCES to whatever files you have.

As to the markup supported -- well, read the source. You're not actually going to use this, are you?

Last update: 2011-11-25, 16:38 UTC.

Markus Demleitner

You probably do not want to mail wrzblg@fjkdiel32x.net, because that's just some random gibberish for you-know-who.