From my bin

Since about 1996, there's been a directory in my home called mybin. It's in my path and full of little (or sometime a bit larger) scripts and hacks I need or needed at some point.

I'm fully aware that mostly, there are better programs for doing some of these things or that the stuff is something of a waste of time for other reasons as well. Therefore, and because most of it isn't really written beautifully, I've not been going through the trouble of actually polishing up the software for release (which is roughly two to ten times the work of writing it).

Still, maybe some of these things solve your problem -- if you stumbled over this page coming from some random search engine, this is not unlikely. I'd be happy if someone asked me to fix this or that up, add docs and whatnot. Just talk to me. Seriously.

Note that much of this stuff probably is not portable. I haven't really reviewed it for my-machine-isms, linuxisms and unixisms. Again, I'll do what I can on request.

Wikipedia in a box

This one is actually used by others, and thus it's got a page of its own.

Pining for the Fjords

My timekeeping stuff also has found some users and has moved to a page of its own.

Arte downloader

There are few things that annoy me more than fat web UIs and video players within web pages. That's why over the last 10 years or so I've been following the various turns of Arte+7 and Arte whatever to keep my video downloader up to date. It's grabarte.py. After following about a dozen crazy twists of arte's video player, I'm now depending on the excellent streamlink for the most complicated part, fiddling out a video stream.

If you call grabarte.py without arguments, it will get a list of videos from Arte (whoa! this used to be RSS and now is BS) and display them in a curses UI. Use cursor or vi keys to move around, return to view the description, q to quit and g to download your selection.

Alternatively, if you happened to use the horrible web UI, you can pass in the URL of a page embedding a video; then grabarte will go ahead and pull the video right away.

Language selection is currently (starting Dec 2019) broken. If you're french (or perhaps even English) and want to use this thing, let me know – I think it's simple to fix, but since the Arte people changed their website three times in 2019 (no joking), I'm not wild to maintain features that I'm not sure anyone in the whole wide world actually uses.

Remote RPG aid

Duing 2020's Corona thing, some friends and I wanted do so some roleplaying without having to be in the same room. Voice is fine with mumble, but what about dice, sketches, and maps?

Well, enter wuerfler.py. That's a one-python-file web server for websockets-based live sharing of

There's some installation instructions (essentially: just starting the python script and pointing the various browsers at http://host:3344, except that you'll probably need https with all the associated pain because browsers might not let you do websockets and cameras via http) in the script.

Also (in the vague hope that some search engine will pick this up): This also contains a twisted-based server-side websockets implementation based on some relatively ancient code that's still around in the remotes/origin/websocket-4173-4 branch of twisted's github repo, ported to python3 (which is not entirely trivial). I'm too lazy to feed it back into the github branch, in particular because I think websockets, while fun to play with, are an abomination. But perhaps someone else can use or re-use that code. By all means feel free to do so.

Redacting PDFs

Recently, I sometimes have to prepare PDFs for publication. This usually incurs some redaction ("schwärzen"). Doing this manually with gimp and company was painful, so I wrote redactor.py3, which is a quick hack using python3, Tkinter, the pillow fork of PIL and ImageMagick.

It's not polished at all at this point (e.g., no scroll bars, canvas scale only updated after page flip, no progress bars even for potentially long-running tasks, during which the UI freezes, generally relatively clunky UI, etc), but it does the job, and I think the python code is still fairly understandable, so it should be easy to expand it to a more user-friendly PDF redaction tool.

A (perhaps) welcome side effect is that the PDF is turned into a 300 dpi bilevel CCITT Group 4 PDF, which is relatively compact. Of course, if you had text before, you lose indexability.

A Battery Monitor

Ok. There are many battery monitors out there. But at least last time I looked surprisingly many could deal with two batteries. And my my main machines have had to batteries since 2005 or so. That's when I re-worked wmacpimon to work with two (and improved a few more things). I think the code was orphaned even back then, and given its current homepage it certainly is now.

Well, I do maintain my fork at least for my my current machine(s) – this could be relevant because the ACPI names of the batteries apparently are not constant. Feel free to use the thing, too: wmacpimon-0.6.3.tar.gz.

Manual Automounting

Call me conservative, but I feel this udev-dbus-udisks type "automounting" of things you insert into your box isn't useful; in particular, it turns into sheer terror when there's multiple partitions on whatever is inserted, and it doesn't work at all for network resources. Rather, I'm a big fan of good old-fashioned automounting for hotplugged hardware -- can it get easier than saying cd /auto/stick?

However, kernel-based autofs doesn't mix very well with fuse file systems (like the great encfs, sshfs), and things stink when ssh agents or the need to enter encfs passphrases come into play. As a replacement, I'm using a shell script with, that lets you have a file enumerating names and mount commands (without the mount points) like

enc encfs ~/.encrypted
host1 sshfs host1
host2 sshfs -o rw,allow_other,workaround=rename user@host2

in ~/.config/with/mounts. After this, you can say with enc or with host2 and work in the mounted directories (which are in your home, named like the mount names). When you exit that shell, the FSes are automattically unmounted (or at least enrolled for unmounting when the last user exits).

Have a quick look at the docs at the top of the script. Among other things, it tells you to add a line eval "$BASH_POST_RC" to your bashrc. If you do, you'll have an easy time spotting shells which have manual mounts in this way.

Simple schleuder replacement

I've been running the OpenPGP-enabled list manager schleuder for years, sometimes cursing it because I didn't quite feel up to maintaining its many lines of Ruby code, aging dependencies, and quirks. When, in Debian jessie, schleuder dropped out of Debian and I got lost in its dependencies, I decided that I didn't actually need a full-blown mailing list manager to replace it -- a "mail exploder" decrypting incoming mail and re-encrypting it for an operator-defined set of recipients was quite enough.

So I wrote simplepgpexploder.py, which is around 200 lines of python and, hopefully, supports the major ways of distributing PGP content. Installation and operating instructions are at the head of the script.

Note: Schleuder has some actual maintenance again, so you may want to ignore this thing here.

Fairly simple dyndns

Since first dyndns itself and then no-ip.com decided to scare away their non-paying non-customers, I finally decided to run a (simple, private) dyndns service myself. If you run a nameserver reading bind-type zone files (the script actually assumes nsd, but only in the notification of the daemon, so that'd easily be fixed) and have a CGI-capable web server on the same machine, then my little update_zone script is for you.

To give you an idea: After some reasonably simple steps of setup described in the script docstring (see the source; they boil down to adding a user that can manipulate your zone file and grant the zone file to it, prepare a directory containing the CGI, make a CNAME for a virtual server running the service, configure apache to use it, and finally enable suexec), all you need to do is add normal A records in your zone file and decorate them with comments like ;DYNNSD:somerandomstring. After that, to update you just need to access http://updater.your.doma.in/somerandomstring from the box with the unstable address and the A record will be updated.

Fancyconfig

fancyconfig.py tries to do for configuration processing what optparse did for command line option processing: A declarative way of handling the main chores. Having used it in quite a few projects now, I'm close to putting it up on PyPI. One piece of positive feedback will do it.

There's minimal documentation in the module docstring, examples in the embedded unit tests, and more examples in pftf, pysmap, spitmonkey, and onthers. I'll provide more docs on request.

This stuff is available through svn as well: http://svn.tfiu.de/pftf/.

Get twitter updates

I'm not into microblogging, and if I were, I'd use my identi.ca account rather than twitter. Still, when great or lesser things happen, Twitter's usually the place to go for gossip, and we all love gossip. However, I certainly do not love the CPU-guzzling javascript mess twitter delivers as a user interface, and thus I wrote twitupdate.py3. You pass some search term (say, a hash tag -- use quotes for those) on the command line and get new tweets matching your term on your console every 100 seconds or so.

Sync GPE calendars

Ok, so syncing calendars and similar is an non-trivial problem. Still, I'm a bit... disappointed in the state of common synchronistation solutions. But then, using Free software throughout the machine park helps, and so I hacked together a short python script that synchronizes the GPE calenadars (the package is gpe-calendar) between my desktop an my N900, sync-gpe-calendar.py.

It doesn't do alarms yet since I don't want alarms on both boxes, but it'd be trivial to add them.

The way I use this is that I have some script that is in the post-up of the N900's usb interface in /etc/network/interfaces, and I have set things up in .ssh/config such that an ssh mikis drops me into a shell on the usb-connected N900 as its user.

With this, the sync script (that runs as root) can do:

CALENDAR_USER=msdemlei
mountpoint=`su $CALENDAR_USER -c "/bin/mktemp -d /tmp/mikis.mountXXXXXXXXX"`
cleanup () {
	fusermount -u $mountpoint
	rmdir $mountpoint
}
trap cleanup EXIT

sudo -u $CALENDAR_USER bash -s << EOF
if [ -z "$mountpoint" ]; then
	logger -puser.error "No mountpoint, WTF?" $?
	exit 1
fi

if ! sshfs mikis:/home/user/.gpe $mountpoint ; then
	logger -puser.error "sshfs mount failed";
	exit 1
fi
cp $mountpoint/calendar $mountpoint/calendar.bak
python ~/mybin/sync-gpe-calendar.py $mountpoint/calendar ||
	logger -puser.error "sync-gpe-calender failed, try manually"
EOF

(where I am msdemlei and mikis is the name of the N900). May look a bit complicated, but there's really less magic here than in all those horrible SyncML constructs.

Annotator

I figured I needed to organize the large body of photos I have accumulated over time. This being the second half of the naughties, it had to be with tags. Of course, the gazillion programs that are already out there for doing just this were, eh, not invented here, so I wrote something myself. The result, ianno-0.3.tar.gz, is now based on an SQLite database and certainly not superior to most of the other photo handling code. There are, however, Tkinter-based timeline and tag cloud widgets in there that might be useful to other people. If you are interested, let me know.

There's not README, so here's the rules: When annotating, run ianno *.jpg or similar. On the right side of the window, you can add and select tags. The rest of the functionality is not very discoverable, so I'll give you the key bindings:

To make the thing run, you'll need PIL (python-imaging in debian)

Once you have annotated your files, you'll want to view them. To do this, call ``iaview``. The key bindings there are:

VIM starters

Offline Mapnik rendering

So, I wanted to use openstreetmap offline. I didn't like tilecache and so I started writing something to control mapnik rendering and displaying tile caches without having to go through web interfaces and the like.

When I had this, I realized I wanted to crop GPX traces before uploading them to osm, and so I added support for doing that. While I was about that, I decided I wanted some privacy enhancement by slightly modifying the time stamps on the traces.

The result is pysmap-0.2a.tar.gz. You can find a subversion repository for this at http://svn.tfiu.de/pysmap/.

Sawfish dock

The window manager sawfish doesn't have a built-in dock. There's the relatively fancy sawdock to remedy this, but for reasons I don't remember I didn't quite like it. Instead I realized that a dock, in the end, is nothing but a fixed-position window manager that can raise and lower the windows it manages in unison.

So I wrote (without actually bothering to understand what this "command" thing of sawfish's is and in general not actually delving too deeply into rep) mydock.jl, which you can drop in your ~/.sawfish/lisp directory. After that, you can put things like these in your .sawfishrc:

(require 'mydock)
(mydock-add-to-dock '(0 . -1) nil "pager")
(mydock-add-to-dock '(0 . 0) "sunclock" "sunclock / clock")
(mydock-add-to-dock '(0 . 58) "wmbubble" "wmbubble")
(mydock-add-to-dock '(58 . 58) "wmacpimon" "wmacpimon")
...

The idea is that the cons cell gives the position, the next argument is the executable to run, and the last argument is the window title. All windows with this title will be forced to this position, which I find useful if I want to restart a docked applet. The source above binds the "toggle front/back" function to the select key (which I've mapped on the windows key on my keyboards).

Three more hints:

One line CLI

onelinecli.py is a bit of a personal "classic"; I've written this about ten years ago and have always wanted to rewrite it in something else to cut the overhead of python/Tkinter or replace it by some other program of its kind, but it never happened.

Anyway, this is one of these "run"-like dialogues, but I mainly use it to select a URL, call it up (it's bound to the "Menu" key on my keyboard) and hit return to open that URL, or a similar thing for search terms; there's a history and tab completion, but I rarely use that (well, the history, sometimes).

The one and single selling point is the simplicity you can add handlers; just add a function handle_something(command) -> int; inspect command (which may be a unicode string), and if you decide to do anything with it, return True, otherwise return False. The machinery will pick up your handler automatically.

Python bindings for the tremor library

My media player had to run on some platforms that don't have an FPU, and so I did a very rough wrap of the fixed-point vorbis decoder tremor (that you'll need before this stuff builds; use the make file for cross-compiling, since cross-compiling with distutils sucks): pytremor-0.1.tar.gz.

netshare

I frequently find myself in a situation in which my machine should act as a router.

To spare myself the intellectual strain of having to figure out the iptables incantations, I have set up a couple of "demo" interfaces in my network/interfaces (you see, I'm not only a Linux bigot, I am, worse, a Debian bigot, though the script itself is not distribution specific), wrote some definitions for the dhcpd and came up with netshare.

The usage is rather simple: You say, e.g., netshare -u usb0 eth0 to do DHCP and NAT on usb0, forwaring the packets to eth0 (think "up") and, when done netshare -d usb0 eth0 (think "down"). When you give no option, you enter the "interactive mode", where the script sets up the forward and waits for your keyboard input until tearing it down again.

Note that the program needs root access to do all that magic. It'll try to sudo itself if necessary. You'll probably want to review this script before giving the password, don't you?

Scan the wireless situation

Now, given that most lawmakers are paranoid technophobe morons (I'll give you that the moron part may be debatable), it may be that the possession and/or use of wlanview.py is illegal in your jurisdiction, but I happen to think it's a nice toy anyway. It shows the WiFi APs in your neighbourhood (green is open, red is encrypted, and from the fact that I don't differentiate between WEP and WPA you can see that I'm really a good guy) with funny dancing bars that get pushed when a beacon comes in and then decay slowly.

It's quite likely that this only works for ipw2100 or similar chipsets.

Unrotate images

This one is in the department "old, horrible sources that may do something useful". Imagine you have a directory full of g3 or g4 tiff scans. Many of them are somehow skewed. To fix them, I've written imgcleanup.py. It gives you a slightly funky interface to specify how to rotate (basically, you follow the outlines of the unrotated pages -- in the GIMP, this is known as "corrective" rather than "traditional") and then comes up with a script that can run unattended to fix the images.

You'll need unrotate.py on your path to do the actual work; it in turn uses ghostscript (thanks to Alberto Accomazzi for that brilliant idea).

Alarm clock

There are times when even I need an alarm clock. For those times, I've written wecker.py. You'll need to give yourself access to /proc/acpi/alarm for this to work, but then you can turn off your machine and it'll come up when the time is right. Now, of course the noise of the machine coming up will probably wake you, but if not, there's music fading in do finish the job.

You'll probably need to adapt the sleepCommand at the top of the script or kill the functionality that the machine goes to sleep after having played its tune. I think it's a bad idea to kill it, though, since it's quite conceivable that machine is in a tight spot when waing up, and you wouldn't want to have it in there when some CPU-intensive task comes up from the depths of cron.

Resolving MACs to vendors

I once was at an astronomer's conference that had some hare-brained access scheme for the wireless network that involved a page dumping the MACs the thing had seen (I really have no idea what rode the implementors to do that). This prompted me to write matchoui.py. For this little hack, you'll have to select the function you want by commenting in and out the toplevel function calls at the bottom of the script.

Also, you need the public OUI listing in your directory (or change the path in the default argument of loadoui). You can then either resolve one OUI from the commandline, resolve all from a a file (which was what I wrote the thing for) or, quite on the verge of becoming unethical, pipe the output of tcpdump -e into the script and see what vendors are on your net in buffered real time (so you'll have to have some patience on networks with low traffic.

By the way, it turned out back then astronomers love Apple notebooks; you can to cheap statistics using the usual unix tools: python matchoui.py silly_dump_from_wlan_control.txt | sort | uniq -c | sort -n.

Symmetrizing people

Once upon a time there was an evening when the topic of symmetric or non-symmetric faces came up, so I wrote symmetrizer.py. Give it the name of an image on the command line, set the line of a supposed axis of symmetry with the mouse, press 'd' and it'll show you an image each from combining the left half with its mirror ("links") and the right half with its mirror ("rechts"). Hit 's' to save the two images to right.jpeg and left.jpeg.

It's surprisingly hard to obtain good photos of faces for that purpose, if only because people's heads always seem to be a bit askew. I've always wanted to develop this program a bit further; I guess marking the nose and the two eyes would give the program enough data to straighten the image a bit and then get more convincing symmetrized faces. Any interest will push my motivation...

El-cheapo accounting

Even a really cool joint like the bicycle self-help repair workshop URRmEL needs a little bit of accounting. To do that, I started off doing some awk hacking but quickly returned to python. This resulted in makereport.py. It's basically a little program do generate reports from receipts coming in.

Check the top of the script for an explanation on how this is supposed to work.

Actually, I just lied. I wrote the thing to manage a large hardware purchase with 5 institutes buying maybe 250 different things. It even worked fine for this. I won't try to convert you from awk, but it beats "office"-type spreadsheets ("Excel", for those who call cream cheese "Philadelphia" and Samba "Nutella") for doing such things any day. And you can enter your data in vi.

EBNF from pyparsing

pyparsingToEBNF.py is a (somewhat incomplete) quick hack at generating a more or less readable variant of EBNF from pyparsing grammars. There are some usage guidelines in the module docstring.

Format pyparsing debug output

The script pyparseFmt is useful when you debug pyparsing grammars. It adds indentation to the Matching/Matched pairs. This is particularly handy when you load the result into vim and say set foldmethod=indent -- you can then fold in and out the various experiments of pyparsing.

Screenscraping

Well, when the web was still relatively young and wild (like, 1999) I wrote a horrible screenscraping library to harvest bibliographic information. After looking at BeautifulSoup I had to realize I did it wrong. BeautifulSoup (plus maybe the getWithCache function contained in dlctmag.py) is the way to go if you are in the unfortunate situation of have to screen scrape.

A quick application is getmaus.py grabs the flv for the great German children program "Die Sendung mit der Maus" off the Web. Since there's only one at one time, you don't even need to give it parameters.

Spitmonkey

This is almost screenscraping, but this one I've polished a bit more, and actually, it's more about analyzing document structure. I probably shouldn't have done this anyway, since people could take this to be an eBay sniper. It's not, really, though it would be almost trivial to build one base on this.

It's basically a command line interface to bidding on eBay. I wrote it after I had tried to get a spare JVC machine but either was the victim of snipers or forgot about the auction on a couple of occasions. So, I came up with spitmonkey.py to leave the actual bidding to someone more reliable that I am. The accuracy of unix' at(1) is more than enough unless you're trying something evil. There's some docs at the top of the file. If you use it at all, do so ethically, please.

Fixup script for pages like this

I know there are more HTML preprocessors than people using them. Here's mine, and it's used for this page, my XP731, Thinkpad X240" and luakit pages: format-howto-page (used to be called fixup.py before I ported it to elementtree and python3). This uses nice XHTML – but so far you'll have to read the source to figure out what elements there are. I'll write docs as soon as the first person says they'd like them (or check out the examples, which you can get by requesting index.phtml in any of the pages linked above).

By the way, the whole thing gets nifty with a Makefile like this:

.DELETE_ON_ERROR:

%.d: %.phtml
	@echo -n ${<:.phtml=.html}: > $@
	@format-howto-page --deps < $< >> $@

%.html: %.phtml
	format-howto-page < $< > $@

SOURCES = index.phtml

index.html:

include $(SOURCES:.phtml=.d)

Almost looks like perl. This way, you'll get automatic dependencies for all those files included.

Last update: 2023-12-25, 06:39 UTC.

Markus Demleitner

You probably do not want to mail wrzblg@fjkdiel32x.net, because that's just some random gibberish for you-know-who.