a backtracking python generator

I love python's `generator`s and needed a backtracking generator to help me out with a YAML parser that I am writing. This is possible using the `send` function, but the documentation and examples are a bit confusing for what I wanted. So I can find this next time I need it, here is a sample generator that  will cycle over anything that is iterable (e.g.: arrays, dicts, files), and all you to push things back using `send()`:

```
#!/usr/bin/pyton

def testGen(source):
    backlines = []
    for line in source:
        backline = None
        usedLine = False
        while usedLine is False or backlines:
            if backlines:
                backline = yield backlines.pop()
            else:
                usedLine = True
                backline = yield line
            while backline: # loops returning None for every send()
                backlines.append(backline)
                backline = yield None
```

And an example of use:

```
#!/usr/bin/pyton

source = testGen([1, 2, 3])
source.next() # returns: 1

value = source.next() # value is now: 2

source.send(value) # push it back

source.next() # again returns: 2
source.next() # returns: 3

source.send(4) # the value does not actually have to be from it
source.next() # returns: 4

```

Have you turned it off, then back on again?

A co-worker just ran into the best dialog box I have ever seen on MacOS:

Yes, you are reading that right: Apple has a dialog box telling you to turn your printer off, and then back on again.

Compiling the mysql-python bindings as a Universal binary

For a project I needed a copy of the mysql-python bindings that would run on every version of MacOS back to 10.5.0. But the compiled versions I could find of both the latest MySQL and the bindings were mostly for 10.6, and were either i386 or x86_64. So besides needing two versions (or to glue them together), that still left 10.5 computers, especially 10.5 ppc computers out in the cold.

All of the instructions that I could find to make Universal Binaries for both MySQL and the python bindings were written before MySQL switched to cmake, or did not cover making Universal Binaries. Since this took my a little while, I would like to have this documented for myself. So here is my documentation, available for general perusal.

To lead off, I am going to recommend that you do all of this on a partition that you don't care about, as following these instructions will install the MySQL binaries, but will not do any of the setup needed to get MySQL up and running (creating databases and users, etc...). So I would install the target OS on a volume that can be erased after you have what you need.

You will need:

• a volume with the minimum OS you want to support on it, preferably one you can mess up
• the XCode tools installed on that volume (or at least gcc)
• the following downloads: MySQL source code (5.5.9), cmake (2.8.4), and MySQL-Python(1.2.3)

I should note the places this process could be improved:

• we don't need to compile or install the MySQL server
• there is probably a way of getting the MySQL-Python bindings to compile from the compiled version of MySQL, rather than the installed version
• it would be better to have a staticly compiled version of the bindings that did not need the libmysqlclient.16.dylib file in the same folder (or installed)
• figure out how to use the sdk properly in the MySQL-Python compile so you could do this from latter versions of the OS targeting older versions

1. Download and install (or make and install) cmake.
2. Download and uncompress the latest versions of MySQL-Python and MySQL server, specifically the "Generic Linux" tar archive.
3. In the terminal create a temprary folder and use cmake to configure the compile of mysql:

   mkdir /tmp/buildTemp
   cd /tmp/buildTemp
   cmake "-DCMAKE_OSX_ARCHITECTURES=ppc;i386;x86_64 -DCMAKE_OSX_SYSROOT=/Developer/SDKs/MacOSX10.5.sdk" /path/to/mysql/source

4. Compile, and then install MySQL (this gets both the client and server, the latter we don't need):

   make
   sudo make install

5. Confirm that MySQL was built with the right architectures:

   file /usr/local/mysql/lib/libmysqlclient.16.dylib

6. Open the "site.cfg" file in the MySQL-Python source folder in your text editor of choice, and un-comment the "mysql_config" line and change it to point at where it at where it defaults to on MacOS X:

   /usr/local/mysql/bin/mysql_config

7. Compile MySQL-Python:

   cd /path/to/mysql-python/source/
   export ARCHFLAGS="-arch ppc -arch i386 -arch x86_64"
   python setup.py build

8. Verify that _mysql.so is built as a Universal Binary:

   file build/lib.macosx-10.5-i386-2.5/_mysql.so

9. Copy libmysqlclient.16.dylib in with the rest of the built files to create a library that is portable (note the exact path can differ):

   cp /usr/local/mysql/lib/libmysqlclient.16.dylib build/lib.macosx-10.5-i386-2.5/

10. Verify that the client works as expected

    cd build/lib.macosx-10.5-i386-2.5
    python
    >>> import MySQLdb
    >>> connection = MySQLdb.connect(host='database.host.name', user='db_user', db='db_name', password='db_password')
    >>> cursor = connection.cursor()
    >>> cursor.execute('select * from `my_table`)
    >>> cursor.fetchone()

You will now have a folder (lib.macosx-10.5-i386-2.5) containing everything you need to import into your project in order to get MySQL working with your Python code. You should be able to copy this folder onto a freshly-installed system with no additional installs and be able to make a connection to a MySQL database running on some computer.

Notes from the field

Working in computer labs you build up a repertoire of little tips and tricks, but I have never really taken the time to write them down and share them. So I am going to make an attempt here. I am going to try and come back and edit this page over time, and things will be a little random (as they pop out of my head), but here goes:

If you work with diverse Windows hardware keep local, uncompressed copies of the DriverPacks

The Windows hardware I have to maintain is a lab full of computers that are all unique hardware configurations. Every one has a different combination of hardware than the others (by design). For those of you who have to deal with Windows imaging you are probably already cringing, but this is what I have to deal with. I eventually stumbled on the DriverPacks web site, and while they are designed to be used in conjunction with SysPrep (which I can't use for other reasons), they are just folders of nicely collected Windows drivers compressed with 7zip. If you download all of them and collect them into a single tree of folders then when you have one-stop shopping for the random driver that you can't find. On Vista and 7 you can even just point the driver installer at the root folder and it will do the job of digging through for you. For XP you are going to have to do it more manually. (tip: driverpacks usually name the driver folders by the initials of the manufacturer, and Goggling the first part of the hardware types in the properties for the unknown device usually gets you the vendor)

I have gone one step further in my lab. I have a script on my local server that scrapes the DriverPacks web site and looks to see when they post new versions. It will then delete my old copy, download the new one, decompress it, and move the folder into place in the server share point. Occasionally I copy the network drivers onto a USB stick, and use that to get new installs onto the network. For the rest of the drivers I point the hardware search wizard at the appropriate point on my server share point, and it does the rest for me. Thus far this has worked wonders and I have to search for relatively few things on my own.

Tie down your cables with zip-ties and nail them into place, with power and networking similarly nailed into place

While it may at first seem a little obsessive-compulsive (CDO) to wrap up all of your cables in zip-ties and then to nail them into place, once things get moving in a lab the clear organization keeps things from becoming total chaos. I inherited a lab in such a state and whenever there was a problem with the wiring (and there was a new one every other day), and wasted enough time trying to solve things one-wire-at-a-time that I finally decided to declare the lab closed for a day, ripped everything out, and redid the whole thing. Since then I have not wasted any more time with bad connections, and the computers all have nice slots with all of the cables ready-to-go when they arrive. And since there are clear spots for the computers, when other people move things around it still stays pretty neat.

I went out and got a bunch of carpet tacks (HomeDepot) and inch-wide elastic bands (hobby store). I pre-drilled two groups of holes a couple of inches apart (mine are on the wider side as I need KVM cables) on the underside of the tables where I wanted a computer to go, and then cut a length of elastic band a little shorter than that distance so it had to stretch a little to fit over, and used the tacks to hammer them in creating a harness for my cables. Then I put in the screws to hold up the power (one surge protecter per station), and networking equipment. Each of those went up so that the screws fit into the holes on the back of the equipment, and then a single nail put at the appropriate end so that it could not slide back off.

Then I used small zip-ties to bind the network, power, and KVM cables for most of their length (zip-ties every 8-14 inches, leaving 8-10 inches on the computer end, and more at the other end), then slip the bundle through the elastic loop and plug things in. Then bind up the extra cable into bundles with bigger zip-ties. Do this the right way once and you can stop dealing with cables for a long time.

Buy three or four toenail clippers and use them to clip zip-ties

For about a doller a piece you can get the large toenail clippers form places like Target. They have both the standard clipper style and the scissor style and I recommend buying both. The clipper style (especially the big toenail ones) are great for taking off the used end of the zip-tie without leaving sharp edges, and the scissor type are good at removing zip-ties from cables in tight spaces. I keep a few in each lab I run ready whenever I need them. Plus you get some funny moments when people try to figure out why there are toenail clippers sitting in the lab, or even funnier moments when you catch people using them for their original purpose.

CDO is like OCD (Obsessive Compulsive Disorder), but with the letters in their proper alphabetic order. *grin*

Writing screensavers for 10.5 and 10.6

I needed to write a screensaver recently, one that would work on both 10.5 and 10.6, but ran into a problem when I tried to build such a project on XCode 3.2. The problem is that with 10.6 the program that does the job of running the plugins that are ScreenSavers (.saver modules) runs in 64bit mode when possible. And since .saver modules are plugins, they also need to be able to run in 64bit mode. However, the 10.5 version of the ScreenSaver.framework only had 32bit modes (PPC and i386), and so you can't build x86_64 (64bit Intel) modules against it. Conversely things changed enough between 10.5 and 10.6 that modules build against the 10.6 SDK will not run on 10.5. Rock meet hard place.

The solution to this problem seems obvious in retrospect: build the 64bit module against the 10.6 SDK, and build the 32bit modules (PPC and i386) against the 10.5 SDK. I am proud to say that I came up with the idea of building two different .saver modules, and then choosing which one to run... but I did not want the hassle of having to maintain two codebases, or even two compiles. But luckily for me there are people who are much smarter than me who already solved the problem. So I tried closing my XCode project and trying to translate what Warren Dodge wrote about his older project file into my newer one. Eventually I got what worked, and then discovered how to to do it in the XCode GUI:

Step 1: Open the project inspector

To do this click on the blue icon at the top of the "Groups & Files" column in the main window of your project. Then click on the "Info" button on the toolbar, or select "Get Info" from the File menu. The should look something like this:

Step 2: Change the "Base SDK for all configurations" to 10.5

Circled in red on the figure 1, the "Base SDK" serves as a master switch for all of the configurations (usually "Debug" and "Release"). While the label might imply that it only sets the targeted SDK, this also changes the build target.

Step 3: Setup the Configurations

Click over to the "Build" tab, then make sure that the "Configuration" selector (circled in red) is set to "All Configurations". This will make sure that the rest applies to both "Build" and "Release" configurations. Then type "10.5" in the search box (to the right of the "Configuration" setting). This will narrow down the really big list to just a few.

Setp 4: Add the BaseSDK setting

Click on the "Base SDK" entry under the "Architectures" section to highlight it (circled in green). Then click on the gear box at the bottom of the page (circled in purple), opening a drop-menu. From this menu select "Add Build Setting Condition". That will add a row underneath the "Base SDK" row. Clicking on that will pop look something like this:

Select "Intel 64-bit" from the menu, then click on the "Mac OS X 10.5" from just to the right of it (the lower one of the two together, not the default one), and from its pop-up menu select "Mac OS X 10.6".

At this point when the compiler goes to work it will use the 10.5 SDK for PPC and i386, but use the 10.6 SDK for x86_64. We are almost there!

Step 5: Add the Deployment Target setting

This final step is very similar to that for the Base SDK. The difference is that you highlight the "Mac OS X Deployment Target" line under the "Deployment" section, then once you have added the "Build Setting Condition" with the gear menu you change the "Any Architecture" selection to "Intel 64-bit" and the "Compiler Default" selection to "Mac OS X 10.6". You can leave the "Any SDK" alone, or set it to "Mac OS X 10.6" either way it will have the same effect.

Now you can close the "Project Info" box, and when you hit compile the settings will be right to have the output work on 10.5 or 10.6 without going through any other tricks (other than getting your code right). The same trick would probably work with setting things to 10.4 for 10.4 compatibility, but my project did not require that (and then I could not have used ObjC 2.0 tricks that I like to use).

Apple refuses to solve my installer problem

A while ago I wrote about two problems I was having with the 10.6 installer and then a little while later how I had solved one of them with a bit of a hack. I had filed the other one as a issue with Apple, and I got a response back on the bug recently.

As a quick refresher: The problem is that there are a number of installers, both from Apple and from third parties, contain scripts that make the assumption that you are always installing on the root volume. Obviously this is a problem with things like InstaDMG, DeployStudio, or even System Image Utility. I managed to solve this class of problem for 10.5 by wrapping the installer in a chroot jail, a solution that worked better than I had hoped. Unfortunately the 10.6 installer breaks when I try to wrap it the same way. My best guess is that it is dying while trying to enumerate the volumes so that VolumeCheck scripts can run... but when using the command line version they are never run.

The answer I got back from Apple on this was a single line of text telling me that installers need to be written correctly to target non-boot volumes. I am more than a bit disappointed and angry at this response from Apple, as it means that this problem will not be fixed, and those in charge of fixing it do not see it as a problem and see the answers to this situation as lying with others. There are a few problems with this attitude:

1. Apple has proven on more than a few instances that it is not capable of consistently authoring packages that do the right thing in these cases. iTunes, the iLife Updaters, and iWork installers are just a few cases. If Apple can not get this right then what hope is there that third parties will get it right (even accepting that Adobe will never get within visual distance of getting it right).
2. Apple does have a product that needs exactly this setup: System Image Utility. Both in the NetRestore-from-installer and the NetInstall paths the installer needs to work on non-booted volumes with a variety of packages. That the SIU team has been very slow to acknowledge problems with their approach in this area is frustrating. They got the iTunes installer finally working (Ya!), but it took me yelling at them personally at a conference for it to happen (Booo!). I expect more from Apple.

So what do I do now? How can I overcome this class of issue? I have a few possibilities:

1. Come up with some brilliant solution that tricks the 10.6 installer into working inside a chroot jail
2. Create a system that unwrapped every form of .pkg (and there are a number of formats), replaces all of the scripts with a version that is wrapped with a chroot jail
3. Write my own version of the installer that does things right
4. Yell at Apple for a while, and get other people to do so as well in the hope that this decision will be reversed
5. Convince every .pkg author out there to write their installers to work on non-booting images
6. Just accept that some installers will never work in InstaDMG, and hope that one of them is never in a software update, or something else absolutely required (ie: give up)

The first three items are ones that I could in theory do (although there is probably no hope for the first, and the latter two are going to be difficult). Of the last three the fourth item is the one that I wish would work (it would be the best solution), and the last one is the one I am most afraid of.

So, if anyone is reading, and would like to do something, please tell Apple how much value there is to you in installers working on non-boot volumes. If you would like to mention the Radar number 7699285 that would be great.

Bug with hdiutil and symlinks

I got an error report from an InstaDMG user who was using symlinks to point at their installer DVD. I had never tried out using symlinks for that, and so tried it out, successfully, and wrote back saying that it was working for me (with a much newer version of InstaDMG), and that they should probably be using the -I flag to specify the disk rather than use a symlink. But I did do some more testing while I was setup this way, and ran into a problem just once after a couple of runs of InstaDMG.

It turns out that there is a bug in hdiutil, at least on 10.6.4, when it comes to resolving symlinks. But this bug only seems to come out on some percentage of runs, and even then the percentage seems to vary with the hardware (or some other variable). On my iMac8,1 I see it 15-25% of the time, while with my iMac5,1 I only see it 0.5% of the time. Granted the older iMac is running a brand-new install, where the newer iMac is running an OS that I constantly beat on.

I have reported this back to Apple as Radar number 8111753, as well as on OpenRadar. But I am curious if other people are getting error numbers like I am, so if you would like to run the following script a few times on your system and post the results in the comments that would be great.

#!/bin/bash

# print the system information
/usr/sbin/system_profiler SPHardwareDataType SPSoftwareDataType | /usr/bin/awk '/Model Identifier:|System Version:/ { $1 = ""; $2 = ""; gsub(/^[ \t]+|[ \t]+$/,""); print }'

# create a temproary folder with three items in it
TEMP_FOLDER=`/usr/bin/mktemp -d /tmp/hdiutilBugTest.XXXX`
/usr/bin/touch "$TEMP_FOLDER/a"
/usr/bin/touch "$TEMP_FOLDER/b"
/usr/bin/touch "$TEMP_FOLDER/c"

# create a compressed image from the temp folder
/usr/bin/hdiutil create -srcfolder "$TEMP_FOLDER" "$TEMP_FOLDER/testImage.dmg" 1>/dev/null

# create the symlink to the image
/bin/ln -s "testImage.dmg" "$TEMP_FOLDER/symlink"

SYMLINK_PATH="$TEMP_FOLDER/symlink"
ABSOLUTE_PATH="$TEMP_FOLDER/testImage.dmg"

PATHS[0]="$SYMLINK_PATH"
PATHS[1]="$ABSOLUTE_PATH"

REPEAT_COUNT=1000
IFS=$'\n'
for THIS_PATH in ${PATHS[@]}; do
 echo "Working on: $THIS_PATH"
 FAILED_COUNT=0
 i=0
 while [ $i -lt $REPEAT_COUNT ]; do
  /usr/bin/hdiutil imageinfo "$THIS_PATH" 1>/dev/null 2>/dev/null
  if [ $? -ne 0 ]; then
   let FAILED_COUNT=FAILED_COUNT+1
  fi
  let i=i+1
 done
 echo "  Failed $FAILED_COUNT out of $REPEAT_COUNT times"
done

# delete the temp folder
if [ ! -z "$TEMP_FOLDER" ] && [ -d "$TEMP_FOLDER" ]; then
 /bin/rm -rf "$TEMP_FOLDER"
fi