Nate Silva

How to destroy your database in two clicks using Querious

Thu, 03 Jun 2010 14:20:00 -0700

Open your database in Querious. Then:

Click “Users.”
Say to yourself, “I wonder what the button with two squares on it does.” Please don’t click the button. Because the instant you do, all of your rights are revoked and there is no way to recover them.

I hope you have another admin login. Otherwise you’re screwed.

Yes, I did this on an Amazon RDS instance. Thank goodness it was a test database and I was just experimenting with Querious. I never imagined such a dangerous operation would happen without warning.

Why on Earth doesn’t Querious have a warning or an “Apply” button so the change doesn’t happen instantly?

See the equivalent screen in Sequel Pro, which has an “Apply” button, making it much safer.

(I suspect you could also screw up your database by un-checking some of the other boxes. Again, there is no “Apply” button in Querious, so you will instantly change your rights.)

How to build the pscyopg2 Python PostgreSQL interface on Mac OS X Snow Leopard

Mon, 29 Mar 2010 10:03:00 -0700

These instructions may work on earlier versions of OS X, but I’ve only tested on Snow Leopard.

A common problem for Python programmers using Mac OS X is how to install psycopg2, the standard Python interface for PostgreSQL.

If you Google for the instructions, you’ll get all kinds of advice, most of which involves using MacPorts or installing a complete server installation of PostgreSQL on your Mac.

I don’t want to do that.

I don’t want to run a database server on my local Mac. I want only the necessary PostgreSQL client libraries and I want to use the OS X native build system, not MacPorts.

It’s actually easy to do this. The instructions may look complicated, but it’s only because I’ve added a lot of explanation to them.

Step 1: Install the PostgreSQL client library

Download the PostgreSQL source code from http://www.postgresql.org/ftp/source/. Don’t download Mac OS X binaries, as that would be a complete server installation.
Un-tar the source code and cd into its directory. Then run:
- CFLAGS="-arch i386 -arch x86_64 -arch ppc" ./configure --with-openssl
  - This CFLAGS setting will get you a universal binary. Without it you would only get a 64-bit Intel binary (assuming you have a 64-bit Intel Mac), or a 32-bit Intel binary (if you have a 32-bit Intel Mac). We also include PowerPC support, for the unlikely event that you have a program that runs under Rosetta that needs to use the PostgreSQL libraries.
We are going install only the client parts of PostgreSQL. These instructions are based on the PostgreSQL manual, section 15.5.:
- sudo make -C src/bin install
- sudo make -C src/include install
  - In my experience, the error about utils/fmgroids.h can be ignored.
- sudo make -C src/interfaces install
- sudo make -C doc install

Step 2: Install Psycopg2

Download and un-tar the Psycopg source code.
Edit the psycopg2 setup.cfg file. Add or update the following lines:
- have_ssl=1
- pg_config=/usr/local/pgsql/bin/pg_config
Build and install it:
- python setup.py build
- sudo python setup.py install
Test it:
- $ python
- >>> import psycopg2
- >>> psycopg2.__version__
  '2.0.14 (dt dec ext pq3)'

Looks good!

An over-compressed JPEG file with bad artifacting. A good...

Tue, 08 Dec 2009 16:36:50 -0800

An over-compressed JPEG file with bad artifacting. A good example of why JPEG should never be used for text.

Guess where it’s from? (Answer)

How to make an Intel wireless card connect to an Apple Airport base station in WPA2 mode

Wed, 18 Nov 2009 10:31:00 -0800

With some notebooks (including a Lenovo Thinkpad and a recent Gateway model) we’ve had problems getting Wi-Fi connected to our Airport Extreme base station.

In the past, we enabled both WPA and WPA2 on our base station and then manually set the laptops to use TKIP (WPA) mode (previous post describing how to do this). This worked until we decided to turn off WPA mode, supporting only WPA2, due to newly-discovered vulnerabilities in WPA/TKIP.

Once again, our notebooks with Intel 4965AGN and Intel 5100AGN wireless cards were unable to connect.

Then one of my awesome coworkers, Matt, discovered a setting that fixes everything: turn on an option called “FIPS compliance.” Here’s what to do:

Open Network and Sharing Center (NSC).
From the list at the left-hand side of NSC, choose Manage wireless networks.
Right-click on your network and choose Properties. (If your network is not in the list yet, you will have to click on “Add” and enter your network information and password before continuing.)
On the Security tab, set Security Type to WPA2-Personal and Encryption Type to AES.
Still on the Security tab, click the Advanced Settings button.
Check the box that says Enable Federal Information Processing Standards (FIPS) compliance for this network.
Click OK several times to close the dialog boxes.

How to use bpython as your Django shell

Tue, 25 Aug 2009 06:23:00 -0700

$ DJANGO_SETTINGS_MODULE=settings bpython

Lenovo Thinkpad with Intel wireless won’t connect to an Airport Extreme base station

Mon, 03 Aug 2009 06:28:00 -0700

Update: This method is no longer recommended due to a recently-discovered vulnerability in WPA/TKIP. I found and blogged about a better solution here: How to make an Intel wireless card connect to an Apple Airport base station in WPA2 mode.

We had a problem with a Lenovo Thinkpad T500 notebook that has Intel wireless networking built-in.

It could not connect to our Airport Extreme wireless network that is secured using WPA/WPA2 encryption. When you try to connect, the Windows wireless status window alternates between “Connecting” and “Acquiring IP address” forever.

The solution is to change the authentication method from AES to TKIP. To do this:

Open Windows’ Network Connections window. One way to do this is to open Control Panel > Network and Internet Connections.
Right-click on the wireless network connection and choose “Properties.”
On the “Wireless Networks” tab, under “Preferred Networks,” locate your network and click “Properties.”
Change “Data Encryption” from AES to TKIP and click OK to close the dialogs.

PHP sessions timeout too soon, no matter how you set session.gc_maxlifetime

Thu, 23 Jul 2009 14:31:00 -0700

The scenario

You’re running Debian Linux or Ubuntu Linux. You want PHP sessions to last longer than the default 1440 seconds (24 minutes). So you do this:

ini_set('session.gc_maxlifetime', 10800);    # 3 hours

With this setting, sessions should remain active for at least three hours, as long as users don’t close their browser.¹

But no matter what you do, sessions keep getting deleted after 24–54 minutes. It seems PHP is ignoring the gc_maxlifetime setting.

Why this happens

Debian and Ubuntu Linux override PHP’s session behavior. If you look closely, you’ll see that session.gc_probability is set to 0, meaning PHP’s garbage collection will never run. Instead, there’s a Debian-specific cron job in /etc/cron.d/php5 that runs every 30 minutes!

The cron job does garbage collection based on the global session.gc_maxlifetime in php.ini. The session.gc_maxlifetime in your app is ignored.

The solution

While you could disable the cron job and/or modify php.ini, I’d prefer to fix the problem without modifying system defaults. A better solution is to create your own sessions directory, somewhere outside the normal one, and then locally enable PHP’s session garbage collection.

To do this, set session.gc_maxlifetime, session.gc_probability, session.gc_divisor, and session.save_path:

# Session lifetime of 3 hours
ini_set('session.gc_maxlifetime', 10800);

# Enable session garbage collection with a 1% chance of
# running on each session_start()
ini_set('session.gc_probability', 1);
ini_set('session.gc_divisor', 100);

# Our own session save path; it must be outside the
# default system save path so Debian's cron job doesn't
# try to clean it up. The web server daemon must have
# read/write permissions to this directory.
session_save_path(APP_PARENT_DIR . '/sessions');

# Start the session
session_start();

¹ You could change this setting in php.ini, but there are several reasons not to. php.ini changes things globally, not just for your application. And if someone else changes it, your application could break. Finally, if you ever need to deploy your application on another server, you would have to remember to change the setting in the new server’s php.ini. I always prefer ini_set() and never modify php.ini except as a last resort. ↩

How to get iChat video chat working behind a SonicWALL firewall

Tue, 04 Nov 2008 10:00:00 -0800

We couldn’t get iChat’s video chat feature to work behind our SonicWALL PRO 3060 firewall. Whenever someone tried to connect, iChat would say it “did not receive a response.”

The solution was to enable the SonicWALL’s “consistent NAT” feature. To do this:

Log in to the SonicWALL administrative web page.
From the menu at the left side of the page, choose VoIP (which stands for voice-over-IP).
In VoIP > Settings check the box that says “Enable consistent NAT.”
Click Apply.

That was all we needed to do to enable a user on a Verizon 3G card to video conference with a user on our internal LAN.

Microsoft Internet Authentication Service (IAS) dies or won't start

Thu, 18 Sep 2008 13:26:00 -0700

Problem

We use the Microsoft Internet Authentication Service (IAS) to provide RADIUS authentication for our wireless network.

From time to time we would notice that IAS had stopped working on one of our servers. In Event Viewer, we would see the following error:

“Service Control Manager, Event ID 7023: The Internet Authentication Service terminated with the following error: Only one usage of each socket address (protocol/network address/port) is normally permitted.”

You could also click on “Internet Authentication Service (Local)” and notice that the green start button was enabled, indicating that the service was not yet started.

You could start the service, but a few seconds later it would stop.

Cause

The problem was caused by the Microsoft DNS server hogging one or more of the ports RADIUS needs! Those would be UDP ports 1812, 1813, 1645 and 1646.

To identify this problem I had used a free program called NirSoft CurrPorts. I could have used Windows’ built-in netstat command, but this is one of those times when a GUI is nice — especially because you can sort by port number.

Why was DNS using those ports? It was because of the recent DNS security update (the one that fixes the Kaminsky port randomization bug). It was Microsoft security update 953230 (MS08-037).

This problem is described in KB56188.

Solution

The solution is described in KB812873.

The short version, for Server 2003, is:

In the Registry Editor, go to HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\
If there is a value called ReservedPorts, open it. Otherwise, create it as a “Multi-String Value”.
Add the following ports to the list: 1812-1813 and 1645-1646

My thoughts on Google App Engine

Tue, 24 Jun 2008 11:24:00 -0700

I recently tried Google App Engine.

Pros and cons

Pros

The canned environment is really nice. It’s great to focus on development instead of setting up infrastructure.
Love Python. Like the Django-based templating engine.
Bandwidth allocation is reasonable for small- or medium-sized apps.
The data store is great!
- Because it’s non-relational there’s no ORM layer to get in the way. Once you understand it, it’s intuitive.
- Reading from the datastore is nice and fast. (But not writing — see next point.)

Cons

The data store sucks!
- There’s no way to import/export data.
  - So you have to write your own.
- Writing to the datastore is extremely slow, so if you write your own import routine, you need to break up your imports so they don’t time out. It took a couple hours to manually load around 2,000 items.
- The Bulk Data Upload utility is essentially useless, because it can’t handle Unicode and it times out when importing large amounts of data. If I wasn’t importing large amounts of data, I wouldn’t need a bulk upload utility, no?
During beta, the quotas are so restrictive that it’s only useful for trivial applications at this time.
- You can’t retrieve more than 1,000 rows from a query (and doing so would probably time-out your request).
- Apps are limited to 500 MB total storage and 1,000 files.
- Developers can only create three apps and you can’t delete or rename an app, so use your three wishes wisely.

Other

If you’re using Google Apps For Your Domain to authenticate, you need to separately set the app up in your domain, which is poorly documented. It seems obvious, but it was confusing when I tried it.

Moving apps to GAE

I considered moving three in-house apps to GAE, but none were feasible.

Build query tool

The first is a build query tool. It lists which components are present in our software builds, organized by category.

Builds have anywhere from 10 to 1,500 components. Our users sometimes pull up the larger builds. The list loads quickly because it’s plain text.

With GAE there’s no way to retrieve it (1,000 row limit) and even if there were, it would probably time out. We could get around this by introducing a search feature, and showing just subsets of the data. But we would lose the ability to browse over the entire contents of a build at once.

And then there’s the nightmare of updating the datastore. Let’s say we add a new build that incorporates a few hundred components. How are we going to update the datastore in an automated fashion? It’s certainly possible with HTTP POSTs but it would take ages because of timeout issues and how slow it is to write to the datastore.

File upload utility

The second is an upload utility. Files are uploaded and then “processed,” which involves storing them into an Amazon S3 bucket and writing a database record.

This was out of the question due to the 10 MB file size limit. And the 500 MB overall limit might be an issue — we occasionally get giant files up to ~750 MB. (The 1,000 maximum file count is fine; once files are processed they would be deleted from the GAE application space.)

Finally, getting a responsive progress bar during an upload is a tricky problem that generally requires some help from the server-side. I don’t know how to do it with GAE, or if it’s even possible. Of course it’s possible to upload without a progress bar, but users need the reassurance of a progress bar when dealing with huge files.

Historical data utility

The third is a historical data utility. It contains old (going back to 1995!) customer records, notes and invoices. These come from an obsolete accounting system that has been decommissioned, so the web UI is the only way our users can get to it. While not frequently used, it is sometimes handy to pull up old data and chart trends.

Application wise, this should work with App Engine. The row, data size and other limitations don’t come into play.

Unfortunately, there are a few hundred thousand rows in the database. It’s around 125 MB so it would not hit quotas but uploading that data could take weeks.

Summary

Bottom line: I would love to use Google App Engine, especially for in-house apps where we can authenticate against our Google Apps domain. But it’s too limited right now — at least for the types of apps I tried. I’m still on the lookout for other apps that might be a better fit.

How to tell if a Mac OS X application is “Cocoa” or “Carbon”

Mon, 02 Jun 2008 11:55:00 -0700

There are two ways that software developers can create Macintosh applications — known as Carbon and Cocoa. As has been said many times before, it doesn’t really matter to users if an application is Carbon or Cocoa, although developers do tend to prefer one or the other when writing applications.

A quick way to identify Carbon and Cocoa applications is to drag them to the bottom of the screen.

Carbon applications will not let you drag their title bar off the bottom edge of the screen.
Cocoa applications will let you drag the title bar partially off the bottom of the screen, but as soon as you let go of the mouse, it will bounce back into view.

This isn’t always accurate, as some well-known Carbon applications (the Finder) have Cocoa-like behavior.

And since it’s possible for Cocoa applications to call Carbon and vice-versa, you might say that many apps are not entirely Cocoa or Carbon, but a mix of the two.

Python PostgreSQL interfaces: pg8000 vs. psycopg2 micro-benchmark

Tue, 20 May 2008 14:14:00 -0700

I have great hopes for the pg8000 project, which is a PostgreSQL interface written entirely in Python.

The current standard in this category is something called psycopg2. It depends on libpq (the official PostgreSQL client library). While it works great, it can be difficult to get it running on some platforms (ahem, OS X), because you need to download all of PostgreSQL to get the client library. You can install just libpq if you can suss out the correct build parameters, but it’s still quite a hassle when deploying software on random client machines.

What pg8000 offers is no dependencies. I can do database stuff without having the client install anything other than Python itself.

But this comes at a cost; namely: it’s very slow.

I used one of our build scripts that records metadata from a directory tree into a PostgreSQL database. The directory tree contains more than 3,000 files. The script reads some data from each file, as well as its last modification time. For each file, it then executes two statements: one SELECT to validate the data, and one INSERT to plunk it into the database. A single commit() is executed at the very end of the whole process.

Using psycopg2 gave the following results:

Processing complete. 3230 files total.

real    0m28.654s
user    0m4.419s
sys 0m6.582s

With pg8000 I got the following results:

Processing complete. 3230 files total.

real    7m53.424s
user    0m13.516s
sys 0m8.267s

So, for this test, pg8000 took 16 times as long to complete.

I haven’t figured out why it’s so much slower. Perhaps pg8000 is opening transactions for each statement instead of using a single transaction. Or maybe an all-Python database interface is just destined to be slow.

For everyday use (i.e., you’re not executing thousands of consecutive statements at once) you probably won’t notice much difference using pg8000. As always, you need to profile to find out where the bottleneck is. If your application only needs to do a few statements per minute then the speed of your database interface probably isn’t a constraint.

When downloading an .EXE file in IE, it loses its file extension

Mon, 05 May 2008 07:28:00 -0700

Internet Explorer has odd behavior when downloading certain files. If you have an .exe file, but it’s served by a URL with a query string, IE will lose the file extension.

An example might be easier to understand.

Let’s say you’re using Amazon S3 to serve your downloads. You’ve got a file, myfile.exe, that your customer wants to download. Using S3 you generate a URL that allows them to download the file, and you can set the URL to expire after a certain time. The resulting URL might look like this:

https://nate.s3.amazonaws.com:443/myfile.exe?Expires=1210007779&AWSAccessKeyId=1HHMQMPRV9GFWH6BTP82 &Signature=5dfcG5OC9BSkDFSzjxQGjlbKNi0%3D

Notice that the URL contains /myfile.exe followed by some query string parameters (everything after the question mark is the query string parameters).

Most web browsers will interpret this as being a request for the file myfile.exe.

Internet Explorer will interpret this as being a request for a web page served by an application on the server called myfile.exe. The resulting download will be called myfile (instead of myfile.exe), which doesn’t work.

There’s a workaround, which is to add the filename to the end of the query string, like this:

https://nate.s3.amazonaws.com:443/myfile.exe?Expires=1210007779&AWSAccessKeyId=1HHMQMPRV9GFWH6BTP82 &Signature=5dfcG5OC9BSkDFSzjxQGjlbKNi0%3D&filename=myfile.exe

There’s no reason in the HTTP standard why this should be required; it’s just a way that Microsoft developed to work around the bug in IE.

More information, including Microsoft’s creative interpretation of the HTTP standard, can be found in Knowledge Base Article 221805. The article says the bug is fixed, except not for .exe and .dll files! This is another case of inconsistent behavior in IE.

How to get some HP network printers to work with Mac OS X Leopard

Wed, 20 Feb 2008 12:25:00 -0800

Some HP network printers — such as the HP Color LaserJet 3600n and 3600dn — don’t have what Apple calls “modern” printer drivers. As a result, you may not be able to print over the network. This will happen if both of the following things are true:

The printer doesn’t have a modern driver.
You are not using Bonjour to print; for example, if you are on a different network subnet than the printer.

The correct driver won’t even be available in the driver list. Apple has a list of supported printers; an X in the right-hand column indicates modern drivers.

The good news is that there’s a solution. For this to work the printer must have built-in networking and it must be an HP printer (Canon and Epson owners see my note below).

In System Preferences, open the Print & Fax Preference Pane.
Click the plus sign to add a printer.
In the dialog that appears, click the “More Printers” icon at the top.
In the More Printers dialog, choose “HP IP Printing” from the drop-down list.
- If your printer is on the same subnet as your Mac, you should see it listed. Select it and click Add.
- If you don’t see it listed, click the “Manual” tab and enter the printer’s IP address, click “Connect”, then “Add”.

That’s it. Most of the time the printer model will be automatically detected and you won’t even have to select a driver.

Note for Epson and Canon owners

The steps above are for HP printers only. But the “More Printers” dialog also offers options for “Canon IJ Network” and “Epson TCP/IP” printing. I have no way to test these, but if your printer has built-in networking choosing one of these may work for you.

Engineers and early adopters vs. the customers

Fri, 15 Feb 2008 05:50:00 -0800

“We thought it was a mistake and made our engineers check the logs again,” said Vic Gundotra, head of Google’s mobile operations.

Google receives 50 times as many searches from iPhone users than from any other mobile handset. It’s the very definition of usability: how many people actually use their phones to browse the web. Apparently only iPhone users.

Products that are popular with engineers and early adopters—the previous generations of smartphones, loaded with features—aren’t necessarily what customers want. They want usability. It looks like Apple has figured that out.

Time Machine only runs if your MacBook is plugged in

Fri, 25 Jan 2008 07:25:00 -0800

Time Machine is a great feature of Apple’s computers. It backs up your Mac every hour, as long as you have a backup disk connected to the USB or Firewire port. Because it only needs to back up whatever changed in the last hour, it only takes a few seconds to do.

But on my wife’s MacBook it was not doing automatic backups:

The “Next Backup” field just contains two dashes.

The solution was easy: plug in the computer.

It turns out that Time Machine doesn’t run automatically when your notebook computer is not plugged in. It does this to preserve battery life. As soon as you plug it in, the “Next Backup” time shows up and automatic backups will continue.

Notes on using Cocoa-Python in Mac OS X Leopard

Wed, 07 Nov 2007 02:24:00 -0800

I’ve been looking at Leopard’s amazing new Cocoa-Python support. It uses PyObjC 2.0 to give Python programmers access to the Cocoa frameworks that Mac OS X is built on.

Working from the Apple tutorial, which is based on an older version of PyObjC, I was able to build a working application. But I hit a few roadbumps and those are what I want to document here for other would-be Cocoa-Python programmers.

The first problem was figuring out how to instantiate my Averager object in Interface Builder. I found the solution on the macosxhints Forums:

Drag an instance of NSObject from the Library into your xib file.
Click on the new instance of NSObject and bring up the Identity Inspector (⌘-6 or find it in the Tools menu).
Set the object’s class name to your class name, which is Averager if you’re following the Apple tutorial.

The second problem was this error:

This was caused by not importing the Averager class. To fix it, add the following to your main.py file:

from Averager import *

Some tips:

The debugger console window (Shift-⌘-R) is your friend.
Beware of Xcode’s tab handling. You can force it to use spaces under Preferences > Indentation.
If you are doing the tutorial and you get “ValueError: invalid literal for float()”, it’s because you’ve passed something that doesn’t look like a number — most likely a comma. Python’s split method only recognizes lists separated by spaces. In the debugger console you can see what exception is raised. You can of course catch the exception and handle it in your Python code.

How to make readline support work in IPython on Mac OS X Leopard

Tue, 30 Oct 2007 12:47:00 -0700

Many Python programmers use the IPython shell for interactively testing their code.

But if you recently upgraded to Mac OS X Leopard then you may have noticed that a key feature — tab-completion — doesn’t work.

It doesn’t work because IPython uses the GNU Readline library, and Mac OS X doesn’t include that. Instead, OS X ships with a similar library called Editline.

You can make IPython work with EditLine by adding the following lines near the top of your ~/.ipython/ipy_user_conf.py file:

import readline 
readline.parse_and_bind ("bind ^I rl_complete")

Note: You must be using a recent version of IPython. You can install the latest version by typing sudo easy_install ipython from the OS X command-line.

(Found in this message thread on the Pythonmac-SIG mailing list)

How to fix an SSL certificate error in Exchange System Manager

Sun, 21 Oct 2007 02:10:00 -0700

For the longest time I’ve had this error in Exchange System Manager when managing public folders:

This can happen when using Outlook Web Access if your webmail URL doesn’t match the server’s internal name. Exchange System Manager expects your SSL certificate to match the server’s internal name. OWA users — including Windows Mobile devices — expect the certificate to match the server’s public URL.

One solution is to manually change the SSL certificate every time you need to manage a public folder, then change it back when you’re done. That gets old quickly.

Another solution, suggested in a knowledge base article, is to turn off SSL for the IIS Exadmin virtual root. This didn’t work: SSL was turned off for Exadmin but the problem remained.

Here’s an alternate solution that works:

Run adsiedit.msc.
Navigate to: CN=Configuration > CN=Configuration,DC=YOURSITE > CN=Services > CN=Microsoft Exchange > CN=YOURDOMAIN > CN=Administrative Groups > CN=first administrative group > CN=Servers > CN=YOURSERVER > CN=Protocols > CN=HTTP > CN=1 > CN=Exadmin.
Right-click on CN=Exadmin and choose Properties.
Find the attribute called msExchSecureBindings and click Edit.
Remove the value :443: from the list.
Click OK twice to close the dialog boxes.

(Found on Jim McBee’s Mostly Exchange Web Log)

How to make Safari work behind a SonicWALL firewall

Mon, 08 Oct 2007 08:26:00 -0700

Do you have a Mac behind a SonicWALL firewall? Do you find that some URLs (like certain pages at the New York Times) won’t load in Safari, but they work in Firefox? Is the problem worse when you’re behind a secondary router, such as a wireless router?

If so, Edward Marczak of Tech Zendo has the solution for you.

The basic solution is to access the SonicWALL’s hidden diagnostics page at http://your.ip.address/diag.html and turn off the checkbox that says “Enforce Host Tag Search for CFS”.

It turns out the SonicWALL drops some web connections where the HTTP request headers are split across multiple packets. It does this to make it harder to bypass content filtering, but it can cause Safari’s legitimate traffic to be dropped.

Thanks Edward!

Followup: 2007-10-10

This support document from SonicWALL explains the problem in more detail. WebKit (Safari) is waiting for a TCP ACK before sending out the next packet; apparently it is not supposed to do that.

Technorati Tags: mac, networking, safari, sonicwall