I've discussed Idea Mining before: every idea that pops into your head, jot it down. When you're looking for inspiration, or have some time to develop some idea, poke around in your mine and see what grabs you.

I started using Evernote, which had the advantage of syncing, but recently switched to Notational Velocity. It lacks syncing, but is much more lightweight. When I'm sitting at my computer, the amount of friction in jotting down new notes is minimized. It's a great tool, but by switching to it, I gave up syncing. This means, for example, I can't jot ideas on my phone and have them show up on my computer.

At least, not by default. But I'm not going to be thwarted by such details. So, I sat down, banged out some settings and scripts that let you sync emails in your GMail account to Notational Velocity.

What follows is a high level overview of the steps. It's not hard to do, although it took a little doing to figure out.

You're going to need the following things:

• OSX Leopard or later
• Developer tools installed (on your OSX install disk, or downloaded from Apple)
• Notational Velocity
• A Gmail account with IMAP enabled

Configure Notational Velocity

Once you have Notational Velocity installed, you're going to need to make one settings change: on the Notes tab of the Preferences window, change "Store and read notes on disk as:" to Plain Text Files. Note also the Folder that you store your notes in on this pane.

Configure Gmail

You need to enable IMAP from the settings pane.

You also need to configure a filter for a pseudo-address. In the filter section of settings, create a filter along these lines:

Matches: to:([your email]+mine@gmail.com)
Do this: Skip Inbox, Apply label "Notes"

This means that, every time you send an email to [your address]+mine@gmail.com, it's actually going to come to your account, but get labeled with Notes (which is a folder in IMAP).

Get SSH Prepared

This step, sadly, does involve a little command-lineage. You can only connect to Gmail's IMAP interface over SSH, and fetchmail needs the certificates where it can find them for this to work.

From /Applications/Utilities fire up Terminal, and type the following:
openssl s_client -connect imap.gmail.com:993 -showcerts

This command will output a big pile of of text, but up at the top, you'll see a large block of output that starts with:

-----BEGIN CERTIFICATE-----
(a big bunch of characters here)
-----END CERTIFICATE-----

(there are two of these blocks, you want the first one, which is actually for imap.gmail.com.

Copy that text (including the BEGIN and END lines) and save it to a file called ~/.ssl/certs/gmailimap.pem. You will likely need to create those directories before you can save it.

You need one more certificate, which you can download from here (where it says "Equifax Secure Certificate Authority (Base-64 encoded X.509"), and save it as ~/.ssl/certs/equifax.pem.

To make these certs usable by fetchmail, you have to run one last command at the command line:
c_rehash ~/.ssl/certs.

This step is a lot easier if you know your way around the command line. The bulk of this SSH section was adapted from here.

Notes.pl

Download this file and save it in ~/Scripts (a directory you'll need to create). You'll also need to make it executable, which involves another command-line bit:
chmod +x ~/Scripts/notes.pl

Open that file in a text editor and change the first two lines per the instructions in the file. Basically, you need to tell it where to find your mail (/var/mail/[your mac username]) and your Notes folder (remember I told you to keep track of that).

This script parses your local Unix mail (deep inside OSX, it has nothing to do with Mail.app or Thunderbird or anything you normally use for email) and turns what's there into notes in Notational Velocity.

We're almost done. The last step is to…

Configure Fetchmail

If you're using Snow Leopard or Leopard, fetchmail is already installed. It's a powerful tool for one task: downloading email from remote systems and putting them on yours. Specifically, into the local Unix mail.

Fetchmail works off of a configuration file. Create a file called ~/.fetchmailrc. In this file, you'll need to paste these settings:

set daemon 1300 #the number is the number of seconds between polling attempts

poll imap.gmail.com proto IMAP port 993 user "youraddress@gmail.com" password "yourpassword" keep folder "Notes" ssl mda "/usr/bin/procmail -d %T" postconnect "~/Scripts/notes.pl && > /var/mail/yourmacusername"

Change "youraddress" and "yourpassword" to your Gmail username and password. Change "yourmacusername" to whatever your username on your Mac is. It may contain spaces, in which case you'll need to escape them like so: /var/mail/Remy\ Porter. Note, that is a "\" followed by a space.

To test this much, send yourself some emails that should be turned into notes (don't forget the +mine!), and then, at the command prompt, simply type the command:
fetchmail.

Once it works, we want to make sure it runs every time you log on, so download this file and put it in your scripts folder. From the command line, type:
chmod +x ~/Scripts/fetchmail.command.

Now, in your OSX System Preferences, go to Accounts, select yours, and add a Login Item that runs fetchmail.command.

TADA!

Using It

Now that you've got it set up, send an email to youraccount+mine@gmail.com. The subject line is the note title, in Notational Velocity, and the body is the content of the note. It'll sync however frequently you want it to, by modifying the indicated line in your .fetchmailrc.

Also, if you want to be able to read notes on your phone, you could use something like Dropbox, which is what I do. Also, the way this works is to append to an existing note. There's no real way to edit.

Warnings

I'm not super sure how this will work if you have multiple accounts on your Mac that you want to use this. You may need to instead configure fetchmail.command to run as a system startup item, and then there may be issues with using "~/Scripts"- you may need to use an absolute path. The Googles are your friend, if that's what you decide you want to do.

Words like "breathtaking", "stunning", "astounding", "amazing", and "incredible" are not positive words. Oh, they're often used in a positive context, but in reality, they only convey a sense of wonder and disbelief. They can be applied to something horrifying just as easily as something delightful.

Such has been my experience over the past few days.

Once upon a time, some fragment of the giant behemoth I work for needed an application to manage some inventory tracking and similar tasks. They found an external contractor that would make it for them. In that era, the app was built in VB6 with an MS Access back end. Horrible, yes, but hey, it was the 90s.

Well, now it's time to upgrade. So they go back to the same contractor. "Hey, we want it to be .NET and use SQL Server. Here's a check."

Well, it wasn't that easy, of course. There are policies. Checkpoints. Oversight. I was part of the oversight, and my job was to get the application through our "gates". Since this was a purchased application, we didn't care if it was terribly well designed, so long as it worked. We weren't going to have to maintain it. So we got some design diagrams from the contractor, sanity checked them, and then called it a day.

More fool I.

Last week, I hear, "Hey, that application- they're not satisfied with the support from the vendor. They want to bring it internal. Hey, Remy, find out what you need to do to make it fit with our standards."

Now, when we develop an app internally, or have a contractor develop an app that we intend to support, the auditing requirements are much stricter. We don't care if an outside vendor has to suffer under horrible support issues, but if we're wasting time on stupid support work, we're unhappy.

So, the first step then, would be to audit the application via a code review. I sat down in a room with three other developers, we pulled up the code, and we started skimming through it.

It was breathtaking.

The first, most glaring thing, was that the actual design and the provided diagram had no relationship to each other. That didn't bode well. And it was all downhill from there. As a sampling of some of the amazing things we saw:

• One window in the application has all of the business and data access code bundled up in it. It's like they took all the functions and dumped them into a bucket.
• But not all functions. For example, when that main window wants to log an error message, it calls out to a class called "Utility". When it does that, the Utility class makes a call against the main window. Not only is the code just a disorganized pile in a big bucket, but sometimes the bucket says, "Hey, you gotta go look in another bucket to find that!" and when you do, the other bucket says, "Nope, it's actually back out in the original bucket."
• All of the database access stuff is hard coded strings. With no SQL injection protections.
• It stores passwords in plaintext
• The most stunning thing, however, is what the application does every time it launches. Every time the application launches, it attempts to add columns to a bunch of tables. If those columns already exist, this would cause an error, so it just ignores the errors. Since, once the app has run once, the columns will exist, that means every time a user runs the app, it's trying to add columns to tables for no damn reason.

There's more, but I won't bore you with the details. I wrote a 2,200 word code review document. Normally, these documents follow this form: "File X, Line 128, variable strFoo should be named foo, we don't use Hungarian Notation." For even large projects, they don't tend to get that long, just because they're so terse. This, this was a 5 page essay. I couldn't even call out flaws by line numbers, just because the developer was so clearly incompetent. I've dealt with some bad code in my day, but this is just special.

There's a punchline. They started development using .NET 1.x, and then partway through switched to .NET 2.0. In .NET 1.x, there was no built in FTP functionality. If you wanted to do FTP, you needed to write your own class to do it, or steal some. Microsoft's developer documentation site, MSDN, had an MSFTP code sample, which demonstrated how one could implement their own FTP code. The developer copypasted this- despite the fact that .NET 2.0 had all the functions he needed- and included a class called "MSFTP" in his project.

He couldn't just use the class as it was, however. There were some quirks in our FTP server it didn't handle. And while he was at it, he added a "Dispose" method to handle cleanup, replacing Microsoft's use of the "Finalize" method. This is actually a good .NET technique for technical reasons that are irrelevant here, so he obviously read a book. He didn't read it closely enough, because his Dispose is actually implemented wrong, but that's neither here nor there.

In the comment above his "Dispose" method, he had the gall to include this:
'M$ should have implemented this. Man, this code is sloppy.

When I read that, I just started cracking up.

Until I saw how much my company paid for this. And then I started crying.

Programmers, when they're really concerned with being efficient, will often start talking about "Big O" or "Order". There's a lot of interesting math behind this, but what it boils down to is a very simple concept- how many operations does it take to solve a certain problem, using a certain algorithm? I'm going to simplify things here, ignore the difference between so called "Big O" and "Little O" and not worry too much about the theoretical underpinnings. This also isn't the only metric used to evaluate how efficient an algorithm is, but it's a major one.

For example, if I hand you a deck of cards, and I tell you to find the Ace of Spades, how many cards would you have to look at in order to find it? Assume you start at the top, and keep drawing cards until you find the one you want. In the best case, the very first card you flip is the one you're looking for. That's on the order of one operation, written as O(1). But, that's the best case. What about the worst? If it were the last card you flipped, that would have taken 52 operations- O(52), or, to be really programmery, O(n), where "n" is the number of cards in the deck. By being general, we can extrapolate this to any collection of cards- be it a full deck, less than a full deck, or even a collection of something else entirely- like an address book.

When talking about the "O" of an algorithm, programmers don't like to worry about constants. We don't like "52"- that's far to specific. "n" is much more useful. But similarly, if there were an algorithm with O(2n+1), we're going to ignore the 2 and the +1 and just call in O(n). Doubling it once just isn't a big enough change to really count. It's details and small stuff, and if "n" is sufficiently large, it just doesn't matter.

Now, searching each individual card in the deck isn't terribly clever, but if the deck is shuffled, that's really all you can do. Picking out randomly, going from top to bottom, bottom to top, it just doesn't matter. But if the deck were sorted- now that could be useful. Think about the phone book- it's sorted, and you don't have to search every single name to find the one you're looking for.

If you were writing a computer program to find a name in the phone book, you'd start by going to the name in the middle. If it's the name you're looking for, great, you're done. If not, does it come before or after the name you're looking for? Once you know that- you've just eliminated half of the names. Take the remaining half, and go to the middle. Repeat the previous process until you find the name you want. Each operation (name you look at) eliminates half of the remaining names. Searching a sorted list is very effeceint, and in this case, it's on the order of "lg n". (lg is the log base 2, or "what power do I have to raise 2 to to get "n"). This, by the way, is called "binary search" and is the goto searching algorithm to use in any search.

In our naive search, where we just checked every element, it's O(n). In a deck of 52 cards, that's 52 operations. Not that bad, but what if I had a list of 1,024 items? That's 1,024 comparisons. But if we used the binary search, it's O(lg n)- 10 comparisons. (2¹⁰ = 1,024). That's a savings of 1,014 comparisons, in the worst case.

Programmers tend to break down algorithms into a few major categories based on their Big O.

Stuff that's O(lg n) is very fast stuff. They scale really efficiently to big sets of data (1,000 elements is 10 operations, a million elements is 20 operations, a billion elements is only 30 operations!).

O(n) isn't as liked, and most O(n) algorithms can be turned into O(lg n) if you're very clever and find ways to cheat (for example, if you need to search an unsorted list, it'll always be O(n), but if you sort it first, even though sorting is expensive, you'll make it up if you search a lot).

Sorting is really expensive, in comparison. The best sorting algorithms are O(n*lg n). That means, for a set of a thousand elements, it would take 10,000 operations to sort. Ugh, but some problems just can't be made any easier. O(n*lg n) is the best you'll get for sorting algorithms, and there are whole classes of problems that fall into this category.

Worst is O(n²), or even larger exponents. This is the demon of programmers. For example, the first sorting algorithm that most CompSci students learn is the Bubble Sort. It's very simple to understand and implement, but it's an O(n²) sort. But, to sort 1,000 elements, you have to do 1,000,000 operations. Algorithms that run in "polynomial" time are ones that programmers hate. It's worth noting that there are a lot of problems that, as far as we know, can't be solved any faster. There's a whole family of problems that are called "NP Complete", which, right now, can only be solved in polynomial time. Whether or not it's possible to find a better way to solve these problems is an open question.

This was brought up today because I'm working on Gravitone, an iPhone instrument that uses gravity to generate music. I took a very naive approach to the problem, and simply cycle through all the masses and objects in the world and apply gravity to every other object. My algorithm is O(n*m), where n is the number of orbs in play, and m is the number of masses. Sort of polynomial, and if I were to try and scale it up so that the orbs could effect each other, it'd be O(n²). Ugh, polynomial time? That's expensive. Or is it?

I did some research in gravity simulation, and found that the best algorithms out there run in O(n*lg n). The one I looked at specifically was called the "Barnes-Hut algorithm", and it's very clever, and something I'll probably use for a different application. It's also fairly complex to implement, and requires a lot of memory, despite being very fast (memory/speed are a common trade off).

It's faster, but should I use it in this application? No!

In my application, I've capped it at 15 orbs in play and 5 masses in play, which means it's going to take 60 operations.

That's 20 objects total, and in the Barnes-Hut algorithm, 20 * lg 20 ≈ 60 operations.

For very small data sets, sometimes, really expensive seeming algorithms are okay. Now, the Barnes-Hut algorithm would be better- it would scale better, and it would allow me to have the different orbs attract each other, and the masses- it'd be a much more compelling simulation, but that's not the point of the application. I don't need the power offered by Barnes-Hut.

All that said, I think making a Barnes-Hut simulation on the iPhone would be kinda neat. Maybe a game, or a different instrument.

I love functional programming. I'm going to present on doing FP in VB.Net in August- this is a feature new to .NET 3.5. I was very excited about the ability to do functional programming in VB.Net. And some of the key features I really wanted, currying and closures, are there.

But the limitations... they almost make it not worth the trouble. Most obviously, VB.Net doesn't support true lambdas. In a true lambda environment, I could do something like this:

f = Function(x as integer) if (x % 2 = 0) then return x / 2 else ... some other code ...

In true lambdas, you can put any code you like inside of your function. C# and F# allow this. VB.Net does not- VB.Net only allows expressions (you can't do ifs or loops or anything like that).

Still, there's a lot you can do with expressions, so that's not too bad. Since you get closures, you can work around that without too much trouble.

But then you start running into the bizarre things. I wanted to do a patterned call. In many functional languages, you can create functions like this: f(1) = 1; f(2) = 2; f(x) = f(x - 1) + f(x - 2);. Calling f(13) will print out the 13th number it the Fibonacci sequence.

Now, I accept that VB.Net wouldn't have an architecture like that built in- it's a somewhat obscure functional trick. But I was hoping I could roll my own. My first attempt at it was to come up with a compiler macro- oops! VB.Net doesn't support pre-proccesor macros. C# does, of course.

Well, okay, what about Attributes? .NET has the ability to define metadata on code, that you can "reflect" on to change runtime behavior. I could do something like this:

Module 1
  '"fib" is the function name, the second parameter is the pattern- if it returns true, execute this
  'operation, otherwise, go find another operation named "fib" to execute.
  <PatternedFunction("fib",function(x as Integer) x = 1 or x = 2)> Function f1(x as Integer)
    Return x
  End Function

  <PatternedFunction("fib",function(x as Integer) true)> Function f2(x as Integer)
    Return PatternedCall("fib")(x - 1) + PatternedCall("fib")(x - 2)
  End Function
End Module

Not as concise as I would like, but hey, it works, right? Wrong. Turns out, since Attributes are evaluated at compile time, you can only pass constant expressions into them. Since a function may possibly contain a closure (even though this one does not), you can't ever treat a function like a constant.

I'm being an FP snob, and I know it. My reason for wanting to do this is less because I have a specific need and more because I want to. I get frustrated when a language implements a potentially awesome feature in a half assed way, but as I think about it, "Potentially awesome, practically useless" describes VB.Net to a "T". If I had my druthers, we'd be a C# shop if we were doing Microsoft at all- I think a big portion of our business would be streamlined by a real RAD language, like Ruby or Python.

All of my complaints would be patched if VB.Net supported compiler macros. I'm stunned that it doesn't- it's not exactly the hardest thing on earth to implement; your average C compiler has had a macro pre-processor since before I was born. C# has one. And here's the real kick in the teeth: most macro engines are language agnostic, so there's no reason they couldn't have wired the C# engine onto VB.Net.

Basically, this is an exercise in driving home the flaws of the language that pays my bills.

Here's a demo of my current project- an iPhone app called "Gravitone".

Not content to have exceeded the assignment's requirements by adding a graphical layer, I went a few steps farther: I added support for animation and multitouch gestures. The latest version of the source lives here, and it's chock full of comments this time.

Once again, it's under a Share-Alike-CC-license. Don't hand it in as your homework.

Stanford University is publishing video and assignments for their iPhone programming course online. I've been following it and doing the assignments, and man... I miss compsci classes. I've been having so much fun doing this. For those that recall me in college, I was a lazy underachiever in most of my classes- but not the programming ones. In those, I always took the assignment and exceeded the parameters. I would show other students how to do the assignments. I'd add little flourishes.

This has been such a breath of fresh air. By day, I slog through tedious code written in tedious languages to do tedious tasks. By comparison, programming on the iPhone is downright sexy. It's fun, it's fast.

But more than that, I'm enjoying my remote college experience. My brain is getting a gentle stretching, and I really like that. Of course, it's very gentle- the course material goes at a painfully slow pace and is treading over the basics of OOP with a leaden step. But then I pick up the homework assignments, and run past the requirements and show off, and I don't care how dull the lectures get.

Not to say I get nothing from the lectures. I finally "get" Objective-C memory management. ObjC has an approach that's someplace between Java-style Garbage Collection and C style malloc/free.

In any case, I've got my first non-trivial iPhone application done. The business logic is pretty trivial- do some stuff with polygon shapes- but the UI has drawing and animations, which is well beyond the goals for the current homework assignment. If you have an Intel Mac, you can download the SDK from Apple (free signup required) and run it if you like. The linked code is distributed under a CC-share-alike license: HelloPoly code.

If anyone is dumbshit enough to try and hand this in to the class, they're going to get owned, because it's pretty obviously not what the assignment called for.

The puzzle I posted about yesterday is fixed.

The immediate cause, that has resolved the issue, turned out to be some bad data in the tables that generate the dynamic SQL. A few rows were lurking that had NULL values, and that meant when they got appended to the other values necessary to make the SQL code. When SQL server appends NULLs to other strings the result is always NULL.

Now, I knew this, and I also designed my code with the expectation that there might be some queries with nothing but dates in the WHERE clause. So I threw in a COALESCE() call that says, "if the WHERE clause is blank, replace it with '1=1'". It's just a placeholder, because there's also some logic that appends "AND period_year=2008...". To keep the "AND" syntactically correct, I just wanted to throw in some condition that would always evaluate to true.

So, the intended behavior was that, if there was no WHERE clause, it would generate the following: "WHERE 1=1 AND period_year = 2008...". There was only one problem: I appended the "AND period_year" bit before I coalesced to 1=1.

Follow with me:
@whereClause is NULL.
I append "AND period_year" to @whereClause, and the result is NULL.
I then COALESCE to convert the NULL to "1=1".
I then send a query over to the remote system with the following WHERE clause: "WHERE 1=1".

And the query never completes. That was the immediate problem, and it's fixed now, but it raises a few other questions:
1) Why did the query complete when run on its own?
2) When this query was #11 in the sequence, why did the output hang at query #6?

My best guess answer to question #2 is that SQL Server's output was lying. It just didn't flush the output buffer and so it told me I was on query #6, but really it was busy hanging on query #11. Which is interesting behavior, and only reinforces how damn difficult it is to debug TSQL.

The moral of the story, of course, is that dynamic SQL is bad. I hate it, and I hate having to do it. This whole application is one of my least favorite projects, and sadly, there's really no way to avoid getting stuck with dynamic SQL given the objectives, unless I wanted to write it in .NET. Oh, that's right, I did want to write it in .NET or SSIS, but the user demanded that each query be implemented via a stored procedure.

Since SQL Server doesn't let you specify the linked server name via a variable, there is absolutely no way to do this without resorting to dynamic SQL. It's awful. Awful. I know people find Oracle frustrating to administer, but when it comes to programming in PL/SQL, I know that I'm not going to bump into arbitrary seeming restrictions, like "functions can't modify temp tables, even if the temp table is a locally created table variable, because we just don't like letting functions modify data, even if the only way to get data back from dynamic sql in Sql Server is to have it insert its results into a temporary table".

Ugh. Lesson learned: next time a customer specifies a technical preference for implementation details, I politely tell them to go fuck themselves and do it my way.

I have a puzzle at work. My puzzle works like so.

I have an application that lives in a SQL Server 2005 database. It builds dynamic SQL based on snippets of code stored in DB tables, and then executes that dynsql against two other databases to get a single number back for each query I run.

So, the general flow is that for location X, I lookup the bits of the first query, execute it against a remote database, and get a number back. Then I move onto the next query. Once I have executed all the remote queries, I update a DB table based on the results.

I have one entire class of locations that all use the same basic query with some variations in the where clause. The account numbers differ, things like that. Nothing big between any of the locations. And all of these locations work- except one.

For one location, partway through the chain of queries it executes, it hangs. If I pick it apart and run one query at a time, each query completes in a timely fashion. No one query hangs. If I start re-adding queries back in to the process, I find that at 10 (of 15) queries, it works fine. But when I execute eleven or more queries, it hangs. These queries all execute serially, so it's not a threading issue or something like that.

Now, here's the kicker: it doesn't seem to matter which 11 queries I execute. Any combination of 11 seems to blow up. And it's not the 11th query that hangs- it hangs on query number 6 pretty consistently. I have about 10 other locations that do the same exact thing, and they all work fine. It's just this ONE location that this happens to. The hang appears to happen when accessing our Oracle Financials system only, never the SQL Server Data Warehouse we get some of our data from. It looks like it's hanging on the remote system, not my local SQL Server.

Suffice to say, I and everyone I've shared this with is utterly perplexed. If it were something logical, like the last query being the source of the hang, or other locations also being problematic, or one specific query that was always failing, I'd be able to dig in and fix it. These inconsistent and irregular bugs are the absolute worst.

Your bonus mindbending physics: the first evidence that we may really be living in a 2D space.

I swear, I will do another post about programming in Prolog. I was on vacation, and I fell off the map a bit myself with work being crazy. Now it's not crazy, but I'm still getting back in the groove from vacation.

On the topic of software design and programming, I want to elaborate on this idea I had:

We need an Emily Post of software. It's not good enough that software be well designed, in must also be polite.

What is "polite" when it comes to software? Well, what is polite when it comes to people?

Many of the same things we expect from civil people we should expect from our software. Polite people don't nag you. They don't interrupt you. They don't waste your time. They don't bore you with details, and they especially don't bore you with intimate personal details^*. They clean up after themselves. You can trust them not to blab your secrets, or let them slip through carelessness. In general, you can expect a polite person to be careful and considerate of those around them.

Using a computer, we're exposed to all sorts of rudeness, and not just from message board trolls. Alert boxes that interrupt our task. Applications that cough up complex gobbledygook only useful to the developer when an error occurs (talk about TMI) or provide nothing more than a cryptic, "An error occurred" message. Applications that make you click through six screens to do a simple task, or remember complex command or dig through complex menus to do simple tasks.

There are all sorts of rules and guidelines about how and why you should design a UI and application behavior. There's a strong (and positive) trend towards less is more thinking, even though it's much harder to do. And that's a factor of being polite to the user. No, not just polite, considerate.

It goes back to the Emily Post "Ms. Manners". The UI guidelines that people use are the rules of etiquette. It's the instruction manual that tells you to put the fork on the left and your napkin on your lap. It tells you the steps to be polite, but the steps aren't what matters. What matters is being a considerate person, or in this case, writing considerate software.

No discussion on this topic would be complete without a list of impolite vices that we see in software. ( Allow me to present: the Bastards of Software )

So- what other Bastards of Software are there? What apps do you use that you feel could be a good example of one of these kinds of Bastards?
^*He says to LJ-land, where TMI is practically part of the URL

Part of my interest in Prolog has come about because I've decided to learn a bit about AI programming. Prolog is popular for that, although the book I'm using to learn from has more of an OOP bias. That book, by the way, is Artificial Intelligence: A Modern Approach. Good book, so far. The topics are broad primers for all the major areas of AI work, which is exactly what I was looking for.

In any case, this is my first major programming "assignment" out of the book. A basic environment simulator to house a simple reflex agent. A simple reflex agent is the simplest approach to AI one can take- look at the world around me and decide on an action to take. Simple reflex, because it doesn't have any memory, or take any steps to model the world. It just looks around and does something. It doesn't have any ability to formulate goals or even find an optimal path to its desired goal.

It's an interesting bit of Prolog code, and it really shows off many of the major features of the language. Heavy use of recursion, tail-call optimizations, and even a prolog-esque approach to making it vaguely object oriented. Lists, some basic IO calls. The whole kit-n-kaboodle.

I'm kinda proud of it, largely because this lays a decent enough framework for a lot of the future exercises from my AI book.

Introduction to this Tutorial

A bit ago, I ranted about tail call recursion and talked a bit about Prolog- a programming language that relies heavily on recursion, and hence benefits from tail call recursion. Buried in the CompSci complexities, I promised a tutorial on Prolog targeted towards non-programmers.

This is that tutorial. Whether you've programmed or not before, I am going to do everything I can to make this tutorial accessible to the average person. I strongly believe that everyone can write programs, and that everyone would benefit from learning a little about it. Between the technical skills, the problem solving-skills, and the general "cool" factor¹, it's a good skill for everyone to pick up.
( This is a long-un )
In this lesson, we discussed what a program is, and mentioned a few of the different varieties of programming languages. We set up our computer to run Prolog, and now we've looked at a simple Prolog program.

---

¹Programmers are cool, right? I think so, but my opinion has a bit of bias.
²There's a hypothetical computer known as a Turing Machine. Alan Turing proved, mathematically, that his hypothetical computer could solve a certain broad class of mathematical problem. Any programming language that can solve those problems is known as "Turing Complete". Nearly all languages in common use are "Turing Complete". This means they can all do the same things, in a general sense. But they may be better at one task than another language.

I want to make coffee. To do that, I need to put six scoops of coffee into our french press. If I were writing this repeated effort as a program, I could do it in two different ways: iteratively or recursively.
( The contents of this post involve recursion, and may not be suitable for minors )I've been learning Prolog with an eye towards learning a bit about AI design, and I learn best by teaching. And that means I intend to take you with me.

I'm going to start a round of Prolog tutorials to explain the basics of programming. I'll start from the ground up, which means you don't need to know anything about programming to get into them. All you need to do is download a Prolog interpreter, and for my examples, I recommend SWI Prolog. It comes with an integrated text editor that makes it very easy to try out programs and query them.

Hey look, the work is partially done for me. Awesome. And Mozilla has an extension for this.

The real trick is to tie it to mobile devices. That's where the future is.

I've been doing some simple Google searches, but I'm looking for information on geolocating web sites. Not websites that can do geolocation, but actually attaching location information to websites, either via a <meta> tag or some sort of xml file (like site.com/geolocation.xml) that contains geographic information.

Here's the idea, as roughly inspired by this blog post. A site places a geolocation.xml file in its root directory or a subdirectory. XML data embedded in a page (probably better). This file will contain lat/longs, probably defining an area (center/radius, corners of a polygon, whatever).

A webcrawler would troll for these files, index websites by them. Client apps would reside on devices like phones, GPS devices, and the like. They'd check the device's location, and would tie that location to "nearby" websites.

There are issues. Spam management. Actually handling the indexing, overhead, optimizing. Client app design, getting buy-in on the geoloc files. Details, at this point. The sketch of the idea is there, and it's clearly a reachable goal. It may not be reachable on my own, but it's reachable.

I love Open Source. It's a great way to develop software. But it has its weaknesses. One of the main weaknesses is consistency. Take a look at these Android apps. When you look at these applications for the Google backed smartphone OS, what you'll see is a complete lack of consistency in the look and feel of applications.

I've done a little work with iPhone app development. I'm an avid user of a wide array of other iPhone apps. Apple provides a pretty standard set of UI widgets. There's some variation; the slider control looks different depending on where it's used, for example. Those variations are largely cosmetic.

If you want to have tabs in your application, there's a built-in tab control. It rests at the bottom of the screen as a set of buttons. Every app that wants to have tabbed screens will use the same widget and the same family of controller classes to do that. Someone could write their own, but it'd be a lot of work when Apple provides a good one already. It would also break the general look and feel- an app that didn't use the standard widgets wouldn't feel like an iPhone app anymore. Inconsistency is not user-friendly and a potential source of confusion.

As a general rule, every app should look more or less the same. We generally rely on the operating system to enforce that. It provides a library of controls and developers just drag and drop. The result is that, unless you go out of your way to be special, every application looks mostly the same as every other application. This gives operating systems a distinctive, recognizable appearance, and it helps minimize the learning curve on new applications.

This is a general rule. There are certain cases in which an application's functionality doesn't fit well in the established paradigm. Adobe products have always used the UI that works for their problem domain, and they've been very successful with that. I'm not trying to argue that every app should look the same, but they shouldn't be different just for the sake of being different either.

The iPhone OS goes a long way to forcing a consistent look and feel on applications, but apparently Android does not. Compare this with this. The "cab4me" app uses a tab control that looks more like a desktop app's control, while Ecorio looks more like the iPhone's approach. Then Locale does the more-like-a-desktop-app thing, but manages to look nothing like Cab4Me's approach.

If you browse through those screenshots, you'll see the same standard problem— a multi-tab interface— solved half a dozen different ways. None of them look alike. The position of the tabs tends towards the top of the screen, but the size of the click area varies. The chrome varies.

The sad reality is that no two of those apps look like they're running on the same platform. Everyone's come up with a new and novel way of solving the same problem. There's not even a consistent color scheme- something that most desktop OSes enforce, and something that carries accessibility concerns.

And that's a weakness of Open Source tools, operating systems and software. It cuts both ways; with no central authority behind the architecture, innovation proliferates at the expense of consistency. As weaknesses go, it's not too bad. It's something that could be addressed.

The big down side is that inconsistency is a Bad Thing when you're trying to promote user adoption. It hurts the Linux desktop, which already looks more consistent than this. I think that it's going to hurt the Linux smartphone.

The iPhone has a lot of weaknesses (being ruled as Apple's private little domain is not a good thing), but it certainly offers consistency. Everything looks like an "iPhone App", and is immediately recognizable as such, promoting usability, user adoption, and branding.

Yesterday, I mentioned my move over to identi.ca, but I gave the site short shrift. The service, and more to the point, the underlying engine, Laconica represents a sea change in social networks.

I'm going to skip to the end, and then go back to what's going on under the hood. Here's the end: Laconica represents a completely distributed social network. Anyone can set up a Laconica server. They can then communicate with users on any other Laconica server. Gone is the, "I can't leave, all my friends are here!" friction. Laconica lets you leave and take your friends with you.

OpenID

How does it do this? Well, there's a couple of very cool technologies running under the hood. First, there's OpenID. Back in the day, you might remember Microsoft Passport. The idea was that you could have one user name and password (owned by Microsoft) and use it to log into any site (that paid its tribute to Microsoft). Shockingly, that never caught on.

OpenID serves the same goal, but with the opposite philosophy. Anybody can set up an OpenID server and give out accounts. LiveJournal is the most obvious example for those reading here. LiveJournal gives out accounts, and those accounts exist inside its OpenID server. Okay, so, you're logged into LiveJournal, and you swing over to Identi.ca.

You click the OpenID link, and put in your LiveJournal address and Identi.ca sends you over to LiveJournal. It's asking LiveJournal, "Hey, you know this guy? You wanna vouch for him?"

LiveJournal sees that you're logged in, but it doesn't vouch for you just yet. It gives you a page that says, "Hey, this Identi.ca guy has been asking about you. What should I say?"

Assuming you hit the "Yeah, this is cool. Tell him you know me," button, you get sent back to Identi.ca with a little note from LiveJournal, telling Identi.ca that this is all cool, and you can talk.

The beauty here is that anybody who wants to can setup an OpenID server. OpenID provides limited identity. It can't prove that I'm Remy Porter, but it can prove that the LiveJournal OpenID server knows me and calls me t3knomanser. This server here will say it knows you and make up a new name for you every time. Completely anonymous.

OAuth

OAuth is another idea like OpenID. I've got an Identi.ca account, and I want to subscribe to you over on SomeOtherSite.com. OAuth is a protocol for me to give SomeOtherSite.com limited permission to access my Identi.ca data. The OAuth site describes it as a "valet key"- I can give SomeOtherSite a key to my "car" that only works for driving a mile or two, and won't open the trunk. I don't have to trust SomeOtherSite very much to give it that data.

Shared Data

So, with OpenID and OAuth, we've got a "trust network". LiveJournal vouches for me at Identi.ca. I can swing past other people's Laconica servers and share information about my identity over at Identi.ca without giving them the keys to the kingdom. The only site in the chain with any important personal information is LiveJournal. Identi.ca trusts LiveJournal when it vouches for me, and SomeOtherSite.com trusts Identi.ca when it passes along OAuth information.

Compare that with new social site Ping.fm. Ping gives you a one-stop-shop for updating everything from Twitter to LiveJournal to MySpace via email and IM. But in order to do that, you have to give Ping.fm the username and password for each of your sites. Ouch- Ping.fm now knows everything about you, and if their data is compromised- well, you're screwed. This is the exact opposite of the OpenID/OAuth architecture. You have to trust Ping.fm a hell of a lot, and you also have to trust all of your other sites to, and put the same personal information in again and again.

So, tell me why this matters

Okay, so what's the big deal. "Trust" and "OAuth" and servers and crap. I don't care. How does this affect the price of tea in China?

The big deal is this: right now, every social network is what we call a "silo". They've got their own box of data about you, and there's no way to get at that data from the outside. Or, if there is a way, it's a site specific tool: the way you get your Twitter data from outside is different from how you download all of your LJ posts.

There are a lot of disadvantages of silos. First, as I already mentioned, when you ditch a social network, you ditch all the friends you've made on that network. You've gotta get them to come with you somehow. With OpenID and OAuth (and services like Laconica), we can reduce that burden. We have protocols that let you keep track of your friends no matter where they are.

Second, LiveJournal owns your posts. If the owners evaporate tomorrow, so does all of your content. Oops. OpenID and OAuth don't, themselves, do anything to stop that. But the fact that you can use those tools to make something like Laconica- well, that means you can be your own LiveJournal, without giving up the social network you've created.

Coming attractions

Okay, this is all pretty nascent. I'm not the sort of person that says, "Hey, run your own server, it's easy!" It's not easy, and it's not worth it for most people. We generally don't worry about LJ evaporating. But the good news is that these tools are going to pressure the silos to open up: LJ's going to need to open up your account data so that you can easily archive and backup your posts to stay competitive. LJ will need to provide a better way to track your friends on other social sites than just RSS to stay competitive. More and more sites are going to grow that are focused on helping you share your information and stay connected to people.

And that's why Identi.ca and Laconica represent a sea-change. It's not the first service to work with a distributed architecture, but it's the first one to put it into the right package at the right time. With the growing Twitter discontent, they're well positioned to create a robust, decentralized network that creates the massively-multisite social network. My powers of prognostication don't let me say that Identi.ca and Laconica are going to be the killer apps that do this, but if they're not, whatever does take their place will use the framework they laid out.

Today would be Alan Turing's 96th birthday. Show your respect to the father of modern computer science and savior of England by wrapping a paper tape about your waist for the day!

Bonus points if your tape is infinitely long!

The new version of Firefox is providing an object lesson on user momentum.

Momentum is mass * velocity. Velocity, as you might recall from high school physics, is a "vector" quantity. That means it has a magnitude and a direction (if you're driving the speed limit, your speed is 55 miles per hour; your velocity is 55 miles per hour due north).

User momentum is user-base size (mass) * workflow (velocity). Firefox, as an application, has a fairly large user base. This user base is biased towards more technical minded types, people who know their way around a computer (let's face it- your average luser is going to just use IE). A technical user is the kind that tends to develop a consistent workflow.

The "AwesomeBar" haters are those kinds of users. They've developed a specific way of working (direction) and gone far down that path (magnitude). They're a subset of the overall Firefox user-base, but big enough, with enough magnitude on their workflow that they've got a lot of momentum.

Newton's First Law tells us that "an object in motion tends to stay in motion unless acted upon by an outside force"; e.g. momentum can only be changed by force. How does one apply force in this case? Well, Mozilla hit the nail on the head: they took away the old address bar behavior entirely. There's no legacy setting, no way to disable the behavior (without disabling all address bar behavior). That's force.

And Newton's Third Law tells us that "for every action there is an equal and opposite reaction". The reaction, in this case, is backlash. "This sucks, I'm going back to FF2." "Mozilla's driving users like me away." Hundreds of angry postings in hundreds of blogs rattle their "I'm dropping Firefox" sabers.

Now, imagine if you will, a train moving down the tracks. You stand on the tracks and attempt to apply force and stop the train. Obviously, you're going to lose: the change in force you have to apply must exceed the momentum of the train (mass * velocity). You can't stop a train. But you can stop a rolling bowling ball. Or a thrown football.

The important question here is: does Mozilla have enough force to counteract the user momentum? Is Mozilla stopping a train, a bowling ball, or a frisbee? In this case, I think it's a frisbee. I think the largest mass of the users (myself included) look at this AwesomeBar and think about how we can work it into our workflow. Either the direction of our workflow isn't too far from what the AwesomeBar tries to do, or the magnitude of our workflow isn't so great. We're willing to change. A subset of the total user base has a great deal of momentum, and there are only two options to deal with them: you apply force to bring them in line (remove legacy compatibility) or you cut them loose so they don't drag your project behind them.

At any rate, after work today, I'm probably going to put together "Remy's Laws of User Motion" to better explain how user behavior can be managed using physics as a metaphor.

I designed an application recently. It's a really simple application, as such things go: Read a record from a database, send the record to the mainframe, and flip a flag on the record to mark it done. Tack on a little logging and error handling, and you've got a very basic project. So basic, in fact, that my boss decides that it's a waste of my time. I'll put the design together and we'll ship it to an offshore contractor.

Now, I've seen the way offshore fucks up projects. So I sit down, and try and figure out how best to prevent this sort of blow up. My solution? I write the program myself, mostly. I supplied design diagrams, documentation, and about 40% of the code, already implemented so that it matches the design. There are a bunch of blanks in the code marked with "TODO: Open the connection to the database and fetch the next record. Map the column values according to the tech spec, a la: 'rec.sMfgCode = dbrecord("sMfgCode")'^*".

Now, it's entirely possible that the developer might screw up filling in those blanks. Mind you, this stuff is (theoretically) basic knowledge for a contractor, but hey- it could happen? But I was surprised by the scope of the offshore fuckup. The developer completely ignored the sample code I gave him. I did most of his job for him and he just pissed it down his leg and reimplemented the application. He ignored my design diagram, didn't create the classes I told him to, and generally fucked the entire thing up.

Also: no comments, bad coding practices, ignorance of key syntax elements, and a horrible horrible habit of doing lots of switching logic instead of using OOD and the config file to control app behavior.

I hate offshore.

^*No, I didn't name the database columns. They really used Hungarian Notation. I hate Hungarian Notation

I needed to grab one of my Greasemonkey scripts to install on my computer at work. So, I log on to UserScripts.org, find the list of scripts that I've uploaded, and grab the one I want.

And then I notice that one of my scripts has been installed nearly 50,000 times. Whoa! What's interesting is that it's one of the simplest scripts I've written. Just goes to show that it doesn't have to be fancy.

I completely forgot that I wrote that script- I thought the "faces only" checkbox was part of the image search page!

Remember my whole shtick about LISP? About how programs were represented the same way as data: lists of symbols.

I wanted to come up with a nice, vivid illustration of that, so I wrote a simple mad-lib application. Your mad-lib is the program. Like so:
(madlib "This is a " (noun) " of the " (adjective) " madlib system. Hopefully it is " (present-tense-verb))

A list of symbols- anything in "double-quotes"^* is just plain-old text. I wanted to avoid using strings, but the "." character broke down when I did that. Pity.

noun, adjective and present-tense-verb are actually functions that I predefined. In this example, they actually prompt you for input, but they could just as easily grab random terms from a dictionary, or do some other wizardry. The details don't matter. The core idea is that someone that doesn't know LISP could write a datafile following the format above. An end user could then load that file into my engine that provides all the functions, and execute it.

The mad-lib is executable code. This is one of the big ideas in LISP and its variants, and it's something that I discussed briefly in my previous bit on the topic. This is an example of the "little-language" pattern- creating small, simple, single-purpose languages. This example is a very little language, but still an interesting exercise.
( code follows )
^*Yes, I know those are unnecessary quotes. It's visual illustration.

I've been working on learning LISP, or more specifically, Scheme (or even more specifically mzScheme). For those unfamiliar, LISP is one of the oldest programming languages in use, as it was originally derived in the 50s. That's right, I'm learning a language that's over 50 years old. Why? Well, it's not for nostalgia's sake.
( Some Background )
( Why LISP is awesome )
( An example )
My simple (and probably clumsy, by an experienced LISPer's standards) program demonstates why LISP is beloved of AI researchers. A sentence is nothing more than a list of words, and a word is a symbol that has meaning. By establishing relationships between words and lists of words, I can build meaning and then manipulate it to derive conclusions.

There's much more, of course. LISP is also a "metalanguage", that is, a language to describe other languages. I can, using LISP, create new languages easily. One of my other learning projects is making an object oriented language all my own (derived from LISP). It's a great way to arrange and store data. Basically, I'm fascinating myself with the language. Cool, cool stuff. If anyone wants, I may build more formal tutorials as I learn.

So, something I didn't know- there's a little stinger hidden in the title text on every XKCD comic. I spent some time tonight hitting the "random" button and learning what I've missed. But I was frustrated- so many of them got truncated by my browser. So I wrote a greasemonkey script to append the title text to the end of the page.

If I wanted to be really cool, I'd link it to a rollover action or something shiny. But it's late.

I twittered this, but it certainly requires explanation. There's a problem with file systems- they treat Folders and Files as separate kinds of object. This is a mistake, and will soon be a bottleneck on innovation. We need a new alternative.

OSX has taken a step in the right direction with the concept of the "package". Applications, iWork documents, and certain iLife documents are not individual files- they're actually folders that bundle up a bunch of related objects. So, for example, if I want to distribute an application that uses sounds and images, etc. on OSX, the user will "see" one file (from the perspective of the file browser). So, Firefox- for example, appears to the user as a single file, labeled Firefox.

A more savvy user, might view the file via the command line. They'd see it listed as Firefox.app/- a folder. The file browser and the terminal both allow you to browse into that folder- revealing a structured object that represents an application. All of the supporting files are contained in that folder, but it appears to the user as a single file.

This is a "contains" relationship. A single "file"- the application has a "contains" relationship with all the components.

But this is still operating in the realm of a traditional file system.

I'm thinking of something very different. In my ideal filesystem, there are no folders. There are two kinds of objects, nodes and relationships. A "node" is more like what we conventionally think of as files and folders. It is an entity that is uniquely identified and has data associated with it. A "relationship" associates two nodes (or perhaps n-nodes) in a meaningful fashion.

So, for example, if I had a document and wanted to insert a picture into it, I could express that as a "contains" relationship- the document "contains" the image. But what if that document references some information in a spreadsheet? Why, that would be a "references" relationship. But there are different kinds of references.

For example, there's a "depends" reference- if the document "depends" on the spreadsheet, then the spreadsheet can't me removed from the filesystem without the document becoming corrupt. A special case of the "depends" relationship could be the "link" relationship- and in this case, I'm referring to a "compiler linkage"- ie. a programmatic dependency.

Another possible link could be a "metadata" link- the image somehow describes the document. Or a "parent-child" relationship, which says that one file somehow inherits (the child) properties from another file (the parent).

There's definitely performance issues with this sort of filesystem layout. There's a reason filesystems are designed the way they are- they're very efficient for read/write operations. They all descend from FSes designed in an era where throughput really mattered- and it still does.

But we're approaching a point where we have solid state storage devices. This sort of high-level filesystem could run on top of a low level FS without a significant performance hit- when you move towards solid state (compared to current rotational storage systems). There are some other advantages too- this would expand very well across a distributed filesystem. It could run on top of HTTP even (HTML sort of supports this via <link> and "rel" features).

I'm sure this wasn't the first time something like this has been suggested.

This guy claims that Java is a dying language.
( I wish! )
A very common mistake in application design is being too flexible. The more flexible you try and make your application, the more work it is to use the application. Java is full of these mistakes in spades.

---

I have another beef with Java. Mainly in that colleges are turning to it as their defacto language for Computer Science programs. This is idiocy, and it does a disservice to the students.
( Programmer snobbery ensues )
So, my ideal CS program would teach: LISP (or a derivative) and C++ for the intro and core classes. Then one of: Ruby, Perl, Python. Then, in senior year, it would get into vendor specific technologies- .NET, Java. Web application design, etc.

Has anyone worked with any .NET ORM products? NHibernate is looking pretty good and I know people really like its Java cousin, Hibernate- but I'd like to benefit from the experience of others.

The problem I'm trying to solve is this:
I have an Oracle DB. The schema is... complex. More complex than it needs to be. When we can get funding, one of our "todos" is to refactor the schema and make it something easier to access and maintain. In the meantime, there's a new project coming down the pipe that wants me to create an interface between a third-party application and that Oracle DB. Obviously, if we refactor the DB, I don't want to rewrite all of that integration code. Also, I'm pushing to do a full rewrite of the application that sits in-front of this DB anyway (it's in classic ASP). If I get the go-ahead to start that project, I'd like to be able to reuse the code from this integration project.

I want to take the "meet-in-the-middle" approach. I'd like to build the object model for the app-side without thinking about the database. I'd like a lightweight mapper to link that abstract model back to the more concrete database. The big hurdle is the fact that, when the DB schema changes, it will probably be nigh unrecognizable. Keeping the object model and the DB in sync through mapping may be too ambitious.

Mutating Decorator

I'm still playing with the Decorator pattern in JavaScript. It's a fascinating pattern in JavaScript because of the ease of modifying the interface of objects. My previous examples have focused on generating new classes based on the interface of the Decorated class. A developer can then append new method variations onto it. That's fine, but the plumbing gets complex and behaves strangely in certain situations. But that got me thinking- is there another way to approach this problem in JavaScript?( Of course there is! )

Javascript Design Patterns

I'm a huge fan of Design Patterns. For those that might not be familiar with Patterns, the summary is this: Design Patterns are reusable fragments of application designs that can be used to solve a variety of problems. Pre-designed solutions that you can plug into your own application, and thus minimize development time and reduce program complexity. If you do any programming and don't know these learn them!

Now, the GoF patterns were designed around traditional OOP languages. In their book, they used C++ and Smalltalk- the latter being one of the original OOP languages, the former one of the most well respected and recognized. Patterns became extremely popular in response to Java development- another traditional OOP languages.

JavaScript is not a traditional language. In fact, it has some pretty important things that make it different.
( Read more... )
These are a few Design Patterns adapted and developed for JavaScript. The important thing to keep in mind is not the details of individual patterns, but the goal of good software design. Patterns are a tool for designing good software, but they do not guarantee that your software well be well designed. They are easily misapplied, misunderstood and misused. Your end goal is to develop easy to maintain and usable applications. Apply patterns insofar as they help you reach that goal, but don't get enslaved to them.

I highly reccomend the MixIn pattern for JavaScript. It's a very powerful way for adding functionality to classes. Used in conjunction with the Decorator pattern, it allows features to be added without modifying the underlying classes- which is a great way to adapt complex code libraries to the specific task at hand. For many JavaScript tasks, however, that sort of flexibility might be more than is really required. JavaScript's functions-as-variables concept allows for lightweight implementations of things like the Strategy pattern, and are worth considering. I all but guarantee that you'll be using the MVC pattern in your JavaScript applications, even if you don't formally discuss it.

EDIT: Upon further testing, the AutoDecorator doesn't nest properly. That is to say, an AutoDecorator should be able to decorate another AutoDecorator. It doesn't, currently. I'll have to poke around in the code some more. This sort of problem is harder than normal to diagnose, because of the use of "eval"- which proves my point- you shouldn't use them.

EDIT AGAIN: I now have a working version of the AutoDecorator. It involves the use of closures, which are complex enough to require their own essay. I'll come back to that later. Closures are actually very important when one starts attempting to do code generation in JavaScript- I'll discuss them in a later essay to give a picture of the details.