Gadgets

I use a few gadgets on a daily basis. Here are a few of my favorites:

1. Sandisk Sansa e280 Mp3 Player

I get a lot of mileage out of this mp3 player. It has 8 gigs of internal memory, plus a miniSD slot for extra swappable memory. Most of my friends have some kind of iPod or iPod Nano, but for my purposes I think the Sandisk player suits me better. I'm not so concerned with how flashy it looks, and honestly I prefer the look of my player to any iPod - I think it looks less like an product made by a computer manufacturer, and more like an electronics gadget. From what I understand, the iPod products require iTunes to really do anything useful. For me that would be a deal-breaker on it's own - I love the ability to use Windows Explorer to just drag and drop stuff, and it's also good that it will recognize pretty much any file format.

Before this player, I used the MPIO FL100 player. This served it's purpose for a reasonably long lifetime - over 3 years, I think. Towards the end it got to the point that the LCD screen display was bleeding so badly that I could only read the 2 leftmost characters. I finally gave in and decided to get something new. I should mention that with this player, I was forced to use an extra program to transfer files back and forth - but there was more than one option, and the protocol appears to be open enough for users to write their own software, such as this one.

2. Sony Headphones

For most of my daily usage while I'm mobile, I insist on these:

I should admit that they have broken many, many times - it usually comes down to the quality of the cable inside the wires, which is hair-thin. I've tried many times to repair them, and a couple of times I've had success - but it's never really because the soldering has just come loose - it's usually because the wire close under the earpiece has completely stripped down to a hair. So I end up just cutting the wire down to that point, and re-soldering to end up with a much shorter cable length. This technique has extended the lifetime an extra 3 months or so.

This hair-thin wire design makes them very light (and explains why they are cheap), but also makes them particularly delicate, and probably way too fragile for my rough excercise and transportation habits. But so far they have always proven easy enough to find replacement for, and they usually cost around 10 pounds (18 dollars) , so it's not too bad. I try to keep the warranty stuff around, but never quite get around to trying to use it. I've been using these for about 4 years now, and have probably had about 8 or 9 sets in that time.

But it's worth it, because while they keep working, they are great. Well, at least for my taste - I don't particularly want something penetrating deep in my ear, and I don't even really care about noise blocking or cancellation. In fact, when I am at the gym or out for a run, I kind of prefer to have the ambient noise - running a path where I need to cross streets with busy traffic, it's especially useful, and would probably be reckless to use noise-cancelling headphones. Since they don't really weigh anything, I definitely prefer the over-the-ear design, even though it probably looks a bit more tacky than a simpler in-the-ear headphone.

I also want to rant briefly about one of my biggest pet peeves - the standard iPod headphones, and people who use them in public. I'm not sure who in the "world's greatest design firm" designed this masterpiece, but from a usability standpoint they are ridiculous. Perhaps it's not really something you notice when you are wearing them, and it's also probably not something you would notice if you never take public transportation. But for those of us who sit on the subway every day and invariably end up sitting between two iPod users, it is like torture. The design of these headphones almost seems to have an "open ambient" technique, which makes them louder externally than internally. Regardless of what type of audio or music the person sitting next to you is listening to, you'll be able to make out very clearly everything they are listening to - even if you have your own headphones on.

For some of the work I do in my studio at home, I prefer the MDR-V700 fully enclosed headphones. These are the same headphones used by many DJs, and give a great combination of excellent quality, encapsulation, and flexibile/portable design.

3. Motorola MPx220

After using the MPx200 phone for about a year until its total extinction, I was more than happy to upgrade to the MPx220 phone. I had a ton of complaints about the reliability of the older MPx200, and I think most of those issues were to do with the older Windows Mobile 2003 operating system, but also probably a few device issues.

This new phone has been going solid for a couple of years now, even though I managed to break the clamshell hinge a good while back. Since then I've had to be extra-careful opening and closing the thing, but suprisingly, it still works perfectly and doesn't even seem to know that its wires could snap at any moment. This is a big improvement from older clamshell phones I've used where they were rendered useless very shortly after breaking the hinge.

I've already purchased a brand new empty casing for the phone off eBay, for about 10 dollars. But it's a bit too complex for me to switch it out without putting some serious study into it, or just taking it down to the electronics center and finding someone more capable to do it for me. But so far it keeps working fine, so I haven't had the motivation to get down there and get it done.

In short, I love the phone and I love the OS - I think they've fixed the majority of stupid issues I had with the old one. I actually used to have to reboot that thing as often as I rebooted my Win2k desktop. But this new one hardly ever gets a reboot - probably only once a month or so, or whenever I have to turn it off to get on an airplane.

One thing I haven't really had an opportunity to try with the new phone is the GSM internet access. I used this quite frequently with my old MPx200, but always had problems with it. For some reason it would hardly ever work with the cable, and I could only really get it working with infrared. I would guess that this is going to work much better with this phone, like everything else does, if I ever get an opportunity to try it out.

I found a very detailed review of this phone here. They even go and open the phone up and show all of it's inner guts.

4. Motorola HS850

I really can't live without my bluetooth headset. I've seen various complaints on websites about this unit, but honestly I haven't had major problems. And considering how much the price has dropped, it's now cheap enough that if one breaks I can just grab a new one without making a big deal about it. So far, I've had one break, after about 6+ months of very frequent usage. I then went out and bought a pair of new ones, in different colors, to go with the two telephones I use (one for UK, one for Italy).

I'm biding my time until this one breaks, so I can upgrade to the next gen model, the Motorola H700 headset, which is lighter and even more streamlined:

My roommate just got one, and it's really suprisingly small. But on a minor negative note, they have once again pulled the famous old Motorola trick - they've switched the power adapter again! So it seems that with every new iteration of products, Motorola loves to switch between the proprietary "fork" plug interface, and the more standard mini-USB format. Here's the rundown I've seen so far, based on my experience (in order of purchase date):

• Motorola MPx200 : mini-USB connector
• Motorola MPx220 : proprietary Motorola connector
• Motorola HS850 : proprietary Motorola connector
• Motorola SLVR : mini-USB connector
• Motorola H700 : mini-USB connector

Most feedback I've seen on the web is more in favor of the mini-USB connection, and I am too. If for no other reason - those are much easier standard cables to find replacements for. They also don't feel as flimsy, and tend to stay connected better.

Oh, by the way, as for the H700 - they even make a flashier gold Dolce & Gabbana version!

Fast Excel Reporting

A recurring topic where I work is how to improve performance of code that exports data to Excel for reporting. Lately I've seen a lot of interesting advice on different options, so I thought I'd post some of it here.

Approach #1: C# with interop for Excel COM API
The fundamental problem with this approach is that the COM API is inherently slow, and all of the marshalling (especially with variant types!) has a lot of overhead. The net effect is that a lot of people who started out with this approach have seen performance degrade. Now, I should mention that in our core reporting engine we still primarily do things this way, and we produce reports with literally 100,000's of rows, ending up with several-hundred-megabyte size files - and yes some of them take a while, but nothing that seems that extremely bad, given what they are doing. From what I recall, the worst is something like a half hour, for almost 1M rows by 50+ columns. And obviously any user who really wants us to put that much data in one spreadsheet is not really going to be able to use it very effectively.

The first approach most people try is cell-by-cell, which is pretty slow, because in addition to marshalling you are dealing with lots of IDispatch-type calls over the interop barrier. An improvement on this, to eliminate the "COM-chattiness" is to pass a large chunk of data to Excel at once, as an array, and then to apply post formatting afterwards. The formatting can be done within a VBA macro, or it can be done via COM - in this case it's best to try to optimize formatting by grabbing ranges of similar cells and formatting them together at once.

Approach #2: Directly calling via COM vtable interfaces
This is a technique that can be used to eliminate a lot of the overhead of calling through a generic COM interop layer. The problem with the typical interop layer is that you are essentially going through the same IDispatch interface that was used back in the VB days. By encoding the vtable interface and linking it in, typically in C++ code, the maximum performance with a COM call can be achieved. This is also theoretically possible in C#, but you must be a master of P/Invoke to set it up properly, and since the Excel API is fairly extensive, to be able to use most of it you would spend a ridiculous amount of time trying to write all of the call signatures, and then debug all of the calls to get marshalling and types right. It sounds like a nightmare to me, but I've heard of people having done it.

Approach #3: Use clipboard as a protocol
Since Excel is integrated with both the text clipboard and also much more extensive clipboard formatting including DDE/OLE formats, you can pretty much pass anything to Excel via the clipboard. This would be particulary useful if you need to put random things like graphics into a spreadsheet. A common approach to sending a large set of data to Excel is to format your data as a big text csv string, copy that to the clipboard, and then paste into Excel, or even use some of the paste special to get formatting and types to work properly. This technique can provide big gains over API calls in some scenarios. One of the problems with this approach is that it is using the shared clipboard, so it's hard to run something like this in the background if there are user apps running as well. Additionally, it presents serious challenges for any kind of multi-threading application.

Approach #4: Generate Excel XML document format
More recent versions of Excel, and particularly the most recent 2007 version, have extensive support for XML. By reverse-engineering the schema from an Excel-generated XML document, you can understand how to control pretty much everything. Then the problem becomes a much simpler one of how to generate XML in a quick way - something that has numerous solutions readily available on the web. I recently came across this article, where someone has taken the time to document this approach step-by-step.

Approach #5: Use a pre-packaged solution
The last time this topic came up, someone recommended Spreadsheet Gear. This is a (not free, not open) component that can be used to generate Excel documents without even having Excel available on the server. It looks like they have basically reverse-engineered the entire document format (the legacy proprietary one, not the new open XML one). Then they've put a much more friendly and performant .NET API on top of that, and added more value around it. I haven't spent a lot of time looking at this yet, but anecdotal evidence on the site claims impressive performance. I'm looking forward to getting a license through our company (it's pretty expensive for what it does!), and then I'll be able to say something more concrete about this - but it definitely looks promising.

Approach #6: XML and Scalability
Assuming the future direction of these document formats is XML, that should provide a new range of possiblities. One idea that comes to mind is to leverage the open format to create a scalable reporting platform. Imagine you really do need to generate a ridiculously large XML-based Excel report. And imagine that the time involved in generating this report is not only document generation, but also back-end number crunching and data processing. With the normal current approaches, you would probably break these two aspects apart, so that the processing bit could be scaled and optimized - while the document generation could be isolated in a non-scalable way.

But, if you can use an XML format, perhaps you could divide the work up in such a way that you could have a scalable system, with many "workers" producing individual pieces of the report, perhaps across many virtual servers. You would end up with many individual XML fragments, and then you could have a final component that pieces everything back together and streams out the final report. This approach would have the benefit of simplifying the architecture of the workers - you wouldn't need to break up the processing bit from the document generation bit. And the scalability should only be limited to the power of your XML parser.

Big Chief Golf

One of my longest-running side projects is a website called BigChiefGolf.com. This website provides a free service to golfers, helping them to maintain their golf scores and view a trend-line of handicap improvement over time.

The most amazing thing about this service is that it has been up and running for almost 15 years in various forms, with amazingly little maintenance or revision. There have been a couple of major phases of revision, first from QuickBasic to Visual Basic, and then to ASP.Net.

I recently got it up and running on ASP.Net 2.0, which was an interesting experience. While porting the majority of libraries and even Windows apps appears to be fairly straightforward, porting the ASP.Net pages was particularly challenging. Firstly, while everything else seems to work fine in some sort of compatibility mode between 1.1 and 2.0 - i.e. I'm able to just use the old 1.1 binaries in many cases - the web stuff absolutely would not work. Then even after recompiling everything else to 2.0, the web stuff still did not work. I think the first reason for this was that they have changed the name of an important page-level attribute from "Codepage" to "CodeFile". Here's an article on MSDN that discusses this. Without this the page can't even compile with 2.0, and so an IIS error occurs right away if you try to access the page.

The second, and more substantial porting task is to change all of the code-behind files to partial classes - the new 2.0 model uses partial classes instead of inheritance to define the code behind an ASP.Net page. This involves, obviously, putting the "partial" keyword in the definition of each code-behind class. Then you need to remove and clean up certain things - in particular, the definitions of all controls should just be removed. With the partial classes approach, those controls are already declared in the portion of the class generated from the ASP code. This is much nicer for new development, as it leads to simpler and less code, but it certainly made porting a chore.

One final thing I had to spend a good amount of effort to fix was to reformat a lot of my HTML as conformant XHTML - i.e. closing my <img src="" /> tags, <br> tags, etc. When I compiled the site initially, after everything else was compiling OK, I was left with hundreds of these schema validation errors. I believe there might be some way to turn off this particular level of validation and remove these errors without fixing anything. But I figured I might as well go ahead and get it done, to ensure my code would compile on any computer with VS.Net, even with the default settings.