Creating an Excel Add-In for Market Data - Part 1 - Excel RTD Component

Since I left my previous job, I have lost my access to Bloomberg real-time data, and so now I have a lot of spreadsheets that use the BDP function for real-time lookup of stock prices, fx rates, etc. - and I don't have the Bloomberg plugin to make them work anymore. Rather than going back to the world of manually filling this data in periodically, I decided to see if I could create my own Excel add-in, and make it look and behave just like the Bloomberg one. This way, if I get access to Bloomberg again in the future, I can continue using the same spreadsheets without changing anything.

I've had plenty of experience writing Excel add-ins in the past, in particular with ManagedXLL. Unfortunately, my license for this was limited to my PC at work, and none of that code seems to work at all from my home PC - and presumably it would violate the licensing agreement anyways. It's too bad that there isn't a free or cheap "non-profit" version of ManagedXLL available.

So now we embark on trying to make it work without the use of ManagedXLL. Some of this turns out to be pretty easy, while other parts are more challenging. So I guess that's the most valuable thing about ManagedXLL - it provides a single nice and neat package to do everything you want, without having to piece together multiple languages and technologies.

For our mock-Bloomberg add-in, the first step is to look at the RTD piece. In ManagedXLL this was really easy - you can create a RTD server by just exposing a class with a few attributes. It turns out this alone is still pretty easy to do in plain C# code. We just need a few attributes, and we need to implement the IRtdServer interface from Microsoft.Office.Interop.Excel. The code for this ends up looking pretty similar to what it would have been with ManagedXLL. Finally we need to make sure our .NET dll gets registered - either using the project properties when we compile - or using RegAsm.exe to register it after we've built it.

Here's an abbreviated version of the code for the RTD server, showing the relevant parts...

[ClassInterface(ClassInterfaceType.AutoDual)]
[ComVisible(true), ProgId(PROGID)]
public class BdpRtdServer : Excel.IRtdServer
{
   public const string PROGID = "Glaze.Finance.ExcelPlugin.Rtd";
   private Excel.IRTDUpdateEvent _rtdUpdateEvent;

   public object ConnectData(int topicId, ref Array requestStrings, ref bool getNewValues)
   {
       string[] fields = requestStrings.Cast().ToArray();
       string ticker = fields[0];
       string field = fields[1];
       // ... add new topic subscription for ticker/field ...
   }
  
   public void DisconnectData(int topicID)
   {
       // ... remove topic subscription and unsubscribe ...
   }
  
   public Array RefreshData(ref int topicCount)
   {
       // ... refresh and notify any topics that have updates ...
   }
  
   public int Heartbeat()
   {
       return 1;
   }
  
   public int ServerStart(Excel.IRTDUpdateEvent rtdUpdateEvent)
   {
       // start background processing
       _rtdUpdateEvent = rtdUpdateEvent;
   }
  
   public void ServerTerminate()
   {
       // shut down background processing
   }
  
   private void NotifyUpdate()
   {
       if (_rtdUpdateEvent != null)
           _rtdUpdateEvent.UpdateNotify();
   }
  
   [ComRegisterFunctionAttribute]
   public static void RegisterFunction(Type t)
   {
       Microsoft.Win32.Registry.ClassesRoot.CreateSubKey("CLSID\\{" + t.GUID.ToString().ToUpper() + "}\\Programmable");
   }

   [ComUnregisterFunctionAttribute]
   public static void UnregisterFunction(Type t)
   {
       Microsoft.Win32.Registry.ClassesRoot.DeleteSubKey("CLSID\\{" + t.GUID.ToString().ToUpper() + "}\\Programmable");
   }
}

The main challenge with the RTD piece then becomes the actual application code, which is naturally a bit more complicated than a simple function because we need to implement all of the real-time server management going on behind the scenes. In my implementation, this involved some background threads waiting for updates and batching them up to notify back to Excel, and some smart dictionary caching to keep track of what we are currently "subscribed" to across all of our client calls.

Blogging about Blog Development

While researching some other stuff, I came across a blog framework called BlogEngine.net, and was so impressed I thought it would be worth grabbing it and giving it a try. In fact, I've been wanting to add blogging and other features to my Big Chief Golf website for some time now, so this might be the perfect technology for it. The guys that maintain this also run their own blog, and have written about new features of their blogging engine.

More about this once I've had a chance to give it a try.

Command Prompt Here

Every time I have a fresh install of Windows XP, one of the million tasks on my todo list is to add in the "Command Prompt Here" context menu. This time I did some searching around, and came across a few interesting results. First, I found this. Now, this page is very informative, and explains several different techniques for installing the shortcut. It also mentions that in Windows Vista, this will finally be a built-in feature. But the one thing it doesn't have is a simple download for an inf or reg script to actually just install it!

Then I found this page, which has the necessary stuff for a simple reg file - just copy and paste.

Still searching, I found this page on Scott Hanselman's Computer Zen blog. He really went the extra mile, and included install scripts for Visual Studio 2003 "Command Prompt Here" and also for Visual Studio 2005 "Command Prompt Here". He also has some other useful tools linked from his blog. Thanks a million, Scott. I won't be wasting time searching for these again.

RE: Not Another C# Versus VB Article

I was just digging through some old emails, and I came across this gem. This was a follow-up discussion I had with Nigel Shaw, author of Not Another C# Versus VB Article posted on CodeProject.

Date: May 26, 2005 1:05 PM
Subject : RE: Not Another C# Versus VB Article

Hi Nigel,

I just finished reading your article, "Not Another C# Versus VB
Article", and first of all let me say thank you for this work - I'm sure it will be very useful for consultants like myself, as a well-thought-out argument for building development teams on and switching them to C#.

As a developer with about 12 years of professional experience, my history breaks down into something like 1 year of BASIC, Pascal, Fortran, etc., 5 years of VB (vb3 through vb6), 2 years of Java, 1 year of VB.Net and C#, and then 3 years of C#, MC++, and C++. Needless to say, there have been periods in my career where I have considered myself a VB Guru, a Java Guru, and more recently, a .NET Guru. Thus, I feel particularly qualified to comment on your article, and also maybe offer some insights.

I absolutely agree with your central premise - that the fundamental difference between VB and C# is one of culture, rather than technology. Prior to the emergence of .NET, after I had a good amount of exposure to Java, I began to develop very similar conclusions about the differences between VB and Java - which were, after all, much more technically different than are VB.Net and C# - but even then, I believed the central premise applied - as different as they were technically, the really fundamental difference had a lot more to do with what kind of culture gravitated towards each technology. That said, at the time I always had the feeling that there was something fundamentally flawed with each of those 2 technologies, and the general culture that they seemed to promote. On the VB side, of course, you have the opposite 20/80 rule that you've commented on well in your article, in terms of developer capability. On the Java side, my experience at the time was that there was a strong culture in terms of technical competence, but that there was also a lack of focus on solving problems efficiently, often opting instead of the elegant or over-architected solution. This often lead me to believe that perhaps the VB culture actually was the bestwe could do, because it surely would not be susceptible to "too much" elegance!

Then when .NET emerged, my initial impressions were that the platform itself would provide a great step forward in terms of combining the benefits of these two cultures, while eliminating many of the negatives. After some experience dealing with cross-VB-C# teams and seeing how the two sub-teams interacted, I would absolutely say that I can confirm many of your assumptions about the culture breakdown being carried forward to these two technologies. I have been amazed at times to see the same exact piece of logic produced in two entirely different ways by VB.NET and C# teams - and with those two different ways demonstrating precisely the difference in cultures we are discussing. When I initially moved back from the Java world into the new .NET world, I was very gung-ho on VB.NET, first because I was sick of much of the flawed ideology of Java and saw much of that carried on to C#, and second because I was similarly enthusiastic to get back into what seemed to be the most MS-oriented technology - VB.NET. But after time, I began to pick up on these culture differences, and as I found myself gravitating towards the C# culture more and more, I found that the technology conventions supported by C#, and more importantly, those more "encouraged" by C#, were much more in-line with the type of development that I wanted to do.

Ok, so a few notes for your article:

1. Under "The Culture of C#", you write the following: "Marc Andreessen, CEO of Netscape and therefore the 'God of the Internet'" - I believe that you are referring to "god" in the pluralistic sense of the word, i.e. "the god of love", "the god of war", etc. - thus it would be not only incorrect English, but also potentially offensive to those that believe the word "God" should only be capitalized when referring to the one God.

2. The statement "Java applications...were more powerful and feature-rich than the vast majority of VB apps..." struck me as a bit off-base when I first read it. While you may be technically correct in a certain sense, I think this statement is a bit loaded and really depends on context and point of view of the reader. For example, if I am thinking from the viewpoint of a Windows GUI developer, this statement would seem absolutely false - even today it's not necessarily true, because even the most modern Java windows toolkits cannot match the power and look-and-feel of the equivalent toolset in VB - thus it is simply not possible to build Java Windows apps that are as "feature-rich" in that sense as VB apps. However, if I am developing server-side multi-threaded distributed messaging clusters, then yes, of course Java offers a lot more capability than VB has offered in the past, or even maybe today. But the point is, the "vast majority" of VB apps up until .NET were most certainly Windows GUI apps, while the vast majority of Java apps has always been server-side. So the statement as you have it is very "apples-and-oranges" to me. I think the point you are trying to get across is about how Java was technically superior on many levels at that time - I would just re-write this in a less ambiguous and debatable way than what you have.

3. I really like the section "Propagation of the Culture in .NET" - but I think you would do better to split up your list into 2 lists or something. About half of the points are what I would consider absolutely undebatable points about what is wrong with VB.NET (1, 2, 3?, 4, 7). However other points (5 and 6, and to a smaller extent maybe 3) are very debatable in my opinion, and seem to have a lot more to do with personal preference. I personally believe VB is the superior one on points 5 and 6 for example, and I don't agree with your conclusions that they detract absolutely from the potential of VB to promote a strong "culture". I believe the opposite could be true in fact - if these features were part of C#, they would be a further benefit to the environment, and would help provide distinguishing factors that set C# ahead of Java (like properties, events, delegates, etc.). So my point here is that at least a couple
of your arguments here seem to be very much based on personal preference, and much less like "absolute truth" that should be logically accepted by anyone reading the argument - and that detracts from the rest of your points, which are much more like absolute truths.

4. Speaking of those absolute truths, I would make some further comment to expand your point #4 - I believe this is truly fundamental to the continuation of the negative aspects of the VB culture. For example, when I first moved to VB.NET, I obviously started looking for parallels and language equivalents between VB and VB.NET. When I saw two options for doing the same thing, for example String.Length and Len(String), I noticed that the first one looked a lot like the Java way of doing things (which I was trying to escape from), while the second looked like good old familiar Len() in VB6. I continued to use this function until it came time to port a section of code from a VB.NET assembly to a C# assembly a colleague was working on - and then it hit me - this non-standard, non-object-oriented VB extensions stuff is not directly portable! If I would have wrote it with the new .NET convention String.Length instead, it would have directly ported, basically with the addition of semicolons here and there. Instead, it because a non-trivial excercise to port code between the two languages, because it was actually more like porting code bewteen two cultures!

So, once I realized this, I had to go back and re-examine a lot of what I had been doing in VB.NET and try to understand the "right way" for true .NET compatibility. This excercise really helped to clarify my understanding of the distinctions between the object-oriented aspects of VB.NET and the non-object-oriented aspects. Even more interesting, when I finally realized that all of these things simply exist in that VB assembly, I thought I should be able to simply reference that assembly from C# and give myself all the same "powers" from within the C# environment. This would have been especially useful, for example, with the famous IsNumeric() function in VB, which doesn't seem to have a .NET equivalent, but which was always very useful in VB. Unfortunately, this doesn't work from C# - you can't reference the VB dll and call all the VB functions, because it just doesn't work! So this led to an ultimate rewriting of a lot of code to remove usage of Instr(), Mid(), Trim(), Len(), IsNumeric(), and countless other non-standard-.NET functions that seem only natural to a VB
programmer.

As a final note, I would recommend a book to you, if you haven't already heard of it - I just finished reading it, after having it reccomended by a friend of mine, and I'm not sure if it is because the content is still so fresh in my mind or not, but while reading your article, I found myself going back to the ideas in the book many times and drawing parallels. The book is called "The Tipping Point", and discusses the ways that ideas turn into trends and then into epidemics, and how specific kinds of individuals are attracted to those ideas.

Thanks,
Geoff

I received a gracious reply from Nigel, and wrote a follow-up:

I'd be happy to be involved in a rewrite. Let me know what you have in mind, and perhaps we could start brainstorming a new outline, new ideas, etc.

Not sure what you think about this, or if there is already significant work out there - but I would consider expanding the scope to cover the "evolution of culture" as it has gone on through the various iterations, including a bit about culture as it applied to both Delphi and Java as well, and maybe even new contrasts between "J2EE" and ".NET" culture. My predictions for the future would be an eventual "survival of the fittest" race towards a re-unified culture in which the best of .NET and the best of J2EE co-exist happily in a single culture.

However, while this topic of general evolution of cultures is kind of a pet subject area for me, it may not be the best idea to expand your overall scope, as the current main premise you have (stated loosely, "C# is better than VB") is very pointed and controversial, and no doubt has a lot to do with what has drawn such a great response for your article throughout the community.

Perhaps along these lines I would consider a separate article as a subsequent chapter in the "series", following on from the basic premises in this one...

Hmm, I'll have to put some thought into it myself...

Geoff

Fast Excel Reporting

A recurring topic where I work is how to improve performance of code that exports data to Excel for reporting. Lately I've seen a lot of interesting advice on different options, so I thought I'd post some of it here.

Approach #1: C# with interop for Excel COM API
The fundamental problem with this approach is that the COM API is inherently slow, and all of the marshalling (especially with variant types!) has a lot of overhead. The net effect is that a lot of people who started out with this approach have seen performance degrade. Now, I should mention that in our core reporting engine we still primarily do things this way, and we produce reports with literally 100,000's of rows, ending up with several-hundred-megabyte size files - and yes some of them take a while, but nothing that seems that extremely bad, given what they are doing. From what I recall, the worst is something like a half hour, for almost 1M rows by 50+ columns. And obviously any user who really wants us to put that much data in one spreadsheet is not really going to be able to use it very effectively.

The first approach most people try is cell-by-cell, which is pretty slow, because in addition to marshalling you are dealing with lots of IDispatch-type calls over the interop barrier. An improvement on this, to eliminate the "COM-chattiness" is to pass a large chunk of data to Excel at once, as an array, and then to apply post formatting afterwards. The formatting can be done within a VBA macro, or it can be done via COM - in this case it's best to try to optimize formatting by grabbing ranges of similar cells and formatting them together at once.

Approach #2: Directly calling via COM vtable interfaces
This is a technique that can be used to eliminate a lot of the overhead of calling through a generic COM interop layer. The problem with the typical interop layer is that you are essentially going through the same IDispatch interface that was used back in the VB days. By encoding the vtable interface and linking it in, typically in C++ code, the maximum performance with a COM call can be achieved. This is also theoretically possible in C#, but you must be a master of P/Invoke to set it up properly, and since the Excel API is fairly extensive, to be able to use most of it you would spend a ridiculous amount of time trying to write all of the call signatures, and then debug all of the calls to get marshalling and types right. It sounds like a nightmare to me, but I've heard of people having done it.

Approach #3: Use clipboard as a protocol
Since Excel is integrated with both the text clipboard and also much more extensive clipboard formatting including DDE/OLE formats, you can pretty much pass anything to Excel via the clipboard. This would be particulary useful if you need to put random things like graphics into a spreadsheet. A common approach to sending a large set of data to Excel is to format your data as a big text csv string, copy that to the clipboard, and then paste into Excel, or even use some of the paste special to get formatting and types to work properly. This technique can provide big gains over API calls in some scenarios. One of the problems with this approach is that it is using the shared clipboard, so it's hard to run something like this in the background if there are user apps running as well. Additionally, it presents serious challenges for any kind of multi-threading application.

Approach #4: Generate Excel XML document format
More recent versions of Excel, and particularly the most recent 2007 version, have extensive support for XML. By reverse-engineering the schema from an Excel-generated XML document, you can understand how to control pretty much everything. Then the problem becomes a much simpler one of how to generate XML in a quick way - something that has numerous solutions readily available on the web. I recently came across this article, where someone has taken the time to document this approach step-by-step.

Approach #5: Use a pre-packaged solution
The last time this topic came up, someone recommended Spreadsheet Gear. This is a (not free, not open) component that can be used to generate Excel documents without even having Excel available on the server. It looks like they have basically reverse-engineered the entire document format (the legacy proprietary one, not the new open XML one). Then they've put a much more friendly and performant .NET API on top of that, and added more value around it. I haven't spent a lot of time looking at this yet, but anecdotal evidence on the site claims impressive performance. I'm looking forward to getting a license through our company (it's pretty expensive for what it does!), and then I'll be able to say something more concrete about this - but it definitely looks promising.

Approach #6: XML and Scalability
Assuming the future direction of these document formats is XML, that should provide a new range of possiblities. One idea that comes to mind is to leverage the open format to create a scalable reporting platform. Imagine you really do need to generate a ridiculously large XML-based Excel report. And imagine that the time involved in generating this report is not only document generation, but also back-end number crunching and data processing. With the normal current approaches, you would probably break these two aspects apart, so that the processing bit could be scaled and optimized - while the document generation could be isolated in a non-scalable way.

But, if you can use an XML format, perhaps you could divide the work up in such a way that you could have a scalable system, with many "workers" producing individual pieces of the report, perhaps across many virtual servers. You would end up with many individual XML fragments, and then you could have a final component that pieces everything back together and streams out the final report. This approach would have the benefit of simplifying the architecture of the workers - you wouldn't need to break up the processing bit from the document generation bit. And the scalability should only be limited to the power of your XML parser.

Big Chief Golf

One of my longest-running side projects is a website called BigChiefGolf.com. This website provides a free service to golfers, helping them to maintain their golf scores and view a trend-line of handicap improvement over time.

The most amazing thing about this service is that it has been up and running for almost 15 years in various forms, with amazingly little maintenance or revision. There have been a couple of major phases of revision, first from QuickBasic to Visual Basic, and then to ASP.Net.

I recently got it up and running on ASP.Net 2.0, which was an interesting experience. While porting the majority of libraries and even Windows apps appears to be fairly straightforward, porting the ASP.Net pages was particularly challenging. Firstly, while everything else seems to work fine in some sort of compatibility mode between 1.1 and 2.0 - i.e. I'm able to just use the old 1.1 binaries in many cases - the web stuff absolutely would not work. Then even after recompiling everything else to 2.0, the web stuff still did not work. I think the first reason for this was that they have changed the name of an important page-level attribute from "Codepage" to "CodeFile". Here's an article on MSDN that discusses this. Without this the page can't even compile with 2.0, and so an IIS error occurs right away if you try to access the page.

The second, and more substantial porting task is to change all of the code-behind files to partial classes - the new 2.0 model uses partial classes instead of inheritance to define the code behind an ASP.Net page. This involves, obviously, putting the "partial" keyword in the definition of each code-behind class. Then you need to remove and clean up certain things - in particular, the definitions of all controls should just be removed. With the partial classes approach, those controls are already declared in the portion of the class generated from the ASP code. This is much nicer for new development, as it leads to simpler and less code, but it certainly made porting a chore.

One final thing I had to spend a good amount of effort to fix was to reformat a lot of my HTML as conformant XHTML - i.e. closing my <img src="" /> tags, <br> tags, etc. When I compiled the site initially, after everything else was compiling OK, I was left with hundreds of these schema validation errors. I believe there might be some way to turn off this particular level of validation and remove these errors without fixing anything. But I figured I might as well go ahead and get it done, to ensure my code would compile on any computer with VS.Net, even with the default settings.

WebServices + AJAX

One thing I've never understood is how there can be so much hype about AJAX, and so much hype about Web Services, and yet no one seems to realize these two are naturally suited for each other. Web Services provides an open XML way to send data over the HTTP protocol, and AJAX provides a client-side mechanism for making HTTP calls to ask for data. What more does it take to make it obvious that these two are made for each other?

Well, from all of the amazing open-source libraries out there supporting AJAX, it seems clear that no one on the AJAX side is thinking about this approach, because they are not built with XML formatted WS compatibility in mind. It seems like AJAX is all about using raw text data formats, or just receiving and replacing HTML content directly, or anything other than a well-structured data format made for transferring structured data properly.

So I've decided to take some of the best of what I've found out there, and extend it to support the WS/XML approach in a more developer-friendly way. In a nutshell, I want to be able to put an include on my page, and then have some simple javascript functions that do display logic, by essentially accessing Web Services and handling the results as objects, rather than doing my own XML generation and parsing. Then an added bonus would be to have some binding capability, to automatically bind requests and responses to certain controls on the page, without having to actually write a lot of javascript to do the binding and processing logic myself.

There are a lot of AJAX libraries and approaches out there, so it would be impossible to really highlight them all. I'm a big fan of the Anthem framework. This framework seems to replicate the "developer feel" of traditional ASP.NET server-side controls, and so it feels more natural from a developer's perspective than, for example, the new .NET framework's AJAX controls. But even the Anthem framework seems to be focused on raw html replacement, rather than a more structured approach to passing data back and forth.

One article I really liked was this one on CodeProject, about integrating the jQuery javascript library for AJAX on an ASP.NET page. It's very simple, and allows me to focus on the WS aspects for now, without having to think about all of the added layers of ASP.NET control libraries. Those libraries add a great nice-to-have set of developer features, but I'm inclined to go back and start from the ground up, focusing first on a more solid communication infrastructure. Once we've got the passing back and forth of data working in a structured WS/XML approach, then we can try to adapt the control library facade to make development easier.

I've already got some code working with WS as an internal communications protocol for a very raw AJAX approach. Stay tuned with a follow-up article where I'll present more progress on this idea.

Against Web Services

I wrote this up last year, as we were just beginning a phase to re-architect a core piece of the system. New resources had just been brought into the team, and so lots of ideas were flying around, some very good, and some not so good.

One big idea being pushed was to change everything to run as Web Services, in an attempt to create a sort of SOA architecture. Now, in principle I certainly wouldn't be against trying out the next big architectural paradigm. But in this particular case, we were dealing with a 10-year-old legacy system, with much more fundamental issues to work out. So I was convinced that trying to throw a new architectural paradigm into the mix - especially one that was, at the time, really a very cutting edge, not yet industry-proven idea - seemed to really be deviating off the right path.


Why we shouldn't use Web Services in this phase of Re-Architecture:

I've been trying to avoid going in to a long drawn out discussion about these points, because, as you'll see when you get to the bottom of this, the answer should be pretty obvious to anyone who fully understands the problems we've faced in the past and are trying to address now. In fact, when an answer is this obvious, it is very hard to come up with a comprehensive argument for it, because most of the assumptions are just taken for granted. But I've tried to really sit down here and enumerate all of the important points to consider here, and to draw a reasoned conclusion from that.

I'm not really looking for a point-by-point debate on each of these points - for me, the real point is that there are clearly a number of debatable problems with the WebServices approach for our team, and so the conclusion is that the fact that there are so many enumerable problems here, and probably many more I haven't considered, should lead us to conclude that there's no reason to introduce so much risk into the project without some clear advantage.

I'll try to break my discussion up into two main categories: Support and Technical.

Support Reasons:

Because WebServices run in the IIS ASP.NET context, they are managed as part of IIS. This means that everyone on the development/testing team would have to become an IIS expert overnight. Given that I don't think we have a great breadth of experience with IIS, and especially with ASP.NET, this is a major learning curve for the team, and would require significant time for everyone to get trained up.

The need to ramp up on IIS with ASP.NET would extend to support people as well. We would have to convert our entire application support team into a web management team. This would mean significant restructuring of and turnover in the support team, probably having to replace or augment a good chunk of it. Currently our support people are experts primarily in AutoSys, and secondly in MQ - but from the user side only, not the system management side - because:

MQ is supported (and centrally hosted/managed) by the Middleware team. AutoSys is supported, hosted, and managed by the Scheduling Management team. The WebServices would need to be hosted on our own servers, and thus supported (at a system management level) by our own support team. There is no central managed hosting of WebServices, because with WebServices we are essentially combining both our communication and application management technologies with our actual application components, so that it is all running in one process on a single machine. The only support offered to help us with WebServices would come from the Web Support team - but clearly this is not our first-line support to guarantee reliability of the system - this would be a sort of second line support that we could call out specifically if we are diagnosing an IIS problem - similar to the kind of support we get when we are diagnosing a DB problem. But the DBAs certainly do not replace the role of our first-line support people in supporting our application - they perform tasks very specific to the DB. And even the DB is something hosted and managed centrally for us. Similarly, the Web Support team would not give us any kind of comprehensive management of IIS - they are just there to help troubleshoot when we have issues managing our own infrastructure. They are ceretainly not going to perform the level of monitoring and performance tuning on our own servers, that the middleware team would provide on the MQ servers. In this case, we are talking purely about taking ownership of all aspects of the system, and not taking advantage of any centralized management or hosting for any of it, other than managing the physical boxes. Thus, we now become responsible for (1) guaranteeing reliability of our communication infrastructure; (2) the application management infrastructure. With MQ and AutoSys, both of these are services provided to us by IT, by groups who are much more focused on the performance of those infrastructural pieces. We would essentially be giving up those benefits.

There is no out-of-the-box centralized process management solution for WebServices, in a way that would be analogous to AutoSys. Our support people clearly need a mechanism for centrally viewing and managing all services across all boxes (as do our testing and development people, really). It would also not be possible to use AutoSys in any meaningful way with WebServices, as a process management tool. So we would have to build something to meet this need for WebServices. Given the complexities introduced by being integrated with the IIS ASP.NET process (see more detail in my "Technical Reasons" comments), it would be very challenging to put together a framework/tool to manage the individual services being hosted in a WebServices envrionment.

There are too many risks and unknowns in the WebServices approach for our team right now. The focus of our current project should be making the application better, by taking the next evolutionary step in re-architecting the way we manage our systems, leveraging all of the collective experience we've gained and problems we've found with our previous approaches. The focus should certainly not be embarking out on an crusade into unknown territory to try out a new technology for the fun of it. Certainly, that could be a future phase. But do we really want to be spending so much of our time right now on learning this entirely new technology, dealing with it, supporting it, and preparing ourselves for the risk involved with all the unknowns? We now have extensive experience with MQ, and know exactly what is involved in running our current applications on MQ and porting further applications to MQ. However, we have absolutely no experience with WebServices, and we have not spent any time thinking about all of the unknowns that may come up with the development, testing, and support phases. Of course, there will be new ground that the application will cover for the first time, and there will be new support responsibilities - but shouldn't those be well-justified as providing something necessary? Here we are discussing introducing what would surely be the most risky and radically new change to our infrastructure, and there is no clear benefit to the problems we are specifically trying to address, which could not also be achieved by something much less risky and much more well known, such as MQ.

Technical Reasons:

There is no inherent guaranteed communication protocol when using Web Services - it is essentially like TCP in this regard - a method gets called directly, and if your server is not available (due to reboot, restart, crash, or even a temporary 10-millisecond network glitch), an error occurs, or the transaction may be lost in the middle if the server crashes, etc. It is, of course, possible to implement a strategy to overcome this limitation, with things like retries, re-requests, re-synchronization, custom heartbeats, etc. - however, this starts to look very similar to a lot of the inherent database problems we are trying to address now - think about all of the issues that have to be addressed in the DataLoader in order to provide a level of guaranteed processing to the database - there are similar concepts there, like retries. So, the point is, you can overcome these limitations with a lot of work, but why would you want to if you have another solution that avoids the problem altogether?

Everything runs in the IIS ASP.NET context. This introduces quite a few issues for the kind of services we run:- In order to restart one service, you typically have to restart all services on that server. For example, that means it would not be possible to restart the ServerA, without also restarting the ServerB and ServerC, assuming you run all those services on the same server. This would cause a major halt in the workflow if you are trying to restart something while you have processing underway. This is a common scenario for us in current systems (restarting a particular service for some reason), and I think we would be nieve to imagine this is going to magically go away.- A crash/corruption in one of our services, could potentially corrupt the state of all other running services. Again this would result in restarting of one service potentially necessitating a restart of all services.- To get around this problem, you would have to manage multiple instances for different services. This means an even more complex infrastructure to manage on each server.

Running in the ASP.NET context introduces a number of technical uncertainties in this company's environment. I personally have a small amount of experience with this, having built our existing web applications - and I would wager the rest of the team has absolutely no experience with this and thus no idea what may come up. To provide a few specific examples of where we have spent extensive time with our existing web apps working around these limitations:

• Accessing databases requires a specific security context, meaning you either have to tweak the ASP.NET user context, or you have to implement custom impersonation. Our web app has extensive impersonation logic to deal with this issue, and I would not wish to have to deal with any of that in a component of this application, if it can be avoided.
• Accessing MQ from a WebService would similarly have security issues, and similarly require impersonation, and potentially conflicting credentials with those needed for the database access.
• Accessing other services such as remoting, windows management instrumentation, or remote processes requires further work involving complex machine configurations and low-level windows local policy settings manipulation.
• Simple things like accessing certain files in certain locations (i.e D: drive shares, dynamic loading of dlls in different locations) can become problematic depending on permissions settings. In setting up our existing web applications on a new machine, a fair amount of time is usually dedicated just to resolving some of the dynamic plugin DLLs and dependencies, as loading them in the ASP.NET context is more problematic than loading them in a normal .exe process context.
• Differences between Win2K and Win2003 and between workstation and server (in terms of ASP.NET permissioning models - the two systems don't even use the same users!) mean that things developed on our local machines typically have numerous problems when we port them to servers. As an example, WebBatch still does not fully work on any Win2003 server because we haven't been able to address some of the policies and permissioning issues for certain pieces of functionality - and we've spent several days on that problem alone.

So I don't mean to say I anticipate having all of these exact same issues again, but just to point out the kind of uncertainties that pop up when you are dealing with the complexity of a managed permissioning context model like IIS and ASP.NET. It's not like normal .NET applications where you just xcopy to deploy and then double click to run, and there are hardly ever issues. It's back to a model like VB where a significant number of tweaks are necessary just to get components installed correctly and up and running on a new system, and the things you have to do are different between our win2k dev workstations and the win2003 servers. Who knows if these problems will creep up when we are trying to integrate with our databases, or any other random integration point we haven't considered? We can't possibly anticipate where we will have the big problematic areas when we try to deploy application components, but why would we possibly want to add this many unkowns to the project if it's not necessary?

WebServices in ASP.NET also introduce a much more complicated set of security issues. They normally run on the HTTP protocol, which is the most notoriously insecure, open, exploitable, and broadly-permissioned communications protocol. They also run on IIS, which is the most notoriously insecure, open and exploited web server. This is exactly the opposite of what we should be thinking of for an internal system with tightly coupled services, which should be communicating in a simple and secure manner. Clearly, this company has numerous levels of firewalls and security layers, but the fundamental point is still there - this is introducing an exploitable hole in the architecture where there was none before. Managing the security of each WebService on each individual server is going to become a very complex task, and will equate to a full-time job for another resource on the team. On the other hand, MQ is fully and simply secured, and the security is managed by the middleware team. I'm not that worried that we are at major risk of putting an exploitable system out there - but this would certainly be another on the long list of things we have to think about and plan for, and it's still not clear what the advantage would be.

There are significant barriers to taking a "generic" load balancing solution with WebServices and converting it to an "intelligent" load balancing solution, once we arrive at the need to do so. WebServices are very good at working with hardware balancers to balance load purely based on generic load decisions like cpu and memory utilization, etc. However, they are very bad at working with a more intelligent software load balancing scheme where you might want to have configurable control over exactly which request goes to a particular instance of a service on a particular machine. WebServices may be one standard out there for generically load balanced solutions, but they are certainly not a standard out there for intelligently balanced solutions, and I've never heard of anyone trying to implement any kind of intelligent balancing with a WebService. This is an area where MQ excels - a solution can be built initially for a generic scenario, and then adapted to an intelligent scenario later, and MQ works well in both.

There is no inherent transactional nature in WebServices, in the sense that we do transactions with MQ. We are able to use MQ transactions to synchronize a potentially complex (but quick) operation like receiving a request into the data loader via a message, begining a transaction to read that request off the queue, writing the data to the database (in a database transaction), and then completing the transaction on the queue in a way that is synchronized with the success of the database transaction - i.e. if the database transaction fails, roll back the transaction on the queue, or even roll back to the end of the queue or to a different queue to redirect to work somewhere else. This is exactly what we already do, for example with xml data loader in stress testing. Transactional mechanisms could certainly be implemented in a custom way on top of a raw WebService protocol, but they are not inherent in the technology, and so this is just one more thing we would have to worry about implementing (and implementing correctly, since it is such a fundamental piece of our component interoperation).

* We subsequently uncovered that there was some work being done on an experimental approach for Web Service enterprise transactions - but this was apparently still in a very experimental stage at the time.

WebServices do not offer us anything in terms of performance improvement over any other technology - we are using MQ successfully in other systems, and while we do have perf issues in those systems related to DB, dataloader, etc.; I certainly don't think we've ever noticed any significant performance issues resulting from MQ. We know that other teams have, but this is more due to improper usage, rather than any inherent MQ problem. Thus, the whole idea of using WebServices to address a performance problem we don't actually have should not really come into consideration - we know where the perf issues are with this application, and they are certainly not in any kind of communications layer. We shouldn't be trying to address problems that don't exist. The whole issue of SOAP performance vs. MQ performance is debatable in any case, but this should not be our guiding factor - it's like comparing using flat files vs. using adatabase based purely on performance aspects. These are two entirely different ways to address communication, and are designed to meet different needs. Assertions that WebServices should be applied to "anything you can think of" are incorrect - there is a place to use them, and there is a place not to use them.

WebServices is not the only "standard". The idea that WebServices is a "standard" or that it is built on "standards" is being thrown around as a single justification to use it. MQ is also a standard in this company, and also throughout the industry - actually it is a much more well established standard than web services, as it has been around much longer, and I'm sure that if you did a comprehensive survey in the company, you would find many more projects using MQ than using WebServices. In any case, the fact that something is a standard does not make it right to solve every problem. Again, each standard or technology has it's place, and there are also areas that are not good for some standards. The only right way to use the argument of something being a "standard" is to argue about it being the best standard to solve a *specific problem* - here we have not identified any specific problem that WebServices might be better than other standards at solving, and in fact, given all the evidence I've presented above, I think it's safe to say the opposite - WebServices, while being a nice standard, is not the right standard to solve the problems we are trying to solve.

WebServices is not the only way to implement load balancing, nor is it the best way. There are many ways to implement load balancing for MQ, Remoting, SmartSockets, or any other communcation protocol. One of the simplest possible ways is exactly what our other application are already doing with MQ - just sharing a single request queue across N instances - this results in an easy, no-hassle, and optimally efficient load balancing mechanism. Each server is guaranteed to pick up a new message when and only when they are freed up - so optimal load balancing is achieved by simply letting the individual clients decide on their own when to pull a message. This is not to say that the the model we have used for our other applications is the model that we must necessarily use for our own load balancing, but just to show that if our goal is either simplicity or optimal balancing, there are even better ways to achieve this than what would be offered with a WebServices / hardware load balancer solution - because that is already more complex. Additionally, if there are additional goals to accomplish with load balancing, that might be better solved by a hardware balancer, then as we have already discussed a hardware balancer can also be used with MQ, for example. There is nothing inherent about WebServices that should make it a special case for LoadBalancing that cannot be achieved as effectively with some other communication mechanism.

Summary: They don't offer anything special to us! So why the extra hassle? - if we have identified a number of support and technical issues that may or may not be addressable, but there is no clear advantage over any other communcation protocol, why are we even considering all of the additional risk to the project? There is most likely an answer to each point that has been brought up above - whether that is to extend the standard WebServices model with something additional for support, or to write code to implement a custom guaranteed protocol on top of the infrastructure, or to apply further tweaks to the design - but the point is, why would we want to spend one iota of our time or have any of the risk introduced by this new and unfamiliar territory, when there is no clear benefit other than using something new and cool?

As an additional point:

Other teams in this area of the company do not think this would be a good idea. I've been around to chat with the tech leads of a number of the major projects in this area, and while these projects do have diverse opinions about technologies and approaches, the consensus regarding WebServices is very consistent - it would not be appropriate for any kind of internal component-to-component communication, and should only be considered for external connection points between disparate systems. Even in the case of external connection points, it should not be considered as the only possibility, but rather as one of a number of possible solutions, depending on the nature of the connection. For example, the connection point between two of our internal sub-systems is probably more tightly coupled than would be appropriate for WebServices. But all teams agree - appropriate technologies for internal tightly-coupled connection points (i.e. component to component) inside a system could be MQ, SmartSockets, Remoting, or raw TCP - but certainly not WebServices. Here's a summary of teams I've spoken with, and the approaches they are using:

• 3 projects using SmartSockets for internal communications
• 2 projects using MQ for internal communications, and 5 projects using MQ for external communications
• 1 project using DCOM for internal communications
• 5 projects using a file-feed approach to external communications
• 3 projects connecting to external systems directly through the DB, but with intentions to move to MQ
• 2 projects using WebServices internally, and 1 using them externally

The interesting thing to note is that all tech leads were very consistent in their views on when it would be appropriate to use WebServices vs. when to use MQ, and all agreed that for the discussion of WebServices as a framework for components in our application, it really would not make any sense. This opinion was even shared among those teams that are using WebServices extensively. It is also interesting to note that those are the two projects that also have a heavy focus on a ASP.NET website GUI, and the WebServices they are building are relatively simple components that are not really meant to be distributed, but rather to provide the business logic behind their web site. Obviously, because they are already fully dependent on ASP.NET for the rest of their application, they don't see any disadvantage from a support or development perspective in respect to continuing to use ASP.NET for underlying business components - and in fact this is a benefit for them. However, for an application that is not already very tied to ASP.NET via a web user interface as the primary piece of functionality, they all agree that there would need to be a very compelling reason to move to WebServices.

Interestingly, even on the discussion of WebServices as an external communications mechanism, most teams agree that this decision still needs to be contemplated vs. the current accepted company standard for this type of interaction, which is still MQ. In fact, for teams that are currently still using Feed and DB mechanisms for connecting to other systems, rather than looking at WebServices as the best alternative for those, they are all considering MQ. One other project in particular is also fairly determined on that approach, and they are receiving a lot of guidance directly from a key individual in the CTO group. They are confident that if we go to the CTO group with this WebServices idea, we would probably not have an easy time convincing them.

References:

Finally, I would offer a few external references (just from the first few results that pop up in google) to show that these are not just ideas I'm inventing, but are actually well known issues in delineating between a Web Services approach and an MQ approach. Additionally, since some of these articles are older than others, this demonstrates that this should not be a new school of thought on the frontier of innovation - some of these concepts are well known and have been around for quite some time now:

http://expertanswercenter.techtarget.com/eac/knowledgebaseAnswer/0,295199,sid63_gci984269,00.html

When wouldn't I recommend Web services in this scenario? Web Services are currently lacking two key capabilities that can be a strong point of a message queue server. First, they have no inherent support for transactions. Most message queue systems have good transaction support and even support transactions across multiple data sources. If this type of cross-application transaction support is key to your implementation, then choose a message queue service.

Second, message queues allow guaranteed delivery. In their current incarnation, Web Services do not. Since messages can be sent asynchronously (fire and forget) with a message queue service, it is vital that the message be guaranteed to eventually arrive at its destination. Hence the idea of message "queues".

In the end, you need to carefully delineate your business needs and prioritize them. Include not only the technical details, but also keep in mind maintenance, support and Total Cost of Ownership.

http://weblogs.asp.net/ahoffman/archive/2004/03/10/87051.aspx

Remember that orthogonal component plumbing provided by either the platform or yourself, is an implementation issue - one not relevant to a specific architectural technology. It applies equally to traditional business logic within a process, as to business logic that implements an "asmx" type web service.

The choice of whether you "scale up" or "scale out" will depend on your particular circumstance and requirements. If your business logic is simple, stateless and largely based on interacting with the data tier, you could scale hrough an application farm. But if your requirements are processing intensive and complex, the most performing solution is likely to be a long living logic block that maintains state - one to which you might pply enterprise services - one that might itself be the implementation for a service.

Microsoft does not propose that business logic call other logic in the same process through XML based messaging. How efficient would that be? Rather, a service oriented architecture remedies the failure of distributed object technologies (like COM, Corba or RMI) to provide seamless program integration outside of a process (or application domain).

http://weblogs.cs.cornell.edu/AllThingsDistributed/archives/000120.html

Web services are frequently described as the new incarnation of distributed object technology. This is a serious misconception, made by people from industry and academia alike, and this misconception seriously limits a broader acceptance of the true web services architecture. Even though the architects of distributed systems and internet systems alike have been vocal about the fact that these technologies hardly have any relationship, it appears to be difficult to dispel the myth that they are tied together
....
A first thing to realize however is that the current state of web services technology is very limited compared to distributed object systems. The latter is a well-established technology with very broad support, strong reliability guarantees, and many, many support tools and technologies. For example, web services toolkit vendors have only just started to look at the reliability and transactional guarantees that distributed object systems have supported for years.

http://www.hanselman.com/blog/ClassicWebServicesVersusPOXXMLOverMQAreYouReallyUsingXML.aspx

For the transport layer, it ultimately will come down to what you are most comfortable with in your enterprise. The advantage of sticking to SOAP over HTTP is that the tool support (both development and management) will be much stronger. While web services are in theory supposed to be transport-neutral, HTTP is certainly the "first among equals" choice. MQ is a good choice when you need to support guaranteed messaging or if your own testing has demonstrated that you have higher scalability and/or throughput for the volumes you are seeing.

http://www.financetech.com/featured/showArticle.jhtml?articleID=14702692

That's because issues like security, non-repudiation (both parties are sufficiently authenticated so the transaction cannot be disavowed), redundancy (a message will continually be sent until its receipt is verified), transport (http vs. MQ Series, etc), transaction semantics (do both functions or neither), and workflow (gather information X from this application and information Y from the next) have yet to be adequately solved by Web-services architects, says Carey.