Wednesday, October 19, 2011

Consuming XML from REST Services

I spent some time recently developing a process that would bump addresses up against the Google Maps Geocoding service, and the Bing Maps Location service, retrieve and parse the returned XML, and compare the resulting addresses. It had been a while since I last used the XmlDocument, and I re-encountered things I thought best to write down, lest I forget again.

If you would like to see the results yourself, the following URLs were used:
Google: http://maps.google.com/maps/api/geocode/xml?address=[address]&sensor=false
Bing: http://dev.virtualearth.net/REST/v1/Locations?query=[address]&output=xml&key=[key] (you’ll need your own key)

Using Web Requests and Web Responses

The first trick to all this is making web requests from your code. I chose to use the HttpWebRequest and HttpWebResponse classes in the System.Net namespace. I created a helper class which would take a URL, generate a GET request to that URL, then process the response stream into a string.

public static string GetWebResponseBody(string RequestUrl)
{
StringBuilder stringBuilder = new StringBuilder();

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(RequestUrl);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
UTF8Encoding encoding = new UTF8Encoding(false);
byte[] byteBuffer = new byte[8192];
int count = 0;

do
{
count = stream.Read(byteBuffer, 0, byteBuffer.Length);

if (count != 0)
stringBuilder.Append(encoding.GetString(byteBuffer, 0, count));
}
while (count > 0);
}
}

return stringBuilder.ToString();
}

This would give back the XML to be parsed, using the LoadXml() method on the XmlDocument, like so:

string xml = WebHelper.GetWebResponseBody(RequestUrl);
XmlDocument xmlDocument = new XmlDocument();
xmlDocument.LoadXml(xml);


The first attempt to the Google service worked great. The XML was returned and parsed successfully. When I attempted the Bing service, however, I received this helpful message: Data at the root level is invalid. Line 1, position 1


Long story short, the Bing service emits a UTF-8 identifier at the very first position, which the XmlDocument does not particularly like. The trick, then, is not to grab the web response stream as a string, but feed it directly into the XmlDocument, using the Load() method:

public static XmlDocument GetWebResponseXml(string RequestUrl)
{
XmlDocument xmlDoc = xmlDoc = new XmlDocument();

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(RequestUrl);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
xmlDoc.Load(stream);
}
}

return xmlDoc;
}

Now, the following works for both Google and Bing, regardless of that UTF-8 identifier:

XmlDocument xmlDocument = WebHelper.GetWebResponseXml(RequestUrl);



 


Looking for XML Nodes


Once you have the XmlDocument, you can use the SelectNodes() or SelectSingleNode() methods to find elements, using XPATH statements. However, you must be aware of whether the root element (document element) has a default namespace.


For example, the XML returned by the Google service is simple and straight-forward:

<?xml version="1.0" encoding="UTF-8"?>
<GeocodeResponse>
<status>OK</status>
<result>...</result>
</GeocodeResponse>


 


There is no default namespace, so the following code works to find the status of the call to the Google service:

status = xmlDocument.DocumentElement.SelectSingleNode("status").InnerText;


 


Now let’s look at the XML from the Bing service (I’ve stripped it down to the essentials):

<?xml version="1.0" encoding="utf-8"?>
<Response xmlns="http://schemas.microsoft.com/search/local/ws/rest/v1">
<StatusDescription>OK</StatusDescription>
<ResourceSets>...</ResourceSets>
</Response>


 


If we try that same bit of code to find the status of the call to the Bing service, you’ll get a Null Reference Exception:

status = xmlDocument.DocumentElement.SelectSingleNode("StatusDescription").InnerText;

Why? Because we have not taken into account the default namespace. We need to reference the namespace in any XPATH statements we use, in this case. Use the XmlNamespaceManager to declare the namespace information for the XmlDocument, then reference it in the SelectNodes() or SelectSingleNode() methods:

XmlNamespaceManager nsMgr = new XmlNamespaceManager(xmlDocument.NameTable);
nsMgr.AddNamespace("api", "http://schemas.microsoft.com/search/local/ws/rest/v1");

status = xmlDocument.DocumentElement.SelectSingleNode("api:StatusDescription", nsMgr).InnerText;


 


You can see that I declared the “api” alias for the namespace, which we find in the Bing XML. Just remember when doing more complex XPATH, that alias prefixes any node you would then try to find.

Wednesday, July 20, 2011

Rate Clock for Android

We are proud to announce our first Android application, Rate Clock! Rate Clock is a free, ad-free tool that allows you to set a billable rate, and like a stopwatch, track your time and billable amount. It also has a rate calculator, to calculate the billable time for a specified date and time range. Simple yet useful, you'll be able to track exactly the time you have spent performing a task or in a conversation!

Rate Clock is available in the Android Marketplace, at https://market.android.com/details?id=com.collabroscape.rateclock&feature=search_result.

Wednesday, May 11, 2011

Automating ClickOnce Installations for .NET 4.0 Applications

One sign of a mature development group is the ability and willingness to automate application builds and deployments. While a variety of tools help us build our assemblies (MSBuild, NAnt, etc.), and other tools help us manage those builds (Team Foundation Server, CruiseControl.NET, etc.), a lot of time can be spent implementing the deployment side of that automation. This article is designed to assist those endeavoring to create ClickOnce deployments for their applications, focusing on those require the Microsoft .NET 4.0 Framework.

About ClickOnce

We use the Mage (Manifest Generation and Editing Tool) executable to create our ClickOnce deployment. This program is shipped with the Windows SDK, and can be found under [Program Files]\Microsoft SDKs\Windows\v[X.X]\Bin, where [Program Files] may vary based on your PC or server OS, and [X.X] is the SDK version – Windows 7 shipped with 7.0A, but at the time of this writing, 7.1 is available.

An important point is that if you are deploying applications for .NET 2.0, 3.0, or 3.5, you should use the executable found in the SDK Bin folder. However, if you are deploying a .NET 4.0 application, you need to use Mage.exe located in the NETFX 4.0 Tools sub-folder.

A ClickOnce deployment has 3 critical parts:

  1. The application files, which include all necessary executables, libraries, resource files, and configuration files.
  2. The application manifest, which is an XML file describing the application files. This file will typically be named YourProgram.exe.manifest.
  3. The deployment manifest, which is an XML file describing the application, manifest, version, and update instructions. This file will typically be named YourProgram.application

You will find that if you generate a ClickOnce deployment from the Visual Studio IDE, it creates an HTML landing page, a Setup executable, and all of your application files are appended with the .deploy extension. The landing page and Setup.exe are created as a matter of convenience, but they are unnecessary when creating your ClickOnce deployment. The .deploy extensions lead to a discussion about your specific environment.

Environment Considerations

The users and locations of your application will dictate how your environment is set up, and how your deployment behaves. Remember, a ClickOnce deployment contains your application executable, DLLs, .config files, and many other items which will be available to your users from a website or file share. Typically, these items are prevented by IIS from being downloaded.

If you are deploying to an intranet site, you will probably have more control over the IIS site that serves up the deployment, specifically the MIME types and file mappings. If you are deploying to an Internet site, or if the deployment will be hosted by the same website as another ASP.NET application or web service, you will not want the user to be able to view or download .config files! This scenario is why it can be helpful to rename your deployed assets with the .deploy extension.

ClickOnce Deployment Overview

An important consideration is how your files will be physically situated. If you will be upgrading the application periodically, and you want to take advantage of ClickOnce auto-updates, placement of the manifest files is very important. For clarity and organization, it is helpful to place your application files in a versioned sub-folder of the installation site.

For this example, we will assume the following:

  • The user will access the ClickOnce installer on a public-facing website, www.yoursite.com/installs.
  • The physical directory of the website is C:\Website\Installs\.
  • The build directory is C:\Builds\YourProgram\.
  • The ClickOnce application will auto-upgrade, if a new version is detected.

At first, we are going to create a new ClickOnce deployment for YourProgram version 1.0.0, and publish that installation. The Mage.exe statements can be run at the command line, but you can also use that to create your MSBuild or NAnt script to call the executable.

Preparing the ClickOnce Deployment

After your build process has compiled the necessary components and assemblies, they will need to be copied or moved to a working location. Remember to include the configuration file, resource files, and, if you want, an application icon file. For this example, these files are copied to C:\Builds\YourProgram\1.0.0\.

Your assemblies and executable need to be strong-named, or signed with a strong-name key file, at compile time. You can modify the project file to sign the assembly with the following lines (in the top <PropertyGroup/> section):

    <SignAssembly>true</SignAssembly>
<AssemblyOriginatorKeyFile>C:\Artifacts\YourCompany.snk</AssemblyOriginatorKeyFile>



In the executable project file, for WPF or WinForms applications, you may also need to add the following lines to the same <PropertyGroup/> section, which can also be created via the project properties:

    <GenerateManifests>true</GenerateManifests>
<UseApplicationTrust>false</UseApplicationTrust>
<SignManifests>true</SignManifests>
<ManifestCertificateThumbprint>[thumbprint]</ManifestCertificateThumbprint>
<ManifestKeyFile>C:\Artifacts\YourCompany.pfx</ManifestKeyFile>



The manifest key file will also be used to sign your ClickOnce deployment. For development, you can create a temporary self-signed certificate, which will expire in a year from the time of generation.


Create a New Application Manifest


To create the application manifest file YourProgram.exe.manifest, execute the following:



mage.exe -New Application
         -ToFile C:\Builds\YourProgram\1.0.0\YourProgram.exe.manifest
         -Name "YourProgram by YourCompany v1.0.0"
         -Version 1.0.0.0
         -FromDirectory C:\Builds\YourProgram\1.0.0
         -IconFile appicon.ico
         -UseManifestForTrust true
         -TrustLevel FullTrust
         -Publisher "YourCompany, Inc."
         -Processor x86|x64|msil


If the application manifest was built without error, it will be found at C:\Builds\YourProgram\1.0.0\YourProgram.exe.manifest.


Sign the Application Manifest


The generated application manifest must be signed with a certificate, which you will have created yourself, or received from a certificate authority:



mage.exe -Sign C:\Builds\YourProgram\1.0.0\YourProgram.exe.manifest
         -CertFile C:\Artifacts\YourCompany.pfx
         -Password password


Append .deploy Extension to Application Files


Once the application manifest is generated, you can append the .deploy extension to all application files, excluding the manifest file. This will allow the files to be downloaded from an IIS website without modifying MIME types or file mappings.


Create a New Deployment Manifest


The application manifest will now be used to generate the deployment manifest YourProgram.application:



mage.exe -New Deployment
         -ToFile C:\Builds\YourProgram\YourProgram.application
         -Name "YourProgram by YourCompany v1.0.0"
         -Version 1.0.0.0
         -AppManifest C:\Builds\YourProgram\1.0.0\YourProgram.exe.manifest
         -ProviderUrl
http://www.yoursite.com/installs/YourProgram.application
         -Publisher "YourCompany, Inc."
         -AppCodeBase 1.0.0\Coll.UI.WPF.MediNexus.exe.manifest
         -Install true
         -Processor x86|x64|msil


Notice that the deployment manifest file has been created in the directory above the application manifest file’s directory. While the file can be generated to any physical location, when the set of files (application files, application manifest, and deployment manifest) is deployed to the install website, the deployment manifest will reside in the parent directory of the deployed version. This makes upgrades more straightforward (and obvious).


Modifying the Deployment Manifest File


Unfortunately, before we can sign the deployment manifest file, we must content with several short-comings with the Mage executable. First, unlike the MageUI application that also accompanies the Windows SDK, we must modify the deployment manifest to indicate that the application files are aliased with the .deploy extension. Secondly, if we want to force the ClickOnce installation to be upgraded automatically when the application is launched, we must also make that modification within the deployment manifest.


If you edit the YourProgram.application file with a text editor, you will see the <deployment/> element looks like this:

  <deployment install="true" minimumRequiredVersion="1.0.0.0">
<subscription>
<update>
<expiration maximumAge="0" unit="days" />
</update>
</subscription>
<deploymentProvider codebase="http://www.yoursite.com/installs/YourProgram.application" />
</deployment>



So that ClickOnce is alerted to the .deploy aliased files, we will add the mapFileExtensions=”true” attribute to the <deployment/> element. Then, to force ClickOnce to check for and install newer versions prior to launching the application, we replace the <expiration/> element with the <beforeApplicationStartup/> element. The <deployment/> element should look now like this:

  <deployment install="true" mapFileExtensions="true" minimumRequiredVersion="1.0.0.0">
<subscription>
<update>
<beforeApplicationStartup />
</update>
</subscription>
<deploymentProvider codebase="http://www.yoursite.com/installs/YourProgram.application" />
</deployment>



Signing the Deployment Manifest


Just like the application manifest, the deployment manifest must be signed:



mage.exe -Sign C:\Builds\YourProgram\YourProgram.application
         -CertFile C:\Artifacts\YourCompany.pfx
         -Password password


Publishing the ClickOnce Installation


Because of how the application files have been assembled, and the manifest files generated, publishing the installation is trivial. Copy the versioned folder C:\Builds\YourProgram\1.0.0 and the deployment manifest file C:\Builds\YourProgram\YourProgram.application to the website physical directory, C:\Website\Installs. In the real world, you will be copying or FTPing to your web servers. Once you direct your users to www.yoursite.com/installs/YourProgram.application, they will be able to install and run the application!


You will likely need to upgrade your deployed application in the future. Luckily, installed copies will check the deployment manifest to see if the version has been changed, and ClickOnce handles the rest.

Tuesday, May 10, 2011

When OO Meets Uh-Oh

This will be an ongoing series on pitfalls and gotchas when developing object-oriented components with the .NET framework.

Working With Nested Objects

When creating business entity classes that represent our enterprise data, we typically encounter a need for nested objects. A good example of this is in the categorization of information, such as Product Category:

  • Tools
    • Hand Tools
      • Hammers
      • Screwdrivers
    • Power Tools
      • Drills
      • Sprayers

At first, we may create a simple class to store such data:

    public class NestedObject
{
public NestedObject()
{
this.Id = 0;
this.Name = string.Empty;
this.ModifiedDate = DateTime.MinValue;
this.ParentObject = new NestedObject();
}

public int Id { get; set; }

public string Name { get; set; }

public DateTime ModifiedDate { get; set; }

public NestedObject ParentObject { get; set; }

}



This class stores the information we need, it compiles without error, and we go on about our business, loading the data from a database or XML feed.

    NestedObject tools = new NestedObject()
{
Id = 1,
Name = "Tools"
};

NestedObject powerTools = new NestedObject()
{
Id = 2,
Name = "Power Tools",
ParentObject = tools
};



However, the problem is exposed when we execute this code, courtesy of the StackOverflowException. Notice in our constructor that we are initializing our Parent Object property. We have inadvertently caused infinite recursion, because as each Nested Object is created, it attempts to create another as its parent, which then also attempts to create its parent, ad infinitum.


Simple Solution


One solution would be to instantiate the Parent Object property as needed, or when the property getter is accessed:

    public class NestedObject
{
private NestedObject _parentObject;

public NestedObject()
{
this.Id = 0;
this.Name = string.Empty;
this.ModifiedDate = DateTime.MinValue;
}

public int Id { get; set; }

public string Name { get; set; }

public DateTime ModifiedDate { get; set; }

public NestedObject ParentObject
{
get
{
if (this._parentObject == null)
this._parentObject = new NestedObject();
return this._parentObject;
}
set { this._parentObject = value; }
}

}



Notice how we defer the object instantiation from the constructor to the property getter. Now, our code still compiles, and when executed, works as expected – unless this object will be XML serialized by a Web or WCF service.


Nested Objects and XML Serialization


When an object, or list of objects, is returned by a method in a Web or WCF service using SOAP, each object is XML serialized for transmission (and then deserialized by the client). Using reflection, the value of each property in the object will be retrieved during serialization, and set when the object is recreated by the client. If a property is another class, composed of members like our Nested Object class, the sub-properties of that class will be discovered, and so forth. 


Let’s create a web service that returns a list of Nested Objects:

        [WebMethod]
public List<NestedObject> GetNestedObjects()
{
List<NestedObject> list = new List<NestedObject>();
NestedObject tools = new NestedObject()
{
Id = 1,
Name = "Tools"
};
list.Add(tools);

NestedObject powerTools = new NestedObject()
{
Id = 2,
Name = "Power Tools",
ParentObject = tools
};
list.Add(powerTools);

return list;
}



When this web method is invoked, we get the same StackOverflowException. The code itself executes without issue, but upon XML serialization, as the properties are discovered, we encounter the infinite recursion issue.


What is the Best Solution?


To answer the question of how best to handle this situation: It depends. Your solution should take this recursion situation into consideration, but the final implementation depends on how you manage your data, code, validation, and everything else. If the data will be displayed in a tree view or similar control, your business entity classes can take the top-down approach (where the children are in a property of the parent). In most cases, however, there is no reason for the entire hierarchy to be constructed and populated in your objects at run-time (but again, this depends on your exact needs).

Wednesday, February 23, 2011

Simplifying Complex WHERE Clauses with COALESCE

We have all received these requirements from the user. Someone on the sales team approaches you with this:

“I would like to search customer data by ID, email address, state, country, and by name. Oh, and I’d like the name search to be exact or partial, but I want to be able to choose.”

Immediately, we begin debating the best (and easiest) way to accomplish this request. We could generate a complex SQL statement within our application, so that the WHERE clause includes only the fields we are interested in as criteria. We could create several stored procedures – Get Customer By ID, Get Customers, Search Customers, etc. We could also create just one stored procedure or parameterized query that will accommodate our requirements.

The COALESCE Statement

The procedure we will create will utilize the COALESCE statement in the WHERE clause. This SQL Server function is similar to the ISNULL function, in that you can replace a NULL value with another value. With COALESCE, however, you can provide numerous alternatives:

COALESCE(@ValueA, @ValueB, @ValueC)



Each value is evaluated until the first non-NULL value is found. When used in a WHERE clause, the COALESCE statement allows you to use both local variables and columns. This enables us to create our simplified query.


Using the COALESCE Statement


Here is an example of a single, simple stored procedure that can retrieve a customer’s details based on several optional criteria. (Note – This query uses the AdventureWorks database)


create procedure [dbo].[GetContact] 
@ContactID int
,@FirstName nvarchar(255)
,@LastName nvarchar(255)
,@EmailAddress nvarchar(255)
as
select * from Person.Contact
where ContactID = COALESCE(@ContactID, ContactID)
and FirstName = COALESCE(@FirstName, FirstName)
and LastName = COALESCE(@LastName, LastName)
and EmailAddress = COALESCE(@EmailAddress, EmailAddress)



As you can see, we now have a single procedure that can be used in several ways – retrieve a customer by their primary key, or retrieve a list of customers by first and last name, or email address. This can also be used as a parameterized query in .NET code.


Our next goal is to design the query to support partial name searches. We modify the input parameters to allow the user to specify whether they are using a partial name search. We also take advantage of SQL Server 2005 and 2008 enhancements regarding LIKE operations and string concatenation.


create procedure [dbo].[GetContact] 
@ContactID int
,@FirstName nvarchar(255)
,@LastName nvarchar(255)
,@PartialNameSearch bit
,@EmailAddress nvarchar(255)
as
declare @Wildcard nvarchar(1)

if (@PartialNameSearch = 1)
set @Wildcard = '%'
else
set @Wildcard = ''

select * from Person.Contact
where ContactID = COALESCE(@ContactID, ContactID)
and FirstName like COALESCE(@FirstName, FirstName) + @Wildcard
and LastName like COALESCE(@LastName, LastName) + @Wildcard
and EmailAddress = COALESCE(@EmailAddress, EmailAddress)

What we have now is a dynamic query that supports multiple criteria, and partial or exact name searches. Because no wildcard is specified on an exact name search, the LIKE operator functions as an equals operation. If the caller indicates the partial name search is desired, the first and/or last name will be partially-searched.


Considerations


1) Obviously, using a dynamic query such as this affects your indexing strategy. SQL Server execution plans take into consideration all criteria in the WHERE clause, so index accordingly.


2) To help avoid deadlocks, the criteria in the WHERE clause in all queries should be similarly ordered.


3) This procedure or query requires non-utilized criteria values to be NULL. The calling procedures or applications must be configured to pass NULL (or DbNull from .NET applications) into these non-utilized parameters.