Wednesday, June 26, 2013

Windows Azure Autoscaling (Preview)

I was working on a production deployment for a client, and found something that (almost) made me forget what I was doing. If you have logged into the Windows Azure portal lately (like, today), you might notice the green PREVIEW badge next to the Scale menu option:

Scale_1

Yes, autoscaling is now an option configurable from the Azure management portal. You no longer need external tools like the Enterprise Library autoscaling application block, which I had been meaning to look into…

Obviously, you can leave autoscaling off, or configure it by CPU or queue, and configure options for each.

Scale by CPU

Scale_2

To scale by CPU, you primarily focus on the target CPU load (indicated in this image by 60-80%). At that range, the autoscaling process would scale the instance up by the specified number until load % is reduced, and continue to do so if needed after the specified wait time.

A real-world scenario where this is useful is a web role handling peak traffic.

Scale by Queue

Scale_3

To scale by queue, the autoscaling process monitors the number of messages in an indicated queue, and can scale up and down based on that number.

The real-world scenario here is a worker role that is responsible for handling messages in a queue. Think ecommerce order fulfillment or a content indexer.

Bottom Line

The bottom line? This is awesome, and Microsoft continues to give developers and companies reasons to build on Windows Azure.

Monday, March 25, 2013

Using DotNetOpenAuth with Constant Contact

I was writing the OAuth portion of my client’s cloud application which will communicate with Constant Contact, allowing them to manage contacts and send trigger emails. Having used DotNetOpenAuth before, it figured to be a great fit here. Now that it is all working, it is a great fit, but I do want to describe some issues I had to overcome.

Before we start, let me just say that the Constant Contact documentation is really well done. The other nice thing is we do not (currently) have to worry about refresh token management, as Constant Contact does not expire their issued access tokens.

Implementation Details

The implementation is fairly straight-forward. I have a static consumer class that handles the access token retrieval, and then in my ASP.NET MVC4 website, a controller which calls the consumer.

Consumer

public static class ConstantContactConsumer
{
    /// <summary>
    /// Constant Contact service description
    /// </summary>
    public static readonly AuthorizationServerDescription ServiceDescription = new AuthorizationServerDescription
    {
        TokenEndpoint = new Uri("
https://oauth2.constantcontact.com/oauth2/oauth/token"),
        AuthorizationEndpoint = new Uri("
https://oauth2.constantcontact.com/oauth2/oauth/siteowner/authorize"),
    };

    /// <summary>
    /// Gets the client using the static service description, client id, and client secret
    /// </summary>
    /// <returns></returns>
    private static WebServerClient GetClient()
    {
        WebServerClient client = new WebServerClient(
            ServiceDescription,
            ConfigurationManager.AppSettings["ConstantContactConsumerKey"],
            ConfigurationManager.AppSettings["ConstantContactConsumerSecret"]
            );
        client.AuthorizationTracker = new TokenManager();
        return client;
    }

    /// <summary>
    /// Begins the authorization process by calling the Authorization endpoint
    /// </summary>
    /// <param name="CallbackUrl"></param>
    public static void BeginAuthorization(string CallbackUrl)
    {
        var client = GetClient();
        client.RequestUserAuthorization(null, new Uri(CallbackUrl));
    }

    /// <summary>
    /// Finishes authorization by passing the authorization code to the Token endpoint
    /// </summary>
    /// <returns></returns>
    public static string FinishAuthorization()
    {
        string accessToken = null;
        var client = GetClient();
        client.ClientCredentialApplicator = ClientCredentialApplicator.PostParameter(ConfigurationManager.AppSettings[CLIENT_SECRET]);

        IAuthorizationState authorization = client.ProcessUserAuthorization();

        if (authorization != null)
            accessToken = authorization.AccessToken;

        return accessToken;
    }

}

MVC Controller

public ActionResult GenerateToken()
{
    string callback = Url.Action("ConstantContact", "OAuth", null, Request.Url.Scheme);
    ConstantContactConsumer.BeginAuthorization(callback);

    // Should not get here
    return RedirectToAction("Index");
}

public ActionResult ConstantContact()
{
    string accessToken = ConstantContactConsumer.FinishAuthorization();

    //
    // Store token
    //

    return RedirectToAction("Index");
}

When you browse to http://localhost:12345/OAuth/GenerateToken, the OAuthController will call the BeginAuthorization() method of the consumer. The user is redirected to the Constant Contact website to log in and grant access, then is redirected to http://localhost:12345/OAuth/ConstantContact. Once there, the FinishAuthorization() method of the consumer is called, and the access token retrieved.

HTTP or HTTPS?

According to Constant Contact’s documentation, the redirect URI should be HTTPS. However, the URI needs to be valid, and the HTTPS requirement is not enforced.

You can also use a localhost URI, which is awesome for testing! In this example, I use http://localhost:12345/OAuth/ConstantContact as the Redirect URI.

The “username” parameter

You can see in Constant Contact’s OAuth2 documentation that when you call the authorize URL, you will be redirected to the callback URL with 2 query string parameters: “code”, which is exchanged for an access token when calling the token URL, and “username”, which is used when calling the Constant Contact API. The resulting URL is http://localhost:12345/OAuth/ConstantContact?code=abc123abc123&username=joesflowers.

What is not obvious or intuitive is how DotNetOpenAuth exchanges the code for an access token. When the FinishAuthorization() method calls the client.ProcessUserAuthorization() method, DNOA pulls the authorization code from the URL, and treats the remaining URL (http://localhost:12345/OAuth/ConstantContact?username=joesflowers) as the Redirect URI. Here’s the problem: Your redirect URI is (or should be) http://localhost:12345/OAuth/ConstantContact.

This is sent by DNOA to Constant Contact:

https://oauth2.constantcontact.com/oauth2/oauth/token?client_id=CLIENTID&client_secret=SECRET:
    code: abc123abc123
    redirect_uri: http://localhost:12345/OAuth/ConstantContact?username=joesflowers
    grant_type: authorization_code
    client_id: CLIENTID
    client_secret: ********

The response from Constant Contact is:

{
  "error": "redirect_uri_mismatch",
  "error_description": "Redirect URI mismatch."
}

Solution

The bottom line here is, while we need to know the “username” when we call the API later, we need to remove that parameter from the Redirect URI DNOA provides to Constant Contact. This is done in the consumer, by modifying the FinishAuthorization() method. All we do is create a modified HTTP Request, which will be passed to the DNOA client’s ProcessUserAuthorization() method:

/// <summary>
/// Finishes authorization by passing the authorization code to the Token endpoint
/// </summary>
/// <returns></returns>
public static string FinishAuthorization()
{
    string accessToken = null;
    var client = GetClient();
    client.ClientCredentialApplicator = ClientCredentialApplicator.PostParameter(ConfigurationManager.AppSettings[CLIENT_SECRET]);

    HttpRequest request = HttpContext.Current.Request;

    // Reproduce request URI
    UriBuilder absoluteUri = new UriBuilder(request.Url.GetLeftPart(UriPartial.Path));
    foreach (string key in request.QueryString.Keys)
    {
        if (key != "username")
            absoluteUri.AppendQueryArgument(key, request.QueryString[key]);
    }

    // Reproduce request headers
    WebHeaderCollection headers = new WebHeaderCollection();
    foreach (string header in request.Headers)
    {
        headers.Add(header, request.Headers[header]);
    }

    // Create request info
    var requestInfo = HttpRequestInfo.Create(request.HttpMethod, absoluteUri.Uri, headers, request.InputStream);

    IAuthorizationState authorization = client.ProcessUserAuthorization(requestInfo);

    if (authorization != null)
        accessToken = authorization.AccessToken;

    return accessToken;
}

I’ll look for a better solution within DNOA, but this does the trick for now.

Tuesday, February 19, 2013

Azure Lessons Learned (Part III)

So you’ve been rocking along, adding new features to your Azure table service-based project, including new fields in your table entities, and everything works as it should – locally, using the emulator. Then you deploy to Azure, and you get this ominous gem:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="
http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  
<code>OutOfRangeInput</code>
   <message xml:lang="en-US">One of the request inputs is out of range.
RequestId:0c01939e-1197-4269-aafc-0e873c9b1b20
Time:2013-02-19T14:25:06.6864009Z</message>
</error>

You have to make sure that any new properties you’ve added to your table entities conform to table storage standards, available here: http://msdn.microsoft.com/en-us/library/dd179338.aspx

 

DateTime Values

Almost every time I have received the request input out of range exception has been when a DateTime property is added to an entity, but not defaulted correctly. Normally, if you declare a DateTime, but do not assign a value, it defaults to 1/1/0001, or DateTime.MinValue. However, Azure table storage does not support DateTime with values less than 1/1/1753. So, be sure to default your DateTime column:

this.MyNewDateTimeProperty = new DateTime(1753, 1, 1);

Thursday, February 7, 2013

Azure Lessons Learned (Part II)

I have a goal of posting things I have learned while developing an Azure solution (www.scuddle.com). The latest issue was this exception:

The context is already tracking a different entity with the same resource Uri

The resolution is pretty straight-forward – detach from the entity! I’ll explain…

I am using a custom Azure table storage provider, as many do, that provides functionality to Insert, Delete, Update, and Query tables for stored entities. Throughout the application (in this case, ASP.NET MVC 4), a new instance is created, entities are retrieved and updated, and then the instance is released.

However, if you perform an Update, you will have attached to a table entity, like this:

public void Update(TElement entity)
{
    this._tableServiceContext.AttachTo(this._tableName, entity, "*");
    this._tableServiceContext.UpdateObject(entity);
    this._tableServiceContext.SaveChanges(SaveChangesOptions.ReplaceOnUpdate);
}

After the update, if you attempt to query the table for that same entity, you might get the “context is already tracking” exception, because you have attached to the entity in question, but only if you are using the same instance of your storage provider.

The solution is to detach. In the code below, I am detaching both before the update, which does not throw an exception, and also after the update. By doing so, queries immediately following the update are successful.

public void Update(TElement entity)
{
    this._tableServiceContext.Detach(entity);
    this._tableServiceContext.AttachTo(this._tableName, entity, "*");
    this._tableServiceContext.UpdateObject(entity);
    this._tableServiceContext.SaveChanges(SaveChangesOptions.ReplaceOnUpdate);
    this._tableServiceContext.Detach(entity);
}

Tuesday, January 15, 2013

Azure Lessons Learned

During the course of my development on Scruddle (http://www.scruddle.com), a digital media aggregation and publishing platform, I have encountered several gotchas and practices which would be wise to share. Scruddle collects quite a bit of data, in varying sizes and formats, from short status messages from Facebook and Twitter, to large news articles and emails. This information is stored using Azure table storage, and so several rules were devised or discovered for content storage.

As you read this, it may be useful to review the MSDN article “Understanding the Table Service Data Model”, at http://msdn.microsoft.com/en-us/library/windowsazure/dd179338.aspx.

Encoding

When aggregating information from the Internet, the content type will not be known. Early on, I encountered issues saving content directly to table storage, in the form of obscure HTTP exceptions from the Windows Azure Storage DLL (in one case, it turned out to be HTML character codes in Meta tags). For Scruddle, the best option was to URL encode text content prior to saving, using the HttpUtility.UrlEncodeUnicode() method in the System.Web assembly. This held true both for the storage emulator and actual cloud storage.

Before URL encoding, the raw data can be indexed for searching, trending, or other activities. Prior to displaying the content, you simply pass the URL encoded content through the HttpUtility.UrlDecode() method, also in the System.Web assembly.

Content Length (Strings)

A single table storage column with the String data type has a limit of 64KB, but there are several points to consider. First, this does not translate to 64K characters, and second, you may need to be prepared to handle string content that exceeds 64KB.

For Scruddle, I modified the business logic that Inserts table entities to examine the string content length. If it exceeded a certain threshold (in Scruddle’s case, 32K length or higher), the table entity’s content would be stored in Blob storage, instead of table storage. Simply set a boolean flag indicating where the content is stored, create and store a blob byte array, and null out the table entity’s content prior to storing.

When retrieving the table entity, the boolean flag indicates whether the content is stored with that entity, or should be retrieved from Blob storage, and the entity is then returned to the consuming processes.

Content Length (Byte Arrays)

Table storage also has a 64KB limit for byte arrays. If binary content might exceed 64KB, the same approach to Scruddle’s string storage can also be utilized. Use a boolean flag to indicate if the content is stored in the table entity or Blob storage, and insert and retrieve the content appropriately.

Decimal Data

It is very important to realize that, for table storage, decimal data must be stored in the double data type, not the decimal data type. That holds true for any kind of decimal data, from currency to geospatial coordinates.

Other Thoughts

I have been extremely pleased with Azure table storage. In fact, I have been more impressed by the cost and flexibility of table storage than I have SQL Azure. While reporting on table storage entities presents a different challenge than traditional database reporting, it is worth considering keeping large datasets in cloud storage.

I’ll probably create another “lessons learned” blog as Scruddle transitions to production. Until then, hopefully this information can keep your head from banging against the wall.