Daniel Root's Blog

Monday, May 6, 2013

You Probably Need Cloudflare

imageFor kicks recently I turned on Cloudflare for my blog.  This service sits between users and your servers and adds several value-add services.  The free sku includes some security, analytics, caching, and CDN.  Paid skus add SSL support and more advanced security checks – including DDoS prevention.  None of the skus charge for bandwidth. Now obviously my blog isn’t the hottest site on the web (though, I am sometimes surprised by the traffic), but so far it’s meant a ~%40 reduction in bandwidth to the blog.  In this case, all that does is save Google Blogger money, but you can imagine for a cloud-hosted site that would translate pretty evenly into a 40% reduction in the bandwidth bill.  They advertise “up to 65% reduction” in bandwidth.  By using a CDN, which it automatically sets up for static content, the site is also served up faster worldwide. 

Setup is fairly straight-forward, and just involves pointing your DNS to them as part of a wizard setup.  They analyze your existing DNS, then once they’ve set up their internal DNS, they tell you the name servers to point to.  This takes minutes.  The bottom line is that if you have any cloud-hosted content, then for 5 minutes and 0$, you stand to save 40-65% of your bandwidth bill.   If SSL or some of the other features drive you to the paid SKUs, then it’s still fairly simple math: if 40% of your bandwidth bill > $20, then use Cloudflare Pro, Business or Enterprise.

Friday, February 15, 2013

Visualizing The Census With #D3js

imageNot long ago D3js came on my radar as a library I needed to look into.  At first I dismissed it as ‘just another jQuery wannabe’.  There is some overlap, but having played with it a little, I’m coming to think of D3 more as jQuery’s nerdy cousin.

I still have some learning to do, but my first foray into it is a quick census data visualization.  Truth be told, I janked one of the examples and tweaked it to allow choosing any census measure, adjust the colors, automatically determine the scale, and fade between color changes.

var width = 960,
    height = 500,
    centered,
    quantize;
 
var path = d3.geo.path();
 
var svg = d3.select("body").append("svg")
    .attr("width", width)
    .attr("height", height);
 
var color = d3.scale.linear()
 .domain([0, 20])
 .range(["#75B7CF", "#00114F"])
 .interpolate(d3.interpolateLab);
 
queue()
    .defer(d3.json, "/content/us.js")
    .defer(d3.csv, "/content/censusquickfacts.csv")
    .await(ready);
 
var us, census;
function showData() {
    var rateById = {};
    var maxVal = 0; var minVal = 0;
    var vals = [];
    census.forEach(function (d) {
        var fips = parseInt(d.fips);
        if (fips != 0 && fips.toString().substr(3) != '000') {
            var key = document.getElementById('measure').value;
            var val = parseFloat(d[key]);
            if (val > maxVal) maxVal = val;
            if (val < minVal) minVal = val;
            rateById[fips] = +val;
            vals.push(val);
        }
    });
 
    quantize = d3.scale.quantile()
     .domain(vals)
    .range(d3.range(20).map(function (i) { return color(i); }));
 
    svg.select('g').selectAll('path')
         .transition()
        .duration(750)
         .style("fill", function (d) { return quantize(rateById[d.id]); });
 
}
 
function prepareMap() {
 
    svg.append("g")
          .attr("class", "counties")
        .selectAll("path")
          .data(topojson.object(us, us.objects.counties).geometries)
        .enter().append("path")
          .style("fill", function (d) { return '#ccc' })
          .attr("d", path);
 
 
    svg.append("path")
         .datum(topojson.mesh(us, us.objects.states, function (a, b) { return a.id !== b.id; }))
         .attr("class", "states")
         .attr("d", path);
}
 
function ready(error, usData, censusData) {
    census = censusData;
    us = usData;
    prepareMap();
    showData();
}

The result is pretty astounding.  With just a few lines of code, we grab some CSV census data and county map vectors, do the math to map the census value to a color, and draw it on the page. 

The ajax and selection are nice, but technically are things jQuery already does pretty well. Similarly, D3’s databinding model is powerful, but not enough to convince me to pick it over Knockout.  What D3 brings that is fairly unique are a slew of useful math, array, and scaling functions.  For example, this would be much more complex without D3:

//Create a function of 20 colors fading from 75B7CF to 00114F
var color = d3.scale.linear()
        .domain([0, 20])
        .range(["#75B7CF", "#00114F"])
        .interpolate(d3.interpolateLab);

As would this:

//Take the values from the CSV and map them to one of the 20 colors.
quantize = d3.scale.quantize()
        .domain(vals)
        .range(d3.range(20).map(color));

Perusing the examples, there’s still a fair amount to learn about this library, but I think it’s safe to say I’ll be putting it in the toolbox alongside jQuery and Knockout.

Tuesday, February 5, 2013

SharePoint Collaboration, Visualized

I recently developed some custom web analytics reporting on top of SharePoint 2010 for a customer.  Among the data shown are number of page requests for each department, broken further into department.

It turns out as a metric, this isn’t necessarily the best.  For example, departments with sites that require lots of clicking have more ‘hits’ than departments with more efficient sites.  Still, it is _a_ metric. 

image

One interesting pattern I’ve seen repeatedly looks like the example above.  The sparkline shows number of requests per day.  For some departments, it routinely comes out looking like this.  Note one employee spiking one day, followed by another employee the next day, and another the next.   I could be over-analyzing, but I believe what this shows is that Employee1 does something, which Employee2 then sees and does something else with, that Employee 3 sees, and so on. 

In other words, by charting the page hits over time by employee, we’ve visualized collaboration within a department.  With a little more smarts, this could be turned into more valuable data- suggesting colleagues, or helping provide insight into business process improvement possibilities.

Thursday, January 31, 2013

Serve up a HTML5 Cache Manifest using ASP.NET MVC

I’ve been working on some mobile HTML5/jQuery Mobile applications that allow for running offline. The magic that allows for offline web apps is HTML5’s cache manifest, which tells the browser which resources, HTML, images, scripts, etc. to store on the device, and which it is allowed to access over the devices network connection.  The format for this file is fairly straightforward:

CACHE MANIFEST
#comments here
NETWORK:
#Urls that the app is allowed to access
*
CACHE:
#Urls that the app should store in cache. 
/index.html
/someimage.png
/somescript.js

However, two gotchas can make working with it difficult.  First, it must be served up with a particular content type, “text/cache-manifest”, that is not always enabled on web servers.  That involves some server-side tweaks, which can be tricky if you don’t have access to the server.  Second, it should change anytime your app changes, so that the browser knows to re-load the resources.  However, often if you change code or markup inside a file, you may forget to update your cache manifest.  Or at least I did enough to come up with this alternative.

Instead of a static manifest, I serve one up via a MVC action.  The action includes the assembly version and a file timestamp so that it is always versioned automatically with any change to the app.

public ActionResult Manifest()
{
var manifest = "CACHE MANIFEST" + Environment.NewLine +
      "# App Markup Date: " + System.IO.File.GetLastWriteTime(Server.MapPath("~/Views/Mobile/Index.cshtml")) + Environment.NewLine +
      "# Server Assembly Version: " + this.GetType().Assembly.GetName().Version + Environment.NewLine +
      "NETWORK:" + Environment.NewLine +
      "*" + Environment.NewLine +
      "CACHE:" + Environment.NewLine +
      Url.Action("Index", "Mobile") + Environment.NewLine +
      Url.Content("~/content/jquery.mobile-1.2.0.min.css") + Environment.NewLine +
      Url.Content("~/scripts/jquery-1.8.3.min.js") + Environment.NewLine +
      Url.Content("~/scripts/jquery.mobile-1.2.0.min.js") + Environment.NewLine +
      Url.Content("~/scripts/knockout-2.2.0.js") + Environment.NewLine +
      Url.Content("~/content/images/icons-18-white.png") + Environment.NewLine +
      Url.Content("~/content/images/ajax-loader.gif") + Environment.NewLine;
 
return Content(manifest, "text/cache-manifest");
}

This assumes a single page application, but could be easily adapted to add additional dependencies.  Using on an application simply involves adding the following to your view:

<html manifest="@Url.Action("Manifest", "Mobile")">

Thursday, January 17, 2013

LEAP Dev Kit Initial Impressions

I was thrilled to get my LEAP dev kit today, and even more excited that the simple code I had whipped up “just worked”.  If you’ve not followed the news on this, it is a motion capture device that tracks hands and fingers, similar to a Kinect.  Unlike the Kinect, this is designed for laptop and PC use, and is about the size of a pack of gum.

The device I received, Rev 6, is sleek brushed aluminum with a black top and bottom.  A USB cable connects it to your PC or Mac.  It’s a very nice 1st gen release, and is the same form factor that will be available at Best Buy soon.  That said, you do get the idea that this could be integrated into laptops etc. instead of or in addition to trackpads, etc. 

From a developer perspective, the provided SDK is extremely easy to work with.  I chose to focus on the JavaScript SDK for now, and have found ramp up super easy.  LEAP has abstracted away all the complex tracking and math, leaving the developer with a rich, but not overwhelming API.  Interestingly, it works by exposing a WebSocket that you can consume from your page to stream in tracking data.  Fortunately, LEAP hides all the WebSocket goo and makes this easy to implement. To work with it in JavaScript, one just has to include the leap client js, then register to receive “frames”:

 Leap.loop(function (frame) {
                    latestFrame = frame
                    self.hands(frame.hands);
                    self.fingers(frame.fingers);
                    self.tools(frame.tools);
                });

6 lines of code for 3D motion capture is pretty sweet.  Some developers might find this lacking.  Videos show off point clouds and implied other lower-level abstractions are possible.  If you’re expecting to use one of these to capture 3D objects, for example, you’ll be disappointed at least in the 0.7.1 version of the SDK. 

Fortunately, I’m more interested in the higher-level abstractions  for now anyway.  I like the idea of being able to just wave my hands and _do stuff_.   Who that has seen Star Wars is not interested in that?  To test the waters, I decided to publish the sample JavaScript, but with a very simple main menu that allows me to choose a demo by holding up a number of fingers":

function chooseLinkByFingerCount() {
    if (lastFingerCount == 1) window.document.location.href = '@Url.Content("~/Scripts/leapjs-master/examples/dumper.html")';
    if (lastFingerCount == 2) window.document.location.href = '@Url.Content("~/Scripts/leapjs-master/examples/camera.html")';
    if (lastFingerCount == 3) window.document.location.href = '@Url.Content("~/Scripts/leapjs-master/examples/visualizer.html")';
    if (lastFingerCount == 4) window.document.location.href = '@Url.Action("Ko")';
}

var lastFingerCount = 0
var selectionTimeout = -1;
Leap.loop(function (frame) {
    latestFrame = frame

    if (latestFrame.fingers.length != lastFingerCount) {
        lastFingerCount = frame.fingers.length;
        document.getElementById('uiFingerCount').innerHTML = lastFingerCount.toString();
        if (selectionTimeout > -1) window.clearTimeout(selectionTimeout);
        selectionTimeout = window.setTimeout(chooseLinkByFingerCount, 1000);
    }
});

If you hold up 2 fingers for a second, then it jumps to the second demo.  In practice, this may not be the best UX (it only goes up to 10 menu items and took longer than mousing!), but it couldn’t have been much easier to implement, and I could feel the Force growing stronger within me as I navigated by simply holding up fingers.

Since then, I’ve also developed a quick demo to display tracking data using KnockoutJS so that I can start to get my head around gesture detection with this.

All in all, I’m really impressed with the kit and look forward to developing more with it.  I have some fun ideas for this thing, so keep an eye out here and on my test site: http://leapthing.azurewebsites.net/

Wednesday, January 16, 2013

SignalR Whiteboard Redux




My previous post noted that in certain networks, SignalR failed to receive messages unless I used SSL. I had assumed that all that was necessary was an open port 80, but after chasing down David Fowler in jabbr, I learned this is not the case. Some network proxies and appliances can indeed break the techniques SignalR uses, and SSL is indeed one of the ways around this.

So, for DrawThing, this means I added a [RequireHttps] attribute to my MVC actions so that requests get redirected to SSL. Since Azure websites handles SSL for none-custom domain names, this 'just works' for now.
Since the last post, I've added a few features: color, size, downloads, and improved path smoothing. It's still a rough little experiment, but has been a fun project.

Tuesday, January 8, 2013

A SignalR Whiteboard

imageAfter spending a little quality time with the JavaScript-based real-time communication library SignalR, I had the idea of putting it to use in a little whiteboard application.  The idea is that users could join a board in their web browser, and then share a collaborative space to doodle.  Of course, it would have to support phones and tablets as well as full PCs.

It’s still very much a proof-of-concept, but I’m happy to say a weekend of tinkering came up with this: https://drawthing.azurewebsites.net/Draw/View/wholeworld

Just join up with your favorite HTML5-supporting browser and start drawing!  Anybody else connected will also see what you draw. To create new boards, you can go here:

https://drawthing.azurewebsites.net

It doesn’t currently have any fancy features like saving boards or color.  But it does illustrate a slightly different use for SignalR beyond chat, and let me learn some iOS web app idiosyncrasies I wasn’t aware of before.  

How I Built It

Part of the beauty of SignalR is the minimal amount of wireup necessary to get real-time communications going.  After a creating an empty MVC project and using NuGet to install Microsoft.Aspnet.SignalR –pre and jQuery, it just takes a few lines on the client:

<script src="~/scripts/json2.min.js" type="text/javascript"></script>
<script src="~/scripts/jquery-1.8.3.min.js" type="text/javascript"></script>
<script src="~/scripts/jquery.signalR-1.0.0-rc1.min.js" type="text/javascript"></script>
<script src="~/signalr/hubs" type="text/javascript"></script>

this.drawHub = $.connection.drawHub;
$.connection.hub.start();

And one line on the server, in Global.asax:

RouteTable.Routes.MapHubs();

To get the actual whiteboard functionality going, the only server-side code necessary was a little to allow for joining groups and sending commands between clients.  SignalR handles all the communications goo and provides the ‘magic’ for shuffling calls between server and client using WebSockets, a variety of long-polling techniques, or carrier pigeon.  Below is the hub for my little whiteboard.

    public class DrawHub : Hub
    {
        public void JoinBoard(string userName, string boardName)
        {
            boardName = "BOARD" + boardName.ToUpper();
            Groups.Add(Context.ConnectionId, boardName);
            Clients.OthersInGroup(boardName).onUserJoined(userName, boardName);
        }

        public void SendPath(BoardPath path)
        {
            var boardName = "BOARD" + path.BoardName.ToUpper();
            Clients.OthersInGroup(boardName).drawPath(path.Points);
        
        }
        public void SendClear(string boardName)
        {
            boardName = "BOARD" + boardName.ToUpper();
            Clients.OthersInGroup( boardName).clear();

        }
    }

If you’ve never worked with SignalR, this code may look a little odd.  First, Hub abstracts away all the usual chat-like goo, like joining and leaving clients.  Clients can remotely call into the methods, and calls like ‘Clients.OthersInGroup(boardName).onUserJoined’ actually send calls _up_ to the other connected clients.  It’s worth noting ‘onUserJoined()’, ‘drawPath()’ and ‘clear'()’ aren’t real server-side methods at all.  SignalR is using some dynamics trickery to capture those calls and shuffle them up to the correct clients, to be called in JavaScript.

The JavaScript, then, looks like this:

 this.drawHub.client.onUserJoined = function (userName, boardName) {
      $('#uiLog').html('Welcome, ' + userName);
 }

Let that soak in a bit: the server is calling up to the clients and executing JavaScript like it was nothing.  Wicked. And we can call back down to the server just as easily:

 self.drawHub.server.sendClear('@ViewBag.BoardName');

To make this happen, SignalR is establishing either a WebSockets or long-polling connection and then using that transport to do client/server communication.  Fortunately, this is almost completely abstracted away, and works with just about any browser.  I even tested on the Kindle ‘Experimental Browser’, and while I wouldn’t recommend it, it did function.

With the basic client/server communication happening, all that was necessary was to take in mouse and touch events, draw on the local HTML5 canvas, and then send those to the other clients.  The key wireup for those looks like this:

self.drawCanvasElement = document.getElementById("drawCanvas");
self.drawCanvasContext = self.drawCanvasElement.getContext('2d');

$('#drawCanvas')
          .mousedown(self.canvasMouseDown)
          .mouseup(self.canvasMouseUp)
          .mousemove(self.canvasMouseMove)
          .on('touchstart', self.canvasMouseDown)
          .on('touchmove', self.canvasMouseMove)
          .on('touchend', self.canvasMouseUp);

This sets up the graphics and mouse/touch events.  When the user draws a path, it draws the path on drawCanvasContext (not shown here- view source if you want to see the drawing code), then sends it to the hub:

 self.drawHub.server.sendPath({ BoardName: '@ViewBag.BoardName', Points: path })

The hub, in turn, calls other clients to draw the path as well:

 this.drawHub.client.drawPath = function (path) {
                $('#uiLog').html('received ' + path.length + ' points.');
                self.drawCanvasContext.beginPath();
                self.drawCanvasContext.moveTo(path[0].X, path[0].Y);
                for (var i = 0; i < path.length; i++) {
                    var point = path[i];
                    self.drawCanvasContext.lineTo(point.X, point.Y);
                    self.drawCanvasContext.stroke();
                }
            }

iOS Long Polling Quirk

This all worked fine, and to my three-year-old son’s amusement, I was soon able to doodle on the iPhone and have it show on 2 other nearby screens. But once I started testing in iOS, I got some strange behavior.  Some paths would show immediately, while others took a while, or seemed to never come at all.  Debugging was tricky- was the problem at the sender, hub, or receiver? I was pretty stumped until I ran across some forum posts suggesting Apple changed the behavior of long polling in iOS6.  It’s a bit unclear what exactly they did, but essentially it reduces the number of connections the browser may have open at a time, meaning that chatty apps like this one can break. As best I can tell, they reduced it to one background thread, that randomly handles any ajax calls.  My app was sending a path every time the user lifted their finger, and iOS would decide to send this path seemingly at random. To work around this, I decided to queue up my calls:

this.canvasMouseUp = function (e) {
                self.isMouseDown = false;
                self.sendQueue.unshift(self.pathToSend);
                e.preventDefault();
            }

this.processSendQueue = function () {
                if (self.sending) return;
                if (self.sendQueue.length == 0) return;
                self.sending = true;
                var path = self.sendQueue.pop();
                self.drawHub.server.sendPath({ BoardName: '@ViewBag.BoardName', Points: path })
                   .done(function () {
                       $('#uiLog').html('sent ' + self.pathToSend.length + ' points');
                       self.sending = false;
                   })
                   .fail(function (er) {
                       $('#uiLog').html('error sending' + er);
                       self.sending = false;
                   });
            }

I use window.setInterval to call processSendQueue every few milliseconds. This ensures that only one path at a time is being sent, and may (or may not- are race conditions possible in JavaScript?) be a good idea to do anyway.

Once in place, iOS fell into line and started working the way you’d expect.

Trouble Brewing?

I do have one issue that I’m currently trying to solve.  On one network I’ve tested on, the browser can send, but not receive paths.  This is odd to me, since it would seem like it would be agnostic to any sort of network conditions: as long as port 80 is open it should work.  I’ve posted more about that on StackOverflow.

What’s Next?

It’s hard to know what I’ll do next with this sort of project. Oftentimes, nothing- this was just a little ‘spike’ to keep me up on the latest cool toys from Redmond.  But, I’d like to at least wire up snapshots of some sort, so that late-comers to a board can see the current image, and perhaps to allow saving images, intelligently scaling across browser sizes, etc.  Color and multitouch also seem like interesting endeavors, as do Evernote/Dropbox integrations.  I’m also slightly embarrassed by the pop-up username prompt, and would rather a more buttery, if still minimal experience.  Or, I reserve the right to leave it as a fun little experiment and excellent massively multiplayer tic-tac-toe platform.