mca blog

those who read my blog know that i favor a hypermedia style when implementing dist-net apps. i don't always use that style (it's not always the proper style for the problem at hand), but when i have the chance to choose which style i implement, hypermedia is often my selection.

so that leads to an obvious question: "why?"

while there are many reasons (too many to cover here), there is one aspect of the hypermedia style that i enjoy very much: it's "action" orientation. not "event-driven" or "behavior-based" but "action" orientation.

problem domains

when working with clients, there is a point (usually early on) where the talk focuses on the "problem domain"; the stuff that must get done. often this information is "bottled up" within the heads of selected members in the client company; sometimes it's well documented in policies & and procedures documents. it can also appear as "source code" from some previous attempt at mapping the problem domain to computer code. this last method of expressing a problem domain is, IME, the least reliable as there have likely been several compromises (to fit the constraints of the coding environment) and often inaccurate translations (due to lack of understanding between the SME [subject matter expert] and the development team/person).

anyway, through some means, the domain is exposed sufficiently to allow the work of mapping that domain onto the selected technology stack. in my case, this is almost always some form of mapping the domain to services running over HTTP. that's what i do most often and, at this point, i rarely get "called" unless the client has "pre-selected" for an HTTP implementation anyway[grin].

so, the next step is how to go about mapping the domain to HTTP. this is not the same as writing code; this is before code. this is the design or architecture part of the experience. and, since it is a design process, there are lots of possibilities, lots of tools, lots of methodologies, etc. and few can be accurately labeled as clearly "right" or "wrong." at this point, this is mostly about the preferences, and proclivities of the architect/designer.

in my recent experience, there are two very common ways to approach this "mapping" task. one that focuses on the entities identified in the problem domain. another that focuses on the actions identified in the problem domain. there are other ways to view the domain (i.e. actors, components, etc.) but i'll focus on these two here: entites and actions.

action orientation

hypermedia designs focus on the actions taking place in the problem domain. things like "get a list of customers sorted by date" and "allow users to filter the order list by sales region" and "allow selected users to add new customers", etc. are all actions. when implementing a system using these actions as the centerpiece, hypermedia designs select the appropriate affordances (within the selected media type) and map those affordances to the identified actions.

for example:

"get a list of customers by date"

<a href="..." rel="customers-by-date">Customers By Date</a>

"allow users to filter the orders list by sales region"

<form action="..." method="get" class="filter-orders">
<input name="region" value=""/>
</form>

"allow selected users to add new customers"

<form action="..." method="post" class="add-customer">...</form>

in the above examples, the HTML.A and HTML.FORM affordances are used to provide the actions that map to the problem domain. note that hypermedia style implementations express the problem domain within messages themselves. this is an important point since it makes it possible to share this message in various ways and still retain the problem domain details (i.e. even if the message was sent via FTP, the affordances could still work and still retain their usefulness).

the use of in-message affordances to carry the semantic details of the problem domain (instead of mapping the domain directly to the network protocol) is sometimes cited as a drawback of the hypermedia style.

you'll notice that the actual URIs for each affordance are of little interest here. in the hypermedia style, URIs are treated as nothing more than transient identifiers that enable the actions. hypermedia style implementations are prepared for (and fully expect) the moment when the URI for an affordance changes (for any number of reasons). that is one of the reasons the hypermedia style expresses problem domain details via messages instead of URIs.

entity orientation

entity designs focus on the entities or objects within the problem domain. for example "customers", "orders", etc. are entities. when implementing systems focused on entities, the most common methodology is to create network identities (URIs) that contain the entity names (i.e. http://example.org/customers/, etc.). implementors then use the high-level transfer protocol's network methods as "operators" for each entity.

for example:

"get a list of customers by date"

GET http://example.org/customers?order=date

"allow users to filter the orders list by sales region"

GET http://example.org/orders?region=...

"allow selected users to add new customers"

POST http://example.org/customers/

in the above examples, instead of using a message format to carry the domain semantics, entity oriented designs map the domain directly onto the available protocol and attempt to make URIs themselves carry meaning.

entity orientation is similar to RDF styles where the URI is treated as carrying intrinsic meaning and is the permanent identifier.

the appeal of this direct mapping of a problem domain to a network protocol is fairly obvious. it removes a layer of abstraction which exists for message-based hypermedia style. humans usually have little trouble mapping the limited set of HTTP protocol methods to simple operations on an "entity" (the classic Create, Read, Update, Delete === POST, GET, PUT, DELETE meme). finally, this entity view is easy to support via "tooling" since it essentially reduces distributed network interactions to passing internal objects around using only four possible operations.

but there are limits to this approach. first, entity orientation - while handy - is not how most humans think of life's interactions. usually they think first of the goal they want to achieve and then set up tasks in order to achieve it. second, it turns out to be quite difficult to get everyone to agree on the rules for forming identifiers. the HTTP world has experienced a decade-long dispute over how to handle permantent network identifiers and it still is not sufficently settled.

finally, it happens that many real world cases do not map well to the limited set of HTTP methods. usually there are some suggested compromises on how to handle certain operations within the problem domain (using HTTP.POST to send large search criteria, etc.). WebDAV solved this problem by adding seven additional HTTP methods that fit the targeted problem domain.

a matter of choice

in the end, it all boils down to choices. currently, implementing disitrubted network applications involves mapping the details of a problem domain to existing network standards. there is no getting around that; no choice there! but where we do have a choice is in how we go about accomplishing our task; the styles we use to map the domain to the standards. there are multiple ways to do this. i usually choose the hypermedia style since it tracks well with how i think about distributed networks, matches well with the existing strandards and technologies we use today ( URIs, HTTP, IANA) and presents very few restrictions when implementing clients and servers.

what implementation style do you use? and why do you prefer it?

this continues to be one of my favorite quotes from Roy Fielding. it captures not just some details about the HTTP protocol, not just the heart of the problem w/ attempts to use URIs as permanent identifiers, but also a fundamental observation of how humans actually operate in the world.

A resource is not the thing that is transferred across the wire or picked up off the disk or seen from afar while walking your dog. Each of those is only a representation. The same is true of physical objects encountered in life and never identified with URI and never made accessible on the net. Yes, it does present a bit of a quandary, but it is one that we have all learned to live with. Our eyes are not powerful enough to see identity through the representations, but our minds are powerful enough to associate identity to that which we see. Do I think of a different identifier every time I see my dog, or do I simply think of my dog as one identity and experience many representations of that identity over time (and on into memory and imagination)?

Roy Fielding - July 2002

this quote brings a smile to my face each time i read it.

it's taken some time, but i now have a pretty good idea of what i will be working on for the next year or so. i even have a sufficiently odd name for this work (always important, of course[grin]):

Afforded Agents

sufficiently, odd, right?

the problem

currently, the "device trend" is going up; we're getting more and more devices connected to the internet every day. last year Ericsson CEO Hans Vestberg predicted we'll reach 50 billion devices by 2020. not bad, eh?

along with this device explosion is a parallel (related) trend to progam each device individually; treat each new model to it's own "native" coding. billions of snowflakes.

another (unrelated) trend is the explosion of "open data." the notion that data should be freely available for anyone to access and use. and this is a big deal. last August, Franz, Inc reported successfully loading and querying a trillion triples.

a lot is good, right?

not all the time.

un-afforded

currently all these devices and all this data are just 'floating about' w/o any real connections. oh, yeah, they can be connected by people writing custom (usually one-off) programs that run on these devices and push and pull the same "open data" around, but that's the smallest of accomplishments. and this kind of work doesn't scale at the WWW level.

first, the data needs to be "afforded" - we need to see not just the "what" in a message, but also the "how." i appreciate the work the Semantic Web community has done in order to get folks thinking about the idea of messages having "meaning"; that's great. unfortunately, the same community has not done well in getting folks thinking about the idea of messages having "affordance" - use-ability built right into the data itself.

the first step in a new level of the WWW is "affording" data; making sure that "use-ability" travels with the data itself. then data is truely open and available. and it's scalable across any device, at any distance, in any location.

not enough agents

while the trend has been going up for devices and data, the trend has been going down for user agents. after more twenty years, the only really viable user agent for the internet is the common Web browser.

and that's just sad.

and worse, some browser vendors see their product as the "platform of the future." that's fine for browsers, but bad for the WWW; esp. if this is seen as the "end game" of user agent development.

the next step: we don't need more powerful user agents, we need more, less powerful, user agents

small, focused, fluent, composable

a much better, more robust future of the WWW is one where there are a wide variety of small, focused user agents. agents that do just a few things (possibly only one thing) very well. agents that understand maybe just one vocabulary; that can navigate a limited space on the WWW. one that understands a small set of data affordances.

and more than just fluent in one small thing, these tiny user-agents need to be 'composable' - need to be able to work together to accomplish more complex tasks by passing things between each other. we don't need a single super-agent that knows lots of vocabularies, understands lots of data affordances, etc. we need tiny SMEs that we can count on to accomplish a single goal effectively and can then pass their results to other SMEs. think of a search agent talking to a purchase agent talking to a delivery agent, etc.

once that starts to happen; once the trend for user agents starts to rise in the same "hockey stick" pattern shown by devices and data stores, then we have some possibilities. then we can get some things done!

we can do it, we have the technology

the good news is everything i've described is do-able. we have a long history of designing and expressing vocabularies, strong understanding of affordances and use-ability that can be applied to data (not just UIs), and boat-loads of data out there waiting to be "afforded." all that is needed, IMO, is some sustained effort on building, testing, and refining this idea of focused user-agents. and that's not complicated, it's just hard work that will take some time.

and that's what i plan to be working on for the near future. who wants to join me?

it's official! i've signed to do a new book for O'Reilly. the working title is "Programming the Web with Cloud9" and it covers the Cloud9 browser-based IDE for building Web apps.

i've been using the IDE for several months (actually put myself on an "all c9, all the time" regime for a while) and have really been impressed. in fact, over the last sveeral weeks i've actually been bummed out when i ran into a few cases where i actually needed to use my local workstation to handle a task instead of relying on "C9 and the Cloud" to do it all. hey, it happens[g].

hardware, software, wetware

while the main focus of the book is the Cloud9 editor, it will also cover other 'cloud-based' developer tools including CouchDB, github, and Heroku and more. in addition to the actual software and hardware, i am also digging into the 'wet-ware' aspects of cloud-based programming. this includes how cloud-based dev tools affect teams, collaboration, project schedules, code quality, etc. along the way, i will have the chance to meet and interview some very smart people who are active in shaping the future of this kind of Web developer's experience.

the journey begins

i'll officially kick off the start of the new project w/ a trip to Node Summit in San Francisco next week. if you're in the area, i'd love to meet you and get your opinions/predictions regarding these (and other) 'cloud-based' development tools. who knows, maybe some of your comments will end up in the book!

over the last couple months, i've been experimenting with a "new" development stack; what i call a "cloud stack." this development environment consists of a set of independent, but complimentary tools and services - all available via the Web. they make up a "full stack" for web development that is, IMO, quite enjoyable to work with. even better, i find it a very productive environment, too.

hello, cloud

i've noticed that, along with the rise of "the cloud" as an infrastructure model, there is also a corresponding development model. this model provides a cloud-based environment that includes editing, debugging, testing, source control, project mgmt, and deployment; all accessible from nothing more than a modern (i.e. HTML5-compliant) browser.

and, just as the process of virtualizing the hardware stack has acted as a distruptive technology for the back office, this emerging virtual programming environment will, i suspect, prove to be just as disruptive, too.

my current cloud stack of choice

here is my current toolkit for cloud-stack development:

Cloud9

i've really enjoyed using the Cloud9 browser-hosted code editor over the last few months. very similar (in some ways, related) to Mozilla's Bespin/Skywriter project, C9 is (for me at least) living up to the copy on it's home page: "Cloud9 is a state-of-the-art IDE that runs in your browser and lives in the cloud, allowing you to run, test, debug, and deploy applications from anywhere, anytime."

Node.js

i started using Node.js just about a year ago for a couple experiments; i've continued to enjoy working with it ever since. it's nice that Cloud9 supports runtime-debugging of Node.js, too. like the web site sez: "Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices."

CouchDB

along w/ using Node.js, i've started using Apache CouchDB as a data store. from the site: "Apache CouchDB is a document-oriented database that can be queried and indexed using JavaScript in a MapReduce fashion." right now i am testing two CouchDB providers: Cloudant and IrisCouch; both seem very solid.

Github

i've used a number of git-based repos lately, but continue to rely on github as my "go-to" source control tool. it helps that the Cloud9 editor also uses git locally and will let you clone any project from github easily. forking and code review are another thing that make github very enjoyable to use.

Heroku

i've recently started using Heroku as a deployment target for my completed Node.js projects ("Agile deployment for Ruby, Node.js, Clojure, Java, Python, and Scala."). again, it's easy since Cloud9 supports git-based deployment to Herkou directly from the editor interface. it took a bit of "tweaking" to get my git master properly 'prepped' for Heroku, but once that's done, incremental updates are a snap.

one of the nicer aspects of the above stack is that you can get started with each of them at no charge. IOW, it's essentially free to experiment. not bad at all.

the bigger picture

so far, this is the stack that works for me. there are a number of other products/services out there that i haven't yet tried, but basically it boils down to the same general programming set: editor/debugger, storage, source control, and deployment. i've left out things like testing and other helpers as i am just getting around to exploring those tools, too.

but, regardless of the details, i think the pattern is clear; there are quite a number of options for putting together your own 'cloud-stack' development environment. and i suspect this kind of environment will grow popular in time. along the way, i bet we'll see other (major) vendors make offerings in this space, too. when that happens i suspect the market for this kind of work will continue to grow and get even better.

have you assembled a 'cloud-stack' programming environment? if yes, let me know. i'd love to hear about other experiences.