REST : Cornelia's Weblog

Posts Tagged ‘REST’

More on PUT Idempotency

February 19th 2011 Posted at REST, ResourceOrientation, WebServices
0 Comments

This has been a hot topic in my world this week. I have a couple of colleagues with whom I have been having very good discussions (in the context of two totally separate projects) and I wanted to capture one particularly interesting thread here.

One of those conversations was started with a reference to a really good blog post from Alex Scordellis addressing the question of how complete representations sent in a PUT need to be. What exactly does the client need to include in the representation that they PUT – all properties? what about hyperlinks? A simple answer might be that they need to provide “everything”, but this simple answer is just not satisfactory.

When a client provides a representation as a part of a PUT (or a POST used for creation), the server may ignore some of that representation. The simplest case is where a client, intentionally or accidentally, provides a new value for a non-writable property – say something like a server generated ID for the resource. Further, as Alex pointed out in his post, the client shouldn’t be responsible for determining the application flow, which per HATEOAS, is provided via hyperlinks in the resource representation; even if the client provides them, the server will likely ignore such links. So, if the server will ignore some of these things anyway, why not allow the client to not provide certain pieces? Makes sense to me. But how do we do this “right”?

Alex’s post summarized some conclusions that he and others reached during a healthy debate/discussion on the subject. I really like what he said there:

In response to GET requests, services serve complete representation of the current known state, including business data and available hypermedia controls. Clients PUT complete representations of the parts for which they are responsible.

He called this a “rule of thumb” and I think it is completely correct, even if it leaves plenty of room for misinterpretation. This was the essence of the conversation I had with a colleague this week.

First, there is something in this rule of thumb that is easy to miss but absolutely critical; the word “complete” in the second sentence. If the client were allowed, in a PUT, to provide only a subset of the parts for which they are responsible then the problem reduces back to the general problem with using PUT for partial updates. That type of a PUT implementation is not idempotent.

But we have to go just a bit further. This only works if every client is responsible for the same parts. You cannot, for example, have one client responsible for only property A and another client responsible for only property B and still have an idempotent PUT (you’d have this example all over again).

There are other edge cases as well. What if the resource state were effected by another resource operation? We know that we can have two resources share parts (or all) of their state; for example, the current version of a document might share the same state as version 3 of a document, specifically, when version 3 is the current version. In this case the set of properties a client is responsible for on a current document cannot be different from the ones the client is responsible for when accessing a particular version of a document. And if you have resources that are what I like to call “composite resources” then you have to watch out for the same issues, but then it starts to get very complex.

There are some choices, however, you can make to simply things a bit. For example, you can choose to make the client responsible for all properties except write-once properties, hyperlinks, and server computed properties (i.e. hash value). In other words, as you are deciding what to make the client responsible for, KISS.

I recognize, of course, that Alex’s post isn’t a specification and hence I do not offer my remarks here as a criticism of his post, rather, because I’ve already seen various ways of interpreting that post in action, I offer this only as an elaboration. The key is that whatever you do with PUT, make sure that it is idempotent. If you get very clever on how you define your service you will have to be equally careful to make sure you respect that constraint.

Jax-RS : Regular Expressions in @Path Annotations

October 20th 2010 Posted at Bug, CXF
4 Comments

Sometimes I really wish I did something else for a living. Okay, perhaps a bit over-dramatic but I’m feeling a bit tired this evening and wasting even 20 minutes on something totally useless (which, let’s face it, isn’t unusual in computing) has me ready to call it a night. But since googling the error codes I was getting netted nothing (other than this 2 year old thread), perhaps I can be of service to the next person bitten by this.

I’m writing a RESTful web service, using Jax-RS with a CXF runtime (v 2.2.10). I’m defining a resource with a URI something like /foo/bar/… – that is, the first part of my uri will have the literals “foo” and “bar” and then I want everything else on the URI to go into a parameter. So I want /foo/bar/a, /foo/bar/a/b and /foo/bar/ab/c to all resolve to the same resource/method with a path parameter bound to “a”, “a/b” and “a/b/c”, respectively. So I create the follwoing @Path annotation:

@Path("foo/bar/{therest : .*}")

My initial source for this tidbit was the Safari Books Online copy of Restful Java with Jax-RS, page 47 which showed an example just like this.

When I tried to launch my web service, however, I got an error that indicated something was wrong with the regular expression.

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'Prototype': Error setting property values;
nested exception is org.springframework.beans.PropertyBatchUpdateException;
nested PropertyAccessExceptions (1) are:PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'serviceBeans' threw exception;
nested exception is java.util.regex.PatternSyntaxException: Illegal repetition near index 27/foo/bar/{therest : .*}(/.*)?...

The answer was really simple, get rid of the spaces around the ‘:’. Argh! Since when do languages care that much about whitespace?! And Argh again! How can a published book have this error?! Or perhaps the bug is in CXF? I’ll look into that when I have a chance.

Why PUT can’t be used for partial updates

September 2nd 2010 Posted at Standards
4 Comments

I’ve been discussing PUT vs. PATCH with some colleagues and finally took the time to come up with a concrete example of why PUT absolutely should not be used for partial updates of resources. One colleague pointed out the following points made by Roy Fielding on the AtomPub listserv:

FWIW, PUT does not mean store. I must have repeated that a million times in webdav and related lists. HTTP defines the intended semantics of the communication — the expectations of each party. The protocol does not define how either side fulfills those expectations, and it makes damn sure it doesn’t prevent a server from having absolute authority over its own resources. Also, resources are known to change over time, so if a server accepts an invalid Atom entry via PUT one second and then immediately thereafter decides to change it to a valid entry for later GETs, life is grand.

Roy’s right, of course, but there is a subtlety that is easily overlooked. Sure, the server has the authority to “fix” the resource representation, for example, it might modify part of the resource representation passed in (i.e. if the client tries to modify the server controlled id property, the server can ignore that), but there is an important constraint that the server has to follow. PUT needs to be idempotent! This means the server can’t use the current state of the resource to “fix” the resource representation passed in. Have a look at the following simple example:

Let’s say that I have a resource that consists of a couple of properties, A and B, and I want to use PUT to support partial updates of this resource. The payload of the PUT will carry the properties that we wish to update; if a resource property is not present in the payload then that property will remain unchanged in the resource. We’re playing the “the server has control over what they do with the resource property” card. Check out the following sequence of operations then:

Ouch! The PUT is not idempotent! Client 1 does the same PUT twice, leaving the resource in two different states – one of which could very well be inconsistent. PUT semantics, when correctly followed, give the client the ability to control the consistency of the resource they are writing.

Hence RFC 5789 – PATCH Method for HTTP. PATCH does not have the constraint of being idempotent so the client cannot simply retry requests for which they never received a response. I do think it is interesting that RFC 5789 includes the following:

A PATCH request can be issued in such a way as to be idempotent, which also helps prevent bad outcomes from collisions between two PATCH requests on the same resource in a similar time frame. Collisions from multiple PATCH requests may be more dangerous than PUT collisions because some patch formats need to operate from a known base-point or else they will corrupt the resource. Clients using this kind of patch application SHOULD use a conditional request such that the request will fail if the resource has been updated since the client last accessed the resource. For example, the client can use a strong ETag [RFC2616] in an If-Match header on the PATCH request.

That is, you could make PATCH idempotent if you use the likes of ETags and conditional invocations. The same could be said for a PUT that does partial updates but the opportune world here is “could” – PUT is required to be idempotent even without the use of things such as ETags.

All of that said, while neither RFC 2616, nor RFC 5789 require the use of ETag like mechanisms, in most cases it’s a good idea.

Second Version of the XML REST Framework Released

May 14th 2010 Posted at HowTo, OpenSource
0 Comments

To coincide with EMC World and the presentation I gave on RESTful SOA, I’m delighted to report that we have released the second version of the XML REST Framework. As I described in a previous post, the April 18 release introduced a framework that allowed a services developer to produce RESTful services over EMC Documentum xDB by “declaring” the resource model, including definition of the uniform interface and media type handling, with Jax-RS annotated POJOs and by using XQuery to implement the operations of the service. This was a good start, yet didn’t really go far enough to even call the resulting services RESTful because the resource representations that flowed into and out of the services where completely devoid of hyperlinks.The new version of the framework now provides the developer a mechanism for insertion of hyperlinks, again leveraging XML standards to do so – this time I’m talking XSLT. Just as with the first version of the framework, the code that goes into your Java POJOs is very thin, essentially calling into the framework that in turn invokes a series of XML-centric steps. The job of the developer then is to inject (using Spring dependency injection) the XML processing artifacts that achieve the required functionality. With the first version of the framework that was an XQuery per operation, and now in version two it is an XQuery plus one or more XSLTs. In version one of the framework you there was only one option available – to execute a single XQuery; in this new version you have the choice of three different “pipelines” – XQuery alone, XQuery followed by an XSLT and XQuery wrapped by XSLTs before and after. Included in the sample application that is part of the framework release are some XSLTs that make it very easy for the developer to simply declare the links they want inserted – no great XSLT expertise required.The EDN postings go into a good bit of detail on how you can leverage the framework. Each posting has two main parts – code and documentation. If you haven’t already downloaded and installed the code from the first release then go directly to the code in the second release; it is a superset of the former. For documentation, please start with the first release – it describes the base framework – and then move on to the docs for the second release; the latter adds to the former rather than replacing it.Finally, yes, there is more to come. We have a very early implementation of the next version of the framework running in the lab (thanks to my very able colleague Xiaotao Liu). Stay tuned, I’ll give you some hints as to what will be included over subsequent posts. I promise it will be fun.

RESTful SOA at EMC World 2010 and …

May 14th 2010 Posted at Speaking
0 Comments

It’s been a really good week for me. I didn’t attend EMC World last year because, well, I didn’t really have much to talk about. But in the last year the work that we have been doing in the CTO Office around an integration architecture that has emerged as Service-oriented and RESTful has reached a point where there is not only plenty to talk about but, wow, there is a stream of additional work that follows on from this. I’ll be busy for a while. Aside from having the opportunity to attend some great sessions and connect with a whole slew of really talented people, I also had the opportunity to present a session in the Developer Track of the conference.The title of the presentation was RESTful SOA, Developing RESTful Web Services for Distributed Applications (Like Cloud). (Hmm, as I type that title now I realize, of course, that “Cloud” is not an application, but I think you know that I mean “like those running in the cloud.”) I was scheduled in the last slot on the last day of the conference – OUCH! All week I spoke with people who said “I’d love to see your presentation but I’ll already be on my way home.” So I expected around 10 people, half of which would be colleagues, and low and behold, I think there were more than 50 in the room!! That really speaks to the interest that people have in this still emerging architectural style.After spending just a moment to introduce what we do in the Architecture Group of the CTO Office (work on an integration architecture) and why we are embracing a RESTful approach, the presentation broke out into two almost equal halves.In the first chunk I talked about REST in general. I still feel that most folks understand only a part of the four fundamental cornerstones of a RESTful architecture. Those cornerstones are:

A resource centricity and addressability
Abiding by a (the?) uniform interface
The use of media types for resource representation
And hyperlinking

For each of these cornerstones I covered what it is and why it’s important. In the discussion on the uniform interface we drilled down the the characteristics of safe, idempotent and cacheable (Joe Gregorio talks about “zippable” as well – I don’t have any experience with that one yet so have little to say at this point). Rather than writing a whole chapter detailing these things let me give you a brief analogy – the web as we know it today. It works as well as it does, it scales as well as it does because it is RESTful. You can access web resources (web pages) via their URLs, new web sites can be added with no performance degradation in part because of the layered web and cacheable operation results, the network can be unreliable yet applications still work because of safe and idempotent operations, a range of different devices and application types all work against the web, and you can navigate to related resources, without going through some central point of control, by simply invoking embedded hyperlinks. Way cool.Following this discussion on RESTful architectures in general, which took us through the first half of the hour, I moved on into pragmatics and talked about an approach for implementing RESTful services. Using the steps outlined in Richardson and Ruby we quickly covered how to implement services that achieve resource orientation and addressability, implement a uniform interface and support various media types. We use Java and Jax-RS annotated POJOs to declare our services and for implementation I showed how you can use XML technologies such as an XML database like EMC Documentum xDB and an XQuery engine (a part of xDB) to avoid a lot of grungy Java code. The steps for the services developer are very prescriptive; assuming the resource model and physical model are designed she will:

Declare the resource model and uniform interface; a Java class per resource, a method per uniform interface operation supported.
Craft an xQuery to mediate from the resource model to the physical model and to implement the service operations.
And glue it all together using spring.

The first version of the XML REST Framework we have created can be found on the EMC Developer Network and the article that delivers the release goes into much more detail on how all this works.BUT, we’re not done. The resource representations we produce and consume using the first version of the framework do not include hyperlinks. Can you imagine a Web where you have to go back to some index each time you want to access another web page or complete your Amazon purchase? Or one where you have to know the exact URL for EVERY page you want to access? Well, without links in resource representations that is exactly what we are forcing the client programs to do. So the final part of the presentation covered the extensions we made to the first version of the framework to enable the services developer to easily insert hyperlinks to related resources within their resource representations, using XSLT. For details on how this all works please see the second version of the XML REST Framework, also found on the EMC Developer Network.Unfortunately, with so much material to cover I had little time left for questions during the session but fielded quite a few after we had broken up. If you were in the session, or even if you weren’t, and have any questions, fire away! Thanks to everyone who attended EMC World this year – and especially those of you who stuck around until the end.

Building Domain Specific RESTful Services with XML Technologies

April 19th 2010 Posted at HowTo, OpenSource
0 Comments

Some colleagues and I have been very busy in the last months putting together a framework to do just that. Let me briefly back up a bit…One of the things that I’ve spent some time on in the last couple of years is the Content Management Interoperability Services (CMIS) standards development effort. One of the styles for exposing this set of services is through a RESTful interface. Through incremental improvements happening through the community development process the result is a pretty nice, and quite RESTful, interface. That said, Roy Fielding levied some pretty harsh criticisms toward it when it was first made public – the vast majority of which are now addressed. One issue from Roy’s list remains but is, frankly, something that CMIS didn’t intend to address, and that is the concern that the services exposed over a CMIS compliant repository do NOT expose the domain specific semantics in the resources. Roy is, of course, dead on on this, what CMIS exposes is the repository model, not your domain specific model. Yes, CMIS does provide a service where object types can be obtained by a client, but those details are not a part of the protocol, rather they are tunneled through the protocol. And, while CMIS resource representations do include hyperlinks between related resources, again the related resources are repository resources (i.e. this document is contained in this folder) not domain specific resources.This is an all too common pattern – we build interfaces that expose the repository models – and, of course, this is not at all without use as it does help consumers more rapidly build applications. Unfortunately, it has the unintended consequence of encouraging architectures where all of the domain specific details are all too often included far too far from the server. My boss calls this the “semantic gorp”, and in the best case it lands on an app server somewhere embedded in some services that in turn call the repository-model-centric services, but in the worst case it ends up far further away. This causes all sorts of headaches, which I’ll go into details on in subsequent posts… for now I’ll just call on your intuition to buy that point.(Okay, so that wasn’t entirely brief, but important context nonetheless.)In the deployment of ECM systems you go through a process where you design your domain specific artifacts, content types, lifecycles, etc. You then (ideally) configure the ECM to provide services for those domain specific entities. Wouldn’t it be cool if you got domain specific services as a result of that process? Now, don’t misunderstand, we haven’t built that (yet), but what we just published to the EMC Developer Network is a step in that direction.We have built a framework that allows for the development of domain specific RESTful services over a repository that supports xQuery. In particular, the implementation available for download operates over EMC Documentum xDB. Using the framework, all of the details of connecting to xDB are abstracted away from the developer, and he or she produces the RESTful services by

“declaring” the resource model in Jax-RS annotated Java classes
defining the uniform interface with more Jax-RS annotations (i.e. @GET, @PUT, @POST, @DELETE, etc.)
producing an XQuery to interface each RESTful operation for each resource to the physical manifestation of that resource
and binds it all together in a spring configuration file

In this first release of the framework we are mostly leveraging Jax-RS for REST features, which means that one of the most important features of RESTful interfaces, hyperlinking, is lacking. This will follow in version 2 of the framework in a few weeks.One of the primary drivers for this work was to simplify the development process for these services – if they are easy to build then we should see more domain specific services where otherwise, generic services might have been. This is already proving true in that we are leveraging the framework on several projects internally at EMC. The developers were able to concentrate on their resource models, representations and interactions rather than getting bogged down in the details on how to implement them. Sure, they still need to touch java code and spring configuration files, but it’s not a huge leap to see that tooling could shelter the developer from those details – we’re just focusing on the developer model, not the tooling(, for now?).The framework, including source code, plus a sample application leveraging the framework is available here. Also, for any of you that will be at EMC World in a few weeks, I am giving a talk on this framework – in the highly coveted last slot of the conference in the developer track (Thursday afternoon at 1PM). Hope to see you there.

What happened to the FUSE->FUSE Project option in the eclipse plugin?

August 27th 2009 Posted at HowTo, OpenSource
0 Comments

A colleague of mine was still running eclipse with an older version of the FUSE plugin – in particular 1.0.0. He had a single option under the New->Project->FUSE category called “FUSE Project”. Selecting this option created what was essentially a web services project (in particular we are using it for RESTful services) that had all of the libraries set up, etc.When I installed the latest version of everything, following the instructions in the FUSE installation guide that option was gone. Under the FUSE category there are now three options that are all essentially Camel focused – EIPs and the like. I did some digging and found that creating a web services project set up for FUSE (Service Mix – CXF) is now done via project options shipped with the Eclipse Web Tools Platform. Here’s how it works:Select File->New->Project and then under Web select Dynamic Web Project. Clicking “Next” presents the following dialog:Under the Configuration section select the CXF Web Services Project v2.5 option and that’s it. Exporting this project to a war includes all of the necessary jars. This does, of course, require that you have set your CXF runtime preferences appropriately. In Windows->Preferences:The only gotcha I ran into was that plugin installation order matters. It’s still not clear to me if and when WTP is included in Eclipse IDE for Java EE Developers – the WTP 3.1 Release page says it does with Galileo but I was constrained to using Ganymede for other plugin compatibility; installing Ganymede Eclipse IDE for Java EE Developers did NOT seem to include the WTP, at least not the CXF pieces. Assuming WTP has to be installed, and assuming I want to install the latest FUSE integration designer, you have to install WTP first. When I did the FUSE plugin first I had several problems, most notably there was no CXF 2.x Preferences option under the Web Services preferences (see screen shot above); this is an option that is added when you install the appropriate parts of the WTP.UPDATE 1/22/2010:I’ve just installed Galileo and while it does include WTP, it does not include the CXF portions. Go to the update sitehttp://download.eclipse.org/webtools/updatesand then expanding each of Web Tools Platform, Web Tools Platform SDK and Web Tools Platform Tests (optional) select the CXF pieces – there is one in each category.

Getting the Apache CXF Jax-RS Basic sample running

May 2nd 2009 Posted at HowTo
0 Comments

I did it in January – just came back to it today and could not remember what I did. Where is my memory? So for my benefit, and perhaps the benefit of others, I’m writing it down. There are hints in the README that ships with the sample, but it was far from step by step.Of course, there are a number of different ways you can get this running – this is just one way.My environment:

FUSE Services Framework (CXF) version 2.1.3.2
Eclipse 3.3.2

And here’s what you do.

Start up Eclipse with a new or existing workspace and use the Java perspective.
Create a new project by selecting File->New->Other… and then Java->Java Project from Exiting Ant Buildfile. Enter the name of the project, something like “Basic Jax-RS Sample”. Browse to {FUSE Install dir}/samples/jax_rs/basic/build.xml. Check the box that reads “Link to the buildfile in the file system” and click Finish. Note that this checkbox does NOT make a copy of the sample directory, nor does it create a separate build.xml, rather uses the one that is in the sample directory. This build file references another build file that is relative to the build.xml file so in order to use the build file without changes it must be invoked from that directory.

This is all you need to do to set the project up. Now you can run it. We are going to do so all inside of eclipse. First we’ll run the server part of the sample and use a browser to invoke the services – just reads. Then we’ll run both the server and a simple client application from within eclipse. Read on.

Running the Server and GETting resources

Expand the Basic Jax-RS Sample project
Right click on the build.xml file and select “Run As->Ant Build…” – be sure to to select the “Ant Build…” with the dot-dot-dot so you can select what you want to build and run.
Now you can fetch some resources – note that a few are created for you when you run the server. Invoke a URL either http://localhost:9000/customerservice/customers/123 or http://localhost:9000/customerservice/orders/223/products/323. Notice in the code that customers, via the path “customers/{id}” and products, via the path “orders/{oid}/products/{pid}” are the only two resources that support GET.

Also notice that the server only stays up and running for 5 minutes by default – if you wish to lengthen this you can change the line of code in the Server.java file that reads:Thread.sleep(5 * 60 * 1000);And that’s it. You are running the application. If you want to test out some write operations…

Running the Server and a Client Application

Both the server and the client can be run from within eclipse.

If your server is not running follow the steps outlined in the section above – basically do a Run As->And Build… and select the server.
Now run the client by right-click on the build.xml, Run As->Ant Build… and this time select “client” (deselect “server”)
You can see the results of the execution in the console pane. Customer number 123 was updated and a new customer was created and assigned an id of 124 by the server. In fact you can now invoke the URL http://localhost:9000/customerservice/customers/124 to see that new customer resource.

That’s it, the basics of getting the application up and running.

Cornelia's Weblog

my sporadically shared thoughts on, well, whatever is capturing my attention at the moment.