OAuth

In my mashup framework, OpenID is used to do authentication which provides a way to make sure the users are really who they claim to be. The mechanism is that the users prove that they own/control the OpenID they provides. Here is what I wrote about OpenID.

However, OpenID just solves authentication. After users are authenticated, OpenID does not specify how the third-party apps access the protected data on service providers. And the requirement that third-party apps can access protected data stored on service providers on behalf of users is not imaginary, but real. In my mashup framework, I need to manipulate data on behalf of users which needs authorization besides authentication. Youtube and Flickr both provide ways for third-party integration. They provide similar authorization procedures (I am using this mechanism right now) which fundamentally match principles of OAuth specification.

OAuth provides a way for users to grant access to third-party apps. The procedure is pretty much the same as OpenID. What is different are parameters in requests and responses. OAuth is an independent specification which means it is not an extension of OpenID. But, in my opinion, it may be better to combine these two together because they are similar to each other. In this way, user experience can be improved because authentication and authorization can be done in the same pass instead of two. However, some sites may just support authentication service or authorization service (in terms of support of open standards like OpenID, OAuth). So how to combine them together without losing flexibility is an issue.

One problem that OAuth does not address is fine-grained access control. I think this issue must be addressed in the long run. However, nowadays, the goal is too far to reach. Actually, current version of OAuth is not supported widely.

DataPortability is a new organization which aims to promote data portability. Of course, it includes OpenID and OAuth with respect to authentication and authorization data portability. I am not sure which ones will survive and stand out  eventually, but this is a good signal that data portability is being considered as a critical issue and I hope it will be addressed in the near future.

Authentication and Authorization in Mashup

Authentication
Currently, I don't build my own independent user authentication system. I make use of OpenID to utilize existing authentication systems, such as Yahoo Flickr, Google Blogspot, LiveJournal... You can check whether you have already had an OpenID here. I wrote a simple introduction. In essence, we delegate the task to existing authentication systems.
After a user is authenticated successfully, OpenID is used to track the user. Servlet session technique is used to associate session id and user information.

Authorization
Currently, I don't ask end users for their usernames and passwords in existing services (e.g. youtube, flickr) because it requires deep trust of end users. The user id redirected to an authorization web page hosted by the destination service(e.g. youtube). If the user accepts the request, then the user browser will be redirected to my mashup application. Generally, an auth token is appended to the URL which will be used in following requests.
Currently, for every backend service, I request the user to authorize the most powerful privilege level. For example, if the service supports READ and WRITE permission control, my application will request WRITE permission. If the service supports READ, WRITE and DELETE permission control, my application will request DELETE permission. This is not always a good strategy. But because of diversity of permission control systems, I must choose the mechanism which can be used in every possible service.
Maybe in the future, I can come up with a better framework with fine-grained access control.

URL Pattern In Mashup

URL Pattern

Note: All variables enclosed in ‘{’ and ‘}’ must be substituted with real values. GET means HTTP GET request type and POST means HTTP POST type.

/cgl/feeds/api/{category}/{service}/users/{userid} GET:     Get information of a user.

/cgl/feeds/api/{category}/{service}/login GET:      Login the service.

/cgl/feeds/api/{category}/{service}/users/{userid}/resources GET:     Get all resources of the user. POST:     Add a new resource.

/cgl/feeds/api/{category}/{service}/users/{userid}/resources/resourceid /cgl/feeds/api/{category}/{service}/resources/resourceid GET:     Get information of a specific resource. DELETE:     Delete the corresponding resource. PUT:     Update the corresponding resource.

/cgl/feeds/api/{category}/{service}/resources/resourceid/comments GET:     Get all comments of the resource. POST:     Add a new comment.

/cgl/feeds/api/{category}/{service}/resources/resourceid/rating GET: Get rating of the resource.

/cgl/feeds/api/{category}/{service}/tag/{tag}?perpage={perpage}&page={page} GET:     Get records by tag. Parameter perpage indicates how many resources you want to retrieve.Parameter page indicates which page you want to retrieve.

/cgl/feeds/api/{category}/{service}/statistics/{standard}? perpage={perpage}&page={page} GET:     Get records by standards.     E.g. most viewed, most recent…

Example:

/cgl/feeds/api/videos/youtube/tag/soccer
get resources tagged as soccer.
/cgl/feeds/api/videos/youtube/statistics/most_viewed
get resources that are most viewed.
/cgl/feeds/api/pictures/flickr/users/testuser/resources
get resources belonging to user testuser.

Javascript Templates

Javascript Templates(http://code.google.com/p/trimpath/wiki/JavaScriptTemplates) is a sub project of project Trimpath. It is a client-side template engine whose functionality is similar to server-side Velocity and Smarty engines. It uses special markup syntax. Besides simple template matching and replacing, it also supports simple control flow, loop and variable declarations... Now I summarize the syntax:
Statements:
(1) Variable Declaration

Syntax	Example
{var variable} {var variable = value}	{var price} {var price = 10}

Note: No ending ';' is needed.

(2) Control Flow

Syntax	Example:
{if expression}     {elseif expression} ...        {else} ... {/if}	{if price > 10}     {elseif price <20}...       {else}... {/if}

Note: No surrounding brackets are needed in test expression. No white space between first '{' and 'if'.

(3) Loop

Syntax	Example
{for varariable in list}    statements go here {/for}	{for price in pricelist}     ${alert(price)} {/for}

(4) Expression

Syntax	Example
${expression}	${car.brand}
${expression|modifier}	${car.brand|capitalize}
${expression|modifier1|modifer2}	${car.brand|default:"chevy"|capitalize}
...	...

Value of the expression will be returned.

Note:
Statement for( ; ; ) is not supported.
These statements are parsed and processed by Javascript Template Library instead of javascript itself.

More:
(5) CDATA Text. Tag is cdata.
CDATA Text will be ignored and will not be processed.
(6) In-line javascript block. Tag is eval.
Multiple line javascript code can be put in a single block to evaluate.
(7) If you have multiple-line js code, you can use minify to tell processor to treat them as a whole.
For example:
<input type="button" onClick="{minify} Javascript event
handling code...
{/minify}">

Note: eval and minify are different althought they are sort of similar.

How to process templates?
API: http://code.google.com/p/trimpath/wiki/JavaScriptTemplateAPI.
Commonly used functions:
(1) TrimPath.parseDOMTemplate ( elementId, contextObject )
This function retrieves content of the element corresponding to the first parameter. It should be a textarea element and its should be hidden.
The second parameter is a context object and it will be "merged" with the template. For example, if a template refers to ${car}, then the processor tries to access contextObject.car.
(2) TrimPath.parseTemplate ( templateContentStr )
This function accepts a string as template instead of a dom element id.
You can parse a string template in this way:
TrimPath.parseTemplate( stringvariable ).process( contextObject ).

Variable Resolvation Precedence:
There are three kinds of scopes:
(1) Processor scope: it consists of all template statements.
(2) Context object scope: All properties of the context object.
(3) Local scope: local variables whic are not defined in template.

Case 1: In template expression and {eval}:
Processor scope is checked first, then context object is checked. At alst, local scope is checked.
For example:
    var data={car : "chevy"};
    var str = '${alert(car)}';
    str.process(data);        //alert "chevy"

    var str2 = '{var car = "ford"} ${alert(car)}''
    str2.process(data);      //alert "ford"

    var car = "toyota";
    var str3 = '${alert(car)}';
    str3.process();      //alert "toyota"

Case 2: All variables defined in template do not have effect in local scope.
For example:
    var str = '{var car="chevy"}${alert(car)}';
    str.process(data);        //alert "chevy"
    alert(car);                   //variable 'car' is not defined.

Template

Here is link which suverys browser-side templating: http://ajaxpatterns.org/Browser-Side_Templating.
The goal of browser-side templating is to separate presentation and logic. And in my application, I think it will be helpful to use browser-side templating to ease the development. Currently, JSON messages are received from server and then formatted accordingly to display them to end users.

In next step, I would like to try JTL(Javascript Template Library) http://code.google.com/p/trimpath/wiki/JavaScriptTemplates.

OpenID

Some text is excerpted from Official Specification.

[Resources]

Official web site: http://openid.net
Authentication specification: http://openid.net/specs/openid-authentication-2_0.html
Following is a very good and detailed tutorial about how to develop a web site which makes use of OpenID: http://www.plaxo.com/api/openid_recipe.
Libraries written in various languages: http://wiki.openid.net//Libraries.
This web site lists many web sites which support OpenID: http://openiddirectory.com/.

[Introduction]
       Previously, when we visit some web sites and want to leave comments, generally we need to sign up an account at first. This is kind of annoying because we might just surf the web and we might not visit the website again in the future. Applying for a new account is not a big deal.
       By using OpenID, we can be authenticated in a new web site by signing in an OpenID provider (e.g. Blogger, Flickr, Youtube…) with existing account. In other words, we could use the same account to log in to multiple web sites/applications. Generally, most of users have account in prevailing OpenID providers. These providers (e.g. Flickr, Youtube ...) provide their own core compelling services (e.g. video sharing, picture share…) besides OpenID support. Besides, there are some new sites which mainly provide OpenID support (e.g. myopenid.com,).
       An assumption is that OpenID providers are trusted by various OpenID-aware web applications.
       The advantage of OpenID is that:
       Users can use OpenID to log in to multiple web sites which support OpenID without having to apply for an account. As a result, users don’t need to remember large number of usernames and passwords, which improves user experience greatly.

       The mechanisms used to achieve the goal are: redirection and callback.

[Architecture]

1) Background
There are two kinds of communication: direct request and indirect request.
Direct request: one party sends request directly to another party to get response. The message must be encoded as a POST body.
Indirect request: one party sends request to another party by redirecting user agent to the destination party with request data (query string or POST body).
There are two mechanisms which can be used to send indirect request: HTTP indirection or form redirection.
HTTP redirec:
       “Data can be transferred by issuing a 302, 303 or 307 HTTP redirect to the end user’s agent. The redirect URL is the URL of the receiver with the OpenID authentication message appended to the query string.”
HTTP Form redirection
       A HTML page which contains an HTML form element can be returned to user. And the form contains a mapping of keys to values. Javascript can be used to automate submission of the form to implement redirection.
More info: http://openid.net/specs/openid-authentication-2_0.html#anchor6

2) Procedure
First, the Relying Party presents the user with a form that has a field for entering User-Supplied Identifier. The form field’s name should be “openid_identifier” so that user agents can detect it automatically and provide some extra functionality.

Steps:

(1)    The user submits the from to Relying Party
After receiving data submitted by user, Relying Party normalizes the input data and then discovers OP endpoint URL based on user-supplied identifier.

(2)    This step is optional. Relying Party can build an association with OP.
An association establishes a shared secret between them which is used to verify subsequent protocol messages and reduce round trips.
More info: http://openid.net/specs/openid-authentication-2_0.html#associations

(3)     Relying Party sends authentication request to user agent.
User agent sends authentication request to OP. Then user is authenticated by the OP.
These two sub steps jointly do this: Relying Party send an authentication request to the OP to obtain an assertion. This is an indirect request. So the request actually is sent by Relying Party to user agent and then redirected to OP.
Generally, OP will display a web page which enable user to accept or deny access request from Relying Party.

(4)    OP sends authentication response to user agent and then user agent sends response to Relying Party.

(5)    After receiving authentication response from OP (indirect communication), Relying Party verifies it and judge whether it is a positive assertion or negative assertion.

More info: http://openid.net/specs/openid-authentication-2_0.html#responding_to_authentication

[More]
       Besides authentication, personal information (e.g. gender, location, age …) can be transferred between Relying Party and OpenID provider as well. Then Relying Party can pre-fill some registration fields so that users don’t need to type the same information many times. Of course, Relying Party needs to convert information from OpenID provider to its own format.

[Delegation]
Delegation is also possible: http://simonwillison.net/2006/Dec/19/openid/. It means users may want to delegate their OpenID to another provider behind-the-scenes. An example from http://www.plaxo.com/api/openid_recipe:
       “If I try to sign up with the OpenID josephsmarr.com, I may have actually delegated that URL to a different OpenID like jsmarr.myopenid.com, and when the provider returns to you to complete authentication, you need to remember that I wanted to sign up as josephsmarr.com and not jsmarr.myopenid.com.”
Key thing is that you add following two lines of HTML code to your delegation HTML page:
       <link rel="openid.server" href="http://openid.server">
    <link rel="openid.delegate" href="http://delegation.site">
You should replace the value of href attributes accordingly.

This mechanism actually adds an additional layer which separates Relying party and OpenID provider. Relying Party gets OpenID endpoint URL by accessing your delegation HTML page. After getting value of open.server and openid.delegate, Relying Party can sends authentication request to corresponding OpenID provider. You can easily switch your preferred OpenID provider by modifying value of attribute href in link elements.

[Myth]
       OpenID is a mechanism which can authenticate users by delegating it to corresponding OpenID provider. So it is not appropriate to use it directly in a third-party application which sits on top of an existing service and wants to access the backend service on behalf of the user. For example, if a web site makes use of Youtube API to do some cool things, it necessarily needs users’ information to utilize Youtube on behalf of the user. OpenID does not solve this problem.
       But, in my opinion, this can be done by extending OpenID specification. In authentication response, OpenID provide can include an auth token which can be used as a handle by Relying Party to access service provided by the OpenID provider on behalf of the user. In addition, access level (Read, Write, Delete …) must be negotiated as well.
       However, the extension deviates from gist of OpenID because authorization delegation is involved besides authentication.

Youtube API usage

Authentication

       http://code.google.com/apis/youtube/developers_guide_protocol.html
       It is recommended that you include the proper authentication headers in all of your requests even if those requests do not explicitly require authentication.
       A request must include the X-GData-Key and Authorization headers.
       The X-GData-Key header specifies your developer key, a value that uniquely identifies your application(s). The Authorization header specifies a token that you obtain for each user using one of two authentication schemes, AuthSub or ClientLogin. Headers are like this:

AuthSub	ClientLogin
Authorization:AuthSub token=<authentication_token> X-GData-Key: key=<developer_key>	Authorization:GoogleLogin auth=<authentication_token> X-GData-Key: key=<developer_key>

(1)    Functions supported in AuthSub
http://code.google.com/apis/accounts/docs/AuthForWebApps.html

(1.1)           AuthSubRequest

       A call to this method sends the user to a Google Accounts web page, where the user is given the opportunity to log in and grant Google account access to the web application. If successful, Google provides a single-use authentication token, which the web application can use to access the user's Google service data. This is done by using GET to a specific formatted URL.

Sample URL

https://www.google.com/accounts/AuthSubRequest?scope=http%3A%2F%2Fgdata.youtube.com&next=http%3A%2F%2Fzhguo.blogspot.com

(1.2)           AuthSubSessionToken

       A call to this method allows the web application to exchange a single-use token for a session token.

Sample request
       curl https://www.google.com/accounts/AuthSubSessionToken --header ‘Authorization: AuthSub token="token"’

Response

Token=DQAA...7DCTN

Expiration=20061004T123456Z

(1.3)          AuthSubRevokeToken

       A call to this method revokes a session token. Once a token is revoked it is no longer valid.

Sample Request
       curl https://www.google.com/accounts/AuthSubRevokeToken --header ‘Authorization: AuthSub token="token"’ 

(1.4)           AuthSubTokenInfo.

       A call to this method verifies whether a specified session token is valid and returns data associated with the token. This operation applies to both one-time-use and session keys.

Sample request

         curl https://www.google.com/accounts/AuthSubTokenInfo --header 'Authorization: AuthSub token="CO7xhO-6GhDT5f2tAwd"'

Sample response

Target=http://www.yourwebapp.com

Scope=http://www.google.com/calendar/feeds/

Secure=true

(2)    ClientLogin

       http://code.google.com/apis/accounts/docs/AuthForInstalledApps.html

       Before using ClientLogin, you must have an existing Google account. The POST request should be structured as a form post with the default encoding application/x-www-form-urlencoded. Parameters should be included in the body of the post.

Action URL parameter: https://www.google.com/accounts/ClientLogin

Sample request format:

POST /accounts/ClientLogin HTTP/1.0

Content-type: application/x-www-form-urlencoded

 

accountType=HOSTED_OR_GOOGLE&Email=jondoe@gmail.com&Passwd=north23AZ&service=cl&source=Gulp-CalGulp-1.05

CURL command
       curl    --location https://www.google.com/youtube/accounts/ClientLogin 
                   --data 'Email=username&Passwd=password&service=youtube&source=Test'
                   --header 'Content-Type:application/x-www-form-urlencoded' -i

Sample Response:

SID=DQAAAGgA...7Zg8CTN

LSID=DQAAAGsA...lk8BBbG

Auth=DQAAAGgA...dk3fA5N

 (3)    Difference of two mechanisms

       To use ClientLogin, a third-party application must own Youtube’s username and password of every user. Then the third-party application can do anything the end user can do. This mechanism is convenient for third-party apps to use because it does not require much intervening from users. However, it requires trust of users who may be afraid that their person information is not stored appropriately.

       To use AuthSub, a third-party does not need to own Youtube’s username and password of user. Instead, the user will be redirected to a web site from google to authorize third-party app’s access to user’s account. At first, third-party app is given a one-time access key which can be used to exchange a session key that can be used longer. This method is more secure for users at the cost of manually authorizing access requests.

Some development detail

(1)    Some videos are not allowed to be embedded in third-party applications.

       In the feed of these videos, mediagroup::content element does not exist!!! I manually construct URL by concatenating http://gdata.youtube.com/feeds/api/videos/ and video id and try to embed it into my app. As I expect, it does not work. However, I figured out a method which can work around this problem. I concatenate http://www.youtube.com/swf/l.swf?video_id= and video id and it works.

(2)    Some videos are not allowed to be commented by some users. These videos should be handled carefully.

       http://gdata.youtube.com/feeds/api/videos/gmNbl5ZTBZE

(3)    Duplicate elimination

       If you upload the same video clip more then once using different metadata, Youtube can figure it out and all uploads but the first one will be rejected. I am not sure how they compare two different video clips, maybe using MD5 to calculate a signature. The duplicate uploads are still displayed in your account. And all metadata is preserved except the video clip itself. The only difference between original upload and following duplicate uploads is that you can not play duplicate video clips.

As shown in above picture, metadata (e.g. description, tags, views, published time…) are still there. But the video clip itself is rejected

(4)    OpenSearch support

       If you search videos on Youtube, in returned feed openSearch extension is supported:

<openSearch:totalResults>274842</openSearch:totalResults>

<openSearch:startIndex>1</openSearch:startIndex>

<openSearch:itemsPerPage>25</openSearch:itemsPerPage> 

(5)    Navigation of many returned results.

       In returned feeds, if not all results are returned, there should be a link element in the feed. It is like this:

<link rel="next" type="application/atom+xml" href="http://... "/>

<link rel="previous" type="application/atom+xml" href="http://... "/>

 

Mashup Architecture

Architecture:

Adapter:

TODO:
Integrate authentication architecture.