State server event arrivial order

State service has been implemented before. Detailed information is here: http://zhenhua-guo.blogspot.com/2007/12/state-service-implementation-zhenhua.html. However, there are several unsolved problems. Recently, I fixed one of them.
A typical procedure is:
(1) End user sends a workflow to agent server.
(2) Agent server transforms the original workflow into an internal format.
(3) Agent server sends the workflow to status server.
(4) Agent server submits the workflow to executor.
(5) Executor sends event messages to status server to report the progress of execution of the workflow.
The following senario is possible:
    When status server receives event messages from executor, it has not received the workflow from agent server. In other words, although workflow is sent by agent server before event messages are sent by executor, the arrivial order at the status server is not guaranteed.
In this case, the event messages will be lost and we have no chance to restore it.
[Solution]
Message buffer.
If the senario above happens, the received messages are buffered/saved temporarily at status server. Then, when a workflow is received by status server, all buffered/saved messages corresponding to that workflow will be applied to it.
[For example]
(1) Executor sends following message to status server.
    user "tony"                                    //user who owns the workflow
    workflow "weatherforecast"           //workflow id which uniquely identifies the workflow
    subjob: "solve equations"                //sub job
    status: started                                //status
(2) Status server receives that message.
    However, by checking its repository, status server finds that the corresponding workflow has not been received yet.
    So, that message is buffered.
(3) Status server receives the following workflow message from agent server.
    user "tony"
    workflow "weatherforecast"
    workflow content "<project>....</project>"
(4) Apply the message to the workflow and status of the workflow is changed to indicate that some element starts to execute.

BugFix and Improvement on Karajan workflow Composition

Last week, I implemented a basic visual Karajan workflow composition interface which eases wokflow composition. This is the related post: http://zhenhua-guo.blogspot.com/2007/12/karajan-workflow-composition.html.
This week, I fixed several bugs and made some improvements on top of it.
(1) The configuration of Karajan workflow is stored in a javascript object like this:

{
    elements: {},
    namespaces: {
    	kernel:{
    		elements:{
    			import:{
				properties: [],
				widgetProps: {}
    			}
    		},
    		namespaces:{
    		}
    	},
        sys: {
            namespaces: { },
            elements: {
                execute: {
                    properties: ["executable", "host", "stdout", "provider", "redirect"],
                    widgetProps: {height:"40px, width:"40px"}
                },
                echo: {
                    properties: ["message", "nl"],
                    widgetProps: {}
                }
            }
        }
    }
}

Note the element of which color is blue. Name of that Karajan element is "import" which is also a keyword of Javascript. As a result, the object above is not legal javascript object!!! So more work is needed here. To work around this problem and make the architecture more scalable, I add one more layer between the configuration object above and the code that uses it.
I construct an element tree called KarajanElementTree of which nodes are KarajanNSNode or KarajanElementNode. KarajanNSNode corresponds to a namespace in Karajan and KarajanElementNode corresponds to an usable element in Karajan. In other words, given a workflow configuration, I build a tree based on it. The tree has a set of fixed interface which can be used by programmers to access the information of various workflow elements. When underlying workflow configuration is modified, I just need to change implementation of the tree with interface staying the same. In other words, workflow configuration and use of the workflow are completely separated so that change of one part does not require change of the other part.
Concretely speaking:
(*) underlying workflow configuration
For those elements of which names are keywords of Javascript, I append a '_' to the element name and add a new property called "truename" to record the real name. Some people may argue that the real name can be obtained by removing the '_' character from the end of the name. Yes, that is right. However, considering the future expansion, my choice is better. For example, maybe one day "import_" becomes a keyword of javascript as well or '_' charater can not be contained in name of a property. Then we need to modify the code which handles the extraction of the real name from the element name.

{
    elements: {},
    namespaces: {
    	kernel:{
    		elements:{
    			import_:{
				truename: "import",
				properties: [],
				widgetProps: {}
    			}
    		},
    		namespaces:{
    		}
    	},
        sys: {
            namespaces: { },
            elements: {
                execute: {
                    properties: ["executable", "host", "stdout", "provider", "redirect"],
                    widgetProps: {height:"40px, width:"40px"}
                },
                echo: {
                    properties: ["message", "nl"],
                    widgetProps: {}
                }
            }
        }
    }
}

(**) Intermediate Element Tree
    KarajanElementNode:
    [ properties ]:
        name: name of the element, this is the name used to retrieve the object corresponding to that name in Javascript;
        truename: real Karajan name of the element.
        properties: properties of the Karajan elements;
        widgetProps: properties of the corresponding widget
    KarajanNSNode:
    [ properties ]:
        name: same as above;
        truename: same as above;
        elements: contains all elements in this namepace;
        namespaces: contains all sub namespaces in this namespace.
    KarajanElementTree:
    [ properties ]:
        root: root of the tree. Typically, type of the root is KarajanNSNode.
(***) Upper layer that uses the Karajan workflow

	var workflowele = getEle( KarajanElementTree, elementname );
	var realname = workflowele.truename;

Now, to get the information of a Karajan element, we don't need to know the underlying mechanism. For example, name of the element can be gotten by accessing property "truename".

(2) Empty namespace and element list elimination
In previous implementation, a new accordion panel is created for every namespace and element list no matter whether they are empty. As a result, the widget toolbox looks jumbly.
Now, I improve it. When a namespace is empty or an element list is empty, don't create an accordion panel for it at all.

(3)Add Pop-Up Window to display element description
In Karajan, there are hundreds of elements. Besides that, users can define their own customized elements. It is hard for a user to remember usage of so many elements. Sometimes, a user has used a certain element, but he/she cannot remember the usage of the element. At this time, a simple suggestive message is enough.
So, I add a new property "description" to every element which describes the usage and functionality of that element. When user moves cursor over a widget, the corresponding description is displayed in a pop-up message window. When user moves cursor out, the window disappears.
Screenshot:
 

(4) Better Support in Element insertion
In previous implementation, a Karajan element can just be inserted into current caret position. It is possible that user wants selected text to be enclosed by a certain element.
For example, we have workflow like this:

<project>
	<transfer srchost="aaa.com" srcfile="file1" desthost="localhost" destfile="file1"/>
	<transfer srchost="bbb.com" srcfile="file2" desthost="localhost" destfile="file1"/>
	<transfer srchost="ccc.com" srcfile="file3" desthost="localhost" destfile="file1"/>
</project>

The three transfer jobs are independent of each other. We would like to let them executed in parallel. Karajan element "parallel" can be used now. If we just support insertion of elements into current caret position, the user needs to first insert element "parallel" somewhere, and then copies "</parallel>" and paste it after the last transfer job. What is better is that user can select the jobs that want to be executed in parallel and element "parallel" will enclose the selected jobs during insertion.
Now, I have implemented this functionality. However, it sometimes does not work well in IE....

(5) Add more Karajan elements into the configuration object
    Karajan workflow contains so many built-in elements so that it is not practical to add all the elements into the javascript configuration object at a time. I decide to gradually add them to the object. Now, I have added many, but still many left...

Karajan workflow composition

Recently, I am focusing on providing visual widgets to ease composition of Karajan workflow. It is not practical to build the application from scratch. I have surveyed several prevailing javascript/Ajax frameworks including qooxdoo, prototype, jQuery, Ext, mootools.
Survey of js frameworks:
(1) jQuery(http://jquery.com/)
   As its name implies, its emphasis is query. At first sight, the framework is beautiful and concise. It supports CSS dom selector and XPath syntax.  Besides, it provides further convenient selection syntax. Some examples follows:
   $("a")
   $("a[@name]")
   $("a[@href=content/resources]")
   $("ul > li")
   $("ul li")
   $("ul .list > a")
   $("#output")
More examples:
   $("li:eq(0)")   //gets the first item
   $("li:lt(3)")    //get the first three items
   $("li:not(.groove)")    //get li elements without class groove
Beautifully, right? We can do lots of work by code of just several lines!!
Chaining: Most functions return a jQuery object so that you can directly invoke more functions.

$('form#login')
    // hide all the labels inside the form with the 'optional' class
    .find('label.optional').hide().end()

    // add a red border to any password fields in the form
    .find('input:password').css('border', '1px solid red').end()

    // add a submit handler to the form
    .submit(function(){
        return confirm('Are you sure you want to submit?');
    });

No matter whether you like this kind of code or not, it is functionality provided by jQuery. I prefer to use multiple lines of code and self-documenting variable names which look clearer.
Plug-ins: jQuery is blooming considering number of its plug-ins. Plug-ins increase sharply recently and many developers contribute to it.
However, jQuery does not excel at UI. In other words, if you want to build fancy user interface, jQuery is not the first choice.

(2)Ext(http://extjs.com/)
   Originally, Ext was based on YUI and it was developed as extension for YUI. Then, Ext broke away from YUI and was developed as an independent project.
   Its emphasis is abundance of UI widgets. It provides many fancy and convenient UI widgets which can be used easily to build our own GUI. The configuration of UI widget looks like this:
   var panel = new Ext.Panel({
      title: "This is title",
      width: 400,
      height: 300,
      border: true,
      layout: "accordion",
      items: [ .... ]
   });
It is more convenient than invocation of bunch of functions to set values of properties(e.g. panel.setwidth(400); panel.setheight(400);...).
Not long ago, combination of jQuery and Ext was announed which is good news to web application developers. However, process of the combination is kind of slow and support of jQuery in Ext 2 is limited and buggy.

(3)Qooxdoo
   This framework is sort of comprehensive which includes almost all common functionalities. Documentation is not bad. It is growing rapidly and seems promising.
   However, current version of this framework is 0.7, which means it is still in beta phase and not appropriate for production use. Apart from that, it aims to control the whole web page by Qooxdoo. So it is difficult for end users to directly access/modify underlying dom element. This drawback is annoying because inevitably  users sometimes want to manipulate underlying elements directly.

(4)Dojo(http://dojotoolkit.org/)
    This framework is so comprehensive and complex. It is the most powerful framework I have ever seen. It provides lots of functionalities: UI widgets, event system, offline support, presentation... As a result, the framework is sort of bloated and cumbersome. Bugs are not rare... Besides, documentation is done badly which makes development more difficult.
    Maybe, in the future, Dojo will become outstanding in term of functionality, performance and documentation. But for now, it is far from that.

(5)Prototype(http://www.prototypejs.org/)
    This framework adds basic OO features to javascript, e.g. inheritance. It is actually a language(javascript) extension library. Moreover, it extends some built-in objects (String, Array...) of javascript to offer more convenient functionalities. Script aculous(http://script.aculo.us/) is built on top of prototype and provides UI widgets.
    I read some articles about prototype and it seems that the support for OO features has problems in some situations.

Karajan Workflow Composition
Anyway, finally I chose Ext as my javascript framework.
Some screenshots about the workflow composition panel:
 
The panel is organized according to namespaces. So if user knows the namespace of an element, it is effortless to find the corresponding widget in the toolbox. And all main panels structured into accordion layout. If user clicks the title bar of a panel, that panel is expanded and all other panels are collapsed.

Karajan workflow element edit panel:
 
After values of various properties are typed, the xml document corresponding to the element will be automatically inserted into output panel. For sys.execute element, the xml snippet looks like this
    <sys:execute executable="..." host ="..." stdout="..." provider="..." redirect="..."/>
Currently, all values are enclosed by double quotation marks. The reason is that Karajan workflow is XML document in nature. For xml document, value of every attribute/property MUST be enclosed by quotation marks.
For other workflow languages,this is not always correct because types of some properties are integer/boolean and these values should not be enclosed by quotation marks.
After user clicks "Save" button, the generated xml snippet is inserted into output panel. The xml snippet is not simply appended to the output panel. Instead, it is inserted into current cursor position.

Scalability and Maintainability
    During design, I always keep a principle in my mind: built-in elements of Karajan are abundant and users can add their own customized elements. As a result, the addition of elements to widget window/toolbox should be easy and scalable.
The configuration is a javascript object:

{
    elements: {},
    namespaces: {
        sys: {
            namespaces: {
                file: {
                    elements: {
                        read: {
                            properties: ["name"],
                            widgetProps: { }
                        },
                        write: {
                            properties: ["name", "append"],
                            widgetProps: {}
                        }
                    },
                    namespace: {}
                }
            },
            elements: {
                execute: {
                    properties: ["executable", "host", "stdout", "provider", "redirect"],
                    widgetProps: {height:"40px, width:"40px"}
                },
                echo: {
                    properties: ["message", "nl"],
                    widgetProps: {}
                },
                parallel: {
                    properties: [],
                    widgetProps: {}
                },
                sequential: {
                    properties: [],
                    widgetProps: {}
                }
            }
        }
    }
}

In Karajan, namespace is supported. For every namespace, there are two properties: elements and namespaces. Property elements contains information about those elements directly in the namespace. Property namespaces contains information about sub namespaces.
In above example, namespace sys contains elements execute, echo, parallel and sequential and it contains sub namespaces file. Then namespace file contains elements read and write and it contains no sub namespaces.
For every element, it contains two properties: properties and widgetProps. Property properties contains list of parameters about the elements. Property widgetProps contains configuration information about how to display the corresponding widget in the toolbox window.
In above example, element execute has properties executable, host, stdout, provider and redirect.
To add more elements, I just need to modify the configuration object shown above. Obviously, it is convenient to modify it.

Improvements we can do in the future
(1) Now I list all properties of an element in the edit panel. For some elements, number of properties is more then ten. But only some properties are used frequently and others are seldom used. In the future, we can first hide the optional properties and only display the necessary properties. If user wants to use all properties, we show those optional properties as well.
(2) Search functionality. Number of elements may be enormous and it is painful to browse all namespaces to find the desired element.
However, this improvement is not necessary. Official website of CogKit provides reference manual for Karjan workflow and detailed information about all elements of Karajan is included. So user can first consult the reference manual for detailed information on the desired element. Then the user will get the fullname of the element which includes name of namespace in which the element is located. According to namespace, it is very easy to find the corresponding widget in the toolbox.

Client Side Enhancement

Obviously, it is not a favorable job to directly compose XML workflow file. Users may spend more time on XML formatting (start tags, end tags ...) than on business logic. So it is good news to provide auxiliary tools to ease composition of workflows. Visual widgets which support drag and drop can be used. In web2.0, there are several mainstream technologies including Ajax, Flash, Flex, Silverlight and OpenLaszlo, etc. I think Ajax is preferred because it can be run on almost all platforms. Support for Ajax is almost built in every prevailing browsers.
If we decide to use Ajax, it is time consuming to build our application from the scratch, considering there are so many mature Ajax frameworks. I can list some frameworks here: dojo, qooxdoo, jquery, prototype, mootools, GWT, YUI, Ext ... I have surveyed these frameworks.
GWT is developed by Google and YUI is developed by Yahoo. GWT provides conversion from Java to Javascript so that programmers can use Java to build web application. However, the customization of the webpage (layout, style ...) is not convenient.
Frameworks Prototype, Dojo, Qooxdoo, Jquery and mootools are all open-source and they have their own good and bad. However, the difference is kind of subtle. I used Qooxdoo in previous project and I don't recommend us to use it. All other frameworks seem good and further investigation is necessary to pick up the most appropriate one for us.

State Service Implementation

Zhenhua Guo

Previous posts gave initial design of the system. They mainly focus on high level abstraction. Now, I have nearly implemented the state server and some minor changes of the design have been made. It is time to elaborate the implementation of every component in the system.

Architecture

Figure 1

【Client】

Clients initiate the job submission. Currently, the Karajan workflow language is supported and it is the only way to represent a workflow. In fact, every user needs an account to submit jobs. However, currently the user management system at the agent server has not been decided. So I use the built-in user management system and authentication mechanism in Apache Tomcat.
Client invokes functions at the agent server by XML-RPC (Jsolait javascript library is used.). At client side, we provide functionality of converting between JSON and XML.
After authentication, what the client sends to the agent is user name and workflow for job submission.
Then the client periodically sends requests to state server to check the state of the workflow. The data is represented in JSON.

【Agent】
Agent uses Apache XML-RPC toolkit (http://ws.apache.org/xmlrpc/) to build server implementation.
After the user is authenticated, agent should receive job submission request which consists of user name and workflow. Then agent transforms original workflow into an internal format. Echo messages are added to report the progress of execution of the workflow.

For example, if original workflow is:

<project>
<include file="cogkit.xml"/>
<execute executable="/bin/rm" arguments="-f thedate" host="gf1.ucs.indiana.edu" provider="GT2" redirect="false"/>
<execute executable="/bin/date" stdout="thedate" host="gf1.ucs.indiana.edu" provider="GT2" redirect="false"/>
<transfer srchost="gf1.ucs.indiana.edu" srcfile="thedate" desthost="localhost" provider="gridftp"/>
</project>

then the internal workflow is:

<project>
<include file="cogkit.xml"/>
<echo message="/2|job:execute(/bin/rm) started|1"/>
<execute executable="/bin/rm" arguments="-f thedate" host="gf1.ucs.indiana.edu" provider="GT2" redirect="false"/>
<echo message="/2|job:execute(/bin/rm) completed|2"/>
<echo message="/3|job:execute(/bin/date) started|1"/>
<execute executable="/bin/date" stdout="thedate" host="gf1.ucs.indiana.edu" provider="GT2" redirect="false"/>
<echo message="/3|job:execute(/bin/date) completed|2"/>
<echo message="/4|job:transfer started|1"/>
<transfer srchost="gf1.ucs.indiana.edu" srcfile="thedate" desthost="localhost" provider="gridftp"/>
<echo message="/4|job:transfer completed|2"/>
</project>

    The statements written in blue are state reporting messages which are added by agent. The format of message is:
       /3/4/2|job:xxx|1
(“/3/4/2” is path, “job:xxx” is job state description and “1” is status code).
    The first part is path which contains all characters appear before character ‘|’. This part indicates the position of the element reported in the whole workflow. The path is /3/4/2 in the sample which means the element reported currently is the second child of the fourth child of the third child of the root element. The root element is <project>.
    The second part is job state description which contains all characters between the two ‘|’ characters. This reports the state of the element.
    The last part is job state code which appears after the last ‘|’ character. This actually is an integer which marks the state. Currently, 1 means the subtask is started and 2 means the subtask is completed.

A unique id is generated for every workflow of a user. I call it wfid(workflow id). In other words, every workflow is uniquely identified by combination of user name and workflow id.

After the transformation is done, agent sends original workflow to state server and sends transformed workflow to executor. Besides the workflows themselves, agent also sends corresponding wfid to state server and executor.

Note executor and state service are both wrapped as web services.

【Executor】

This part is wrapped as web service. Executor receives transformed workflow from agent and submits the workflow to underlying grid infrastructure by using Java CoGKit. Currently, I invoke command line tool provided by CoGKit to submit workflows to grid infrastructure. In the future, this may be changed. API of Java CoG may be a better choice.

When the workflow starts to be executed, executor sends message to state server to indicate the starting of execution of workflow.

During the execution, state messages are generated to report progress of execution. Every time a state message is generated, executor sends state message to state server.

When the workflow is completed, executor sends message to state server to indicate the completion of execution of workflow.

【State server】

This part is wrapped as web service.

During job submission, state server receives the workflow from agent. Then state server builds an element tree based on the workflow. State data is stored in every node of the workflow. Helper function is provided to serialize the element tree into JSON format or XML format (XML format serialization has not been implemented).

During job execution, state server receives state message from executor. Format of state message is defined above. State server updates state of corresponding node in the element tree. There are two kinds of state messages here. One is state messages related to state of whole workflow. The other is state messages related to state of a specific element in the workflow.

Issues:

1. After a user sends a workflow to agent, what should be returned?

Option 1: unique id of the submitted workflow

The end user can check state of this submitted workflow by using the unique id. However, it is not easy to get the output of the workflow execution. To do this, executor or agent needs to store the result temporarily. Then when end user checks the result, agent can return it.

Option 2: output of the workflow execution

Because end user does not know id of the workflow which was submitted just now, the end user can only get state of all submitted workflows so far. This is current choice.

2. Reliable messaging

Executor sends state messages to state server. Semantically, order of the state messages should be guaranteed which reflects the true order of execution of subtasks. In current system, this is not guaranteed. I am considering using WS Reliable Messaging to do this.

3. Guarantee of the order of messages.

In description of state server, we know that state server should handle workflow from agent before it handle state messages from executor. In figure 1, it means step should occur before step . However, this can not be guaranteed now. Because of unpredictable network delay and process/thread scheduling, I have no way to satisfy that requirement without modifying code of state server.

If state server handles state messages from executor before it handles workflow from agent, obviously it will not find the corresponding workflow in its database.

One solution:

State server preserves the state messages from executor. How long should the state messages be preserved? Fixed time? Indefinitely time until state server receives workflow from agent?