DAG construction

After communication with Mike, I got the dag.k file. Then I implemented the construction of DAG workflow based on all jobs and their relationship in a workflow. In other words, I put together all this information in a large Karajan workflow by using DAG. Note: this work is done at client side, not at server side. After construction of the huge workflow, it can be submitted to server just as a small job. Then id of the new submitted workflow will be returned. Based on this workflow id, a user can query state of the workflow. In addition, the user can access output files of the workflow by using common HTTP GET request.

More details about DAG construction:
Assume that we have four jobs in a workflow: job1, job2, job3, job4.
And their dependencies are:
job1 -> job2 ( this means job2 depends on job1 )
job1 -> job3
job3 -> job4
job3 -> job5
job1 -> job5
These dependencies are represented in following graph:
 
Then constructed DAG workflow looks like this:
<project>
<include file="cogkit.k"/>
<include file="dag.k"/>
<discard>
    <dag>
        <node>
            <string>job1</string>            //Here is name of the job.
            <element>
                <quotedlist/>
               content of job1
            </element>
            <edges>
                <string>job2</string>         //Here, it describes that job1 is prerequisite of job2, job3 and job5
                <string>job3</string>
                <string>job5</string>
            </edges>
        </node>
        <node>
            <string>job2</string>
            <element>
                <quotedlist/>
                content of job2
            </element>
        </node>
        <node>
            <string>job3</string>
            <element>
                <quotedlist/>
                content of job3
            </element>
            <edges>
                <string>job4</string>
                <string>job5</string>
            </edges>
        </node>
        <node>
            <string>job4</string>
            <element>
                <quotedlist/>
                content of job4
            </element>
        </node>
        <node>
            <string>job4</string>
            <element>
                <quotedlist/>
                content of job4
            </element>
        </node>
    </dag>
</discard>
</project>

Karajan Workflow Formats: .k and .xml

In CoGKit, there are two supported formats in Karajan workflow -- .k and .xml.
Personally, I like xml because of its prevalence and openness. There are many handy tools which can process xml documents in various ways. Unfortunately, xml support in Karajan workflow is not comprehensive. In recent programming, I need to use element dag which is used to support Directed Acyclic Graph.
(1) At first, Karajan workflow supports DAG. But it does not give users a dag.xml file to import. Instead, just dag.k is provided. So I need to find a way to convert dag.k to dag.xml. Thank Mike and Gregor to help me get out of the struggle. Following method can be used to do the task:

cog-workflow -intermediate dag.k

Although following error pops up, I still can get the file I need (dag.xml).

Execution failed:
Variable not found: #channel#defs
        kernel:export @ dag.k, line: 44

Note: actually, what is generated is dag.kml not dag.xml. Currently, I assume they are the same because I have no better choice.

(2) In document I can find, sample workflows about DAG are written in .k format.
This is the unique resource I find helpful: http://wiki.cogkit.org/index.php/Java_CoG_Kit_Workflow_Guide#Direct_Acyclic_Graphs.
Once again, I used command cog-workflow to do this task. But the generated .kml(xml) file is lengthy and not appropriate for human readers. So, I decided to convert it manually by myself. I found this useful link(http://wiki.cogkit.org/index.php/Java_CoG_Kit_Karajan_Workflow_Reference_Manual_4.1.5) which describes both formats in detail.

How to access results?

Now, it is time to consider how to make users easily and conveniently access the results of their workflows. There are several questions here:
(1) How to track output files in Karajan workflows?

The first option is to analyze content of the Karajan workflow to figure out output files. For example, for element execute, attribute stdout indicates the name of output file.
<execute executable="/bin/date" stdout="thedate" host="gf1.ucs.indiana.edu" provider="GT2" redirect="false"/>
However, if we use this method to track all output files, it is difficult and time-consuming because it is possible that many elements generate output files. As a result, we must capture possible output files from all these elements.
Another option I can think of is kind of tricky. The newly submitted workflow is executed in a newly created directory. After execution, the files (except workflow file) in the directory are output files. This is the method I am using in my implementation.

(2) How to organize output files?

For the same workflow, we can categorize it based on different criteria. For example, we can categorize a workflow based on the date on which it is submitted, or the date on which it is completed... I would like to make use of workflow id and user id to categorize the workflows. All workflows submitted by a user belong to the same group which can be accessed by this user. Within these workflows, workflow id is used by the user to access a specified workflow. The id of every workflow belonging to a user is unique.
So, the directory layout may look like this:
users/user1/workflow_122/output_file1

users/user1/workflow_122/output_file2
users/user2/workflow_1/output_file1
...

(3) How can users access output files?

After talking with Marlon, I would like to provide RESTful interface by which users can retrieve output files. In my implementation, URLs to access output files look like this:
http://domain:port/resources/user_name/workflow_id/ This retrieves list of all output files for the corresponding workflow.
http://domain:port/resources/user_name/workflow_id/output_file This will retrieve the specified output file directly.

RESTful web services in Java

Recently, I read some articles about RESTful web services and I am looking for some java libraries which have great support for REST. REST is supported by Axis2. However, the support is very limited. First, document(http://ws.apache.org/axis2/1_3/rest-ws.html) of REST support in the official web site is horrible. The content is very very brief so that I have more questions and confusions than what have been addressed by that document. Support for REST in Axis2 relies on a new feature in WSDL2 which enables HTTP binding. Here is a good article about it:http://www.ibm.com/developerworks/webservices/library/ws-rest1/. However, HTTP binding doesn't enable programmers to implement a full REST style system. I did not dig into WSDL2 to get more knowledge about its HTTP binding. Here(http://wso2.org/blog/footballsoccerpainting/949) is an article from someone else who complains Axis2.
Then, I found that JCP published a Java API specification about RESTful web services. It is JSR-311(http://jcp.org/en/jsr/detail?id=311). That looks pretty good because now we have a standard about how to use REST in Java. Jersey(https://jersey.dev.java.net/) is reference implementation of the specification. Besides Jersey, Restlet(http://www.restlet.org/) is another implementation which provides more features. Note that the specification itself is still in beta phase.

I decided to try Jersey. First download and unpack it.
Sample code:

@Path("/")
public class RESTTest{
    @HttpContext UriInfo uriInfo;	
    
    @GET
    @ProduceMime("text/plain")
    public String getUserAll(){
    	return "You want to retrieve information about all users.";
    }
    
    @Path("{user}")
    public UserResource getUserInfoAsText(@UriParam("user") String userid){
    	return new UserResource(userid);
    }
}

Most readers have noticed that Java annotation is used frequently. I think this method is handy and convenient.

Workflow Queue Support

In previous implementation, I separated jobs from workflows. A job is a small task which currently is written in Karajan workflow language. Here, it is kind of confusing. Karajan workflow language is used to represent a job. A workflow consists of jobs number of which is variable. A workflow queue consists of some workflows. Workflows in a workflow queue can be executed in arbitrary order, which means that these workflows are totally independent. However, jobs in a workflow are usually related so that they should be executed in a certain order. Essentially workflow queue and workflow do not have fundamental difference because we can convert between them. So, difference is in logical level.
Now, workflow queue is supported.

Workflow management panel
 

When a new workflow is added, a new tab is created.
When an existing workflow is removed, the associated tab is removed from the user interface.
 

When a user creates and adds a new job, it can add this job to any existing workflow. Because we have created a workflow called "new_workflow1", this workflow appears in the drop box.

When a user switches to the workflow panel, all jobs belonging to the workflow will be displayed in a canvas. What's more, relationship between jobs is displayed as well.

When a user wants to see details of all jobs in a workflow, he/she can click button "click to see all jobs" and then a pop-up window is displayed to show detailed information.

JS API

Google Doc address: http://docs.google.com/Doc?id=dmxthpg_102fvp7vvhk

(1) Workflow:

Description:

This class represents a workflow. A workflow can contain more than one job. Note: a job in workflow can be another workflow.

Functions:

Workflow.addJob( jobname, job )

A new job is appended to this workflow.

Workflow.deleteJob( jobname )

The job with the name specified by parameter jobname is deleted. As a result, all related dependency is deleted as well.

Workflow.listJobs( )

List all jobs in a workflow.

Workflow.searchByName( jobname )

search job in terms of name.

Workflow.addDependency( jobparent, jobchild )

Add dependency which says that job jobparent should be executed before job jobchild.

Workflow.removeDependency( jobparent, jobchild )

Remove dependency between job jobparent and job jobchild.

Workflow.toggleDependency( jobparent, jobchild )

Toggle dependency. If there is a dependency, remove it. If there is not, create a new one.

(2) WFQueue

Description:

This class represents a workflow queue in which all workflows have no dependency on each other. In other words, workflows in WFQueue can be executed in arbitrary order.

Functions:

WFQueue.addWorkflow( name, workflow )

Append the workflow specified by parameter to a workflow queue.

WFQueue.removeWorkflowByIndex( index )

Remove a workflow according to parameter index.

WFQueue.removeWorkflowByName( name )

Remove a workflow with name specified by parameter name.

WFQueue.clearAll( )

Remove all workflows in the queue.

WFQueue.searchByName( name )

Return the index of the workflow with the name specified by parameter. If no

corresponding workflow exists, return -1.

(3) Executor

Description:

This class mainly submits workflow to remote server.

Functions:

Executor.Executor( serverURL )

Constructor of class Executor. The parameter specifies URL of remote server where following submissions will be sent.

Executor.submitWF( workflow );

Submit workflow to remote server.

Executor.submitWFQueue( workflow );
Submit workflow queue to remote server.

Executor.transfer( from, to )

Shortcut for transferring a file. Note: this task can be done by putting a corresponding job into a workflow, then submitting the workflow to remote server. However, job “transfer” is used so frequently that I decide to separate it.

(4) Queryer

Description:

This class is used to query state of workflows submitted by end users.

Functions:

Queryer.query( username );

Get state of all workflows submitted by a user.

Queryer.query( username, workflowid )

Get state of a specified workflow submitted by a user.

Queryer.query( username, workflowids )

Get state of specified workflows submitted by a user.

(5) Authenticator

Functions:

Authenticator.login( username, password )

Authenticator.logout( username, [ password ] )

Add Support for Job Dependency Edit

Lately, I have been working on support for client-side job queue management and job dependency management.
This two parts can be designed and implemented separately. But I think putting them together is better which is more user-friendly.
Before jobs are submitted, they are maintained at client side.
Currently, following functionalities are supported:
(1) add a job to job queue
(2) remove a job from job queue
(3) edit dependency between jobs.
To make the system easy to use, I provide visual widget interface.

Main interface:

Job addition:
After a use inputs the workflow description in the text area and workflow name in the text field, he/she can add the workflow to job queue by clicking button "Add to Queue". If a job with that name has already existed, a prompt window is popped up. Then user can choose to overwrite current job or modify the name.
By clicking button "Job Management" or tab "Job Management", user can be redirected to job management panel.
Note: name of every workflow must be specified. In other words, value of text filed "Workflow Name" can not be left blank. Moreover, different workflows/jobs can not have a same name. So name of every job must be different.

Job Management Panel:
As described in the picture, every rectangle represents a job and every line with arrow represents dependency between two jobs.
When a use adds job1, job2,...jobn, the default relation is that job2 depends on job1, job3 depends on job2 and jobn depends on jobn-1.
When you move curse over a rectangle for a few seconds, a pop-up window is displayed which contains the content of that job.
 

When you right click (click with right button) a rectangle, a context menu is displayed. This menu contains several items.

Currently, items are "from", "to", "delete" and "edit".
(1) If menu item "delete" is clicked, the corresponding job will be deleted permanently from job queue.
   When a job is deleted, all related dependency is deleted as well. There are two kinds of dependency: one is that some other jobs depend on this job; the other is that this job depend on some other jobs.
(2) If menu item "from" is clicked, the corresponding job is marked as the starting point of dependency. Assume it is called parentJob.
(3) If menu item "to" is clicked, the corresponding job is markded as the end point of dependency. Assume it is called childJob.
Then there are three possible steps:
  (3.1) If parentJob is null.
    In this case, it means the user has not selected a job by clicking menu item "from". Then nothing will happen.
  (3.2) If job parentJob is not prerequisite of job childJob.
    And then a line is drawn from the rectangle which represents job parentJob and the rectangle which represents job childJob. And job parentJob is considered as prerequisite of job childJob.
  (3.3) If job parentJob is already prerequisite of job childJob.
    In this case, there must be an existing line drawn from the rectangle which represents job parentJob and the rectangle which represents job childJob. Then this relation is deleted and the line is removed from display.
(4) If menu item "edit" is clicked, the system will redirect user to the workflow edit panel.

User can use drag-and-drop to move the rectangles to anywhere in the screen. Related lines (two situations: one is that some lines starts from this job; the other is some lines ends at this job) are moved as well. Note: You can not use drag-and-drop to move the lines.
Following picture is a sample job dependency graph I got:

Next step:
Currently, all operations above are carried out at client side and no interaction with server is involved.
Issue:
Next step is related to how to send the job queue to server side.
In Karajan, I don't think workflow-level composition is supported directly. However, Karajan provides two elements parallel and sequential which can control the execution sequence of subtasks in a workflow.
So one idea is that all jobs in a job queue are put into a single big workflow which uses elements parallel and sequential to represent original relationship.  One natural question is that whether elements parallel and sequential are enough to express any possible relationship among jobs.
My answer is no.
For the job dependency shown above, I can not think of a way to represent it  with Karajan elements parallel and sequential.
Solution 1:
So, if my conclusion is correct, we can implement a sub system which manages sequence of job submission to underlying grid infrastructure. In my opinion, it is better to be put at server side.
Solution 2:
Aother solution is that we can simplify this issue at cost of losing performance. We can get a job submission sequence by using topological sort. In other words, all jobs are submitted sequentially. Obviously, performance is not the best because some jobs actually can be executed in parallel.