Wednesday, September 10, 2008

Some notes about tomcat/hibernate/servlet

Tomcat

For tomcat applications, resources (configuration files...) should be put in directory WEB-INF/classes. The files cannot be symbolic links. They MUST be regular files in order for tomcat to handle them correctly. Of course, you can embed resources into your .jar files.
For Tomcat, the regular CLASSPATH environment variable is IGNORED. The resources (jars, configuration files...) must be put into your tomcat app directory. Read this post to see what is under the hood.
Resource/class search order for web apps: 
  • Bootstrap classes of your JVM
  • System class loader classses (described above)
  • /WEB-INF/classes of your web application
  • /WEB-INF/lib/*.jar of your web application
  • $CATALINA_HOME/lib
  • $CATALINA_HOME/lib/*.jar
The System class loader is the only place where CLASSPATH can be parsed and used. However, tomcat startup script(bin/catalina.sh) totally ignore environment variable CLASSPATH. You will figure it out if you read file bin/setclasspath.sh. In my case, first several lines are :
#  Set CLASSPATH and Java options
# 
#  $Id: setclasspath.sh 589060 2007-10-27 08:19:24Z jfclere $
# -----------------------------------------------------------------------------

# First clear out the user classpath
CLASSPATH= 
Now, you know the root of the problem right? If you want to include CLASSPATH, you can modify the bash file. I have not tried it. It should work.

Hibernate

For hibernate app, corresponding mapping tables must be created before you can run your app. Hibernate will NOT create those tables automatically for you. For the primary key property/column, If you use generator "native", you must set property of the primary key column correctly in order to let database generate primary key values correctly. For example, you can use AUTO_INCREMENT to make database automatically generate primary key values in MySQL. You can choose to use generator "assigned" to assign primary key values in your programs.
For database operations in Hibernate, you should always EXPLICITLY start and commit a transaction.

Servlet/JSP

Method Forward declared in RequestDispatcher should be understood correctly. It is an INTERNAL control transfer. It means this control transfer is TRANSPARENT to end users. One consequence is that the links in the target page, to which you forward control, MUST use absolute path instead of relative path.
For example, first a user sends request to http://domain/users/gerald.jsp. At server side, your servlet internally transfers control to /users/profiles/g/gerald.jsp. In the target page, if you have links with relative path, the paths are relative to the original request path (/users/ in this example) instead of path of the target page (/users/profiles/g/ in this example).
Your servlet should not have generated any response ( in output stream to request sender ) when redirection or forwarding is carried out.

Sunday, September 07, 2008

Hibernate installation (run on MySQL)

Recently, I tried to install Hibernate on gridfarm machine. The version I used was 3.3.0 SP1.
Configuration document can be found here. However, the document seems not to be updated.
(1) Configuration file hibernate.properties vs. hibernate.cfg.xml.
In the document, either one can be used. If both are used, configurations in hibernate.cfg.xml would override configurations in hibernate.properties. The document says using hibernate.properties is an easy way to configure Hibernate. So I chose it. However, after trial I found that in Hibernate 3.3.0 SP1 hibernate.cfg.xml. In other words, you MUST have a configuration file "hibernate.cfg.xml". Of course, you can use programmatical configuration which is not covered in this post.
(2) Logger
In documents, I saw "Hibernate logs various events using Apache commons-logging.". However, when I tried to run my Hibernate app, it complained that logger class can not be found. After inspection, I figured out that Hibernate 3.3.0 does NOT use Apache common-logging any more. Instead it uses slf4j. But in Hibernate 3.3.0 SP1 distribution only slf4j-api jar is included while an implementation is needed to run Hibernate(If you turn off logging, I don't know whether it will still complain). In other words, to use logging functionality, you must download an implementation of logging service supported by slf4j. I used slf4j-simple.
(3) JNDI bindings (for session factory and data source).
Because I just wanted to write a simple prototype without deploying in J2EE server, I chose not to use JNDI bindings for data source or session factory. After several hour trial, I could not get it to work!! I always got the error "org.hibernate.impl.SessionFactoryObjectFactory - Could not bind factory to JNDI".It seemed that Hibernate always was always trying to use JNDI to bind Session Factory. However, the official document says

"A JNDI bound Hibernate SessionFactory can simplify the lookup of the factory and the creation of new Sessions. Note that this is not related to a JNDI bound Datasource, both simply use the same registry!

If you wish to have the SessionFactory bound to a JNDI namespace, specify a name (eg. java:hibernate/SessionFactory) using the property hibernate.session_factory_name. If this property is omitted, the SessionFactory will not be bound to JNDI. (This is especially useful in environments with a read-only JNDI default implementation, e.g. Tomcat.)

When binding the SessionFactory to JNDI, Hibernate will use the values of hibernate.jndi.url, hibernate.jndi.class to instantiate an initial context. If they are not specified, the default InitialContext will be used.

Hibernate will automatically place the SessionFactory in JNDI after you call cfg.buildSessionFactory(). This means you will at least have this call in some startup code (or utility class) in your application, unless you use JMX deployment with the HibernateService (discussed later).

If you use a JNDI SessionFactory, an EJB or any other class may obtain the SessionFactory using a JNDI lookup.

We recommend that you bind the SessionFactory to JNDI in a managend environment and use a static singleton otherwise. To shield your application code from these details, we also recommend to hide the actual lookup code for a SessionFactory in a helper class, such as HibernateUtil.getSessionFactory(). Note that such a class is also a convenient way to startup Hibernate—see chapter 1. "

It turned out that a property called "hibernate.current_session_context_class" must be set.
My hibernate.cfg.xml looks like:
<!DOCTYPE hibernate-configuration PUBLIC
    "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
    "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">

<hibernate-configuration>
    <session-factory>
        <property name="show_sql">true</property>
        <property name="current_session_context_class">thread</property>
        <property name="connection.username">username</property>
        <property name="connection.password">password</property>
        <property name="connection.url">jdbc:mysql://localhost/hibernate</property>
        <property name="connection.driver_class">com.mysql.jdbc.Driver</property>
        <property name="dialect">org.hibernate.dialect.MySQLDialect</property>
        <mapping resource="User.hbm.xml"/> 
    </session-factory>
</hibernate-configuration>
Then, I tried a very very simple read-only JNDI service provider - simple-jndi. My simple-jndi configuration file (jndi.properties which should be put into root of your classpath) looks like:
#The first (required) parameter, org.osjava.sj.root, is the location 
#of your simple-jndi root, which is the location in which simple-jndi
#looks for values when code asks for them.

org.osjava.sj.root=/home/zhguo/share/simple-JNDI/config
java.naming.factory.initial=org.osjava.sj.SimpleContextFactory
Under directory pointed to by org.osjava.sj.root property, I have a jndi data file called hibernate_jndi.properties:
mysql.type=javax.sql.DataSource
mysql.driver=com.mysql.jdbc.Driver
mysql.url=jdbc:mysql://localhost/hibernate
mysql.user=username
mysql.password=password
So, my new version of hibernate.cfg.xml looks like:
<hibernate-configuration>
    <session-factory>
        <property name="show_sql">true</property>
        <property name="current_session_context_class">thread</property>
        <property name="connection.datasource">hibernate_jndi.mysql</property>
        <property name="dialect">org.hibernate.dialect.MySQLDialect</property>
        <mapping resource="User.hbm.xml"/> 
    </session-factory>
</hibernate-configuration>
Note: in the file, session factory name is not set! This is because we are using a read-only JNDI. If you use a read-write JNDI, you can set the session factory name in one of the following ways:
(*) In hibernate.properties file:
hibernate.session_factory_name hibernate/session_factory
(*) In hibernate.cfg.xml file:
<session-factory name="your name goes here">

Wednesday, August 27, 2008

Hadoop and HBase port usage

Hadoop port usage
On name node
50090: org.apache.hadoop.dfs.SecondaryNameNode
33220: ditto.(This port is not fixed and may be changed when hadoop is restarted)

50070: org.apache.hadoop.dfs.NameNode
9000:ditto
46684: ditto(This port is not fixed and may be changed when hadoop is restarted)

9001: org.apache.hadoop.mapred.JobTracker
50030: ditto
60502: ditto(This port is not fixed and may be changed when hadoop is restarted)

On data node
50075: org.apache.hadoop.dfs.DataNode
50010: ditto
45868: ditto(This port is not fixed and may be changed when hadoop is restarted)

50060: org.apache.hadoop.mapred.TaskTracker
55027: ditto(This port is not fixed and may be changed when hadoop is restarted)

HBase port usage
On master
60000: org.apache.hadoop.hbase.master.HMaster start
60010: ditto

On data node
60030: org.apache.hadoop.hbase.regionserver.HRegionServer
60020: ditto

Monday, August 25, 2008

Insert pubchem data into HBase

HBase shell
HBase provides a shell utility which lets users to execute simple commands. The shell can be started up using:
${HBASE_HOME}/bin/hbase shell
Then input command help to get help document which describes usage of various supported commands. These commands can be used to manipulate data stored in HBase. E.g. command list can be used to list all tables in hbase. Command get can be used to get row or cell contents from hbase. Command put can be used to store data into a cell.

Data insertion
Data source is ftp://ftp.ncbi.nlm.nih.gov/pubchem/. I modified python scripts and C source code given by Rajarshi.
Data retrieval and processing steps:

  1. Download all information about compounds from ftp://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF. I finally got 123 GB.
  2. Decompress those files
  3. Extract information from these .sdf files and write it to .txt files. Library openbabel is used to compile the C++ code.
  4. Combine those .txt files generated in step 3 into one big .dat file
  5. Write a ruby script to insert all data in the .dat file into HBase.
    Command is like this: ${HBASE_HOME}/bin/hbase org.jruby.Main rubyscript.rb

Why did I write Ruby script instead of Java program in step 5?
HBase is written in Java and so provides Java API. However, to compile Java programs is kind of cumbersome -- set lengthy CLASSPATH ... 
So I chose to write scripts which can be executed directly by HBase shell. I found useful information on this page. There is a section called "scripting" in that post. But the information there is far from complete. It does not tell readers how to write the scripts. At first, I wrote a script which included some shell commands, one command per line, and then fed it to hbase shell. Unfortunately, it didn't work. After enumerous trials, I found that Ruby scripts could be fed to shell. Ruby scripts cannot make use of existing shell commands directly. Ruby binding of original Java APIs must be used.

I have not learnt Ruby at all before. So I must teach myself to grasp basic knowledge about Ruby. Ruby is sort of different in terms of syntactic flexibility. It supports so many shorthands to improve productivity. Anyway, "Ruby is easy to learn, but hard to master". By the way, Ruby documents seem not to be abundant compared with Python, Perl...

How to write Ruby scripts for HBase?
This site http://wiki.apache.org/hadoop/Hbase/JRuby contains related information. But I could not run the sample script successfully because of errors in the script!!! Damn it! I wonder whether the author had tested the code before he released it. Some errors are so obvious.
After the ruby script is completed, it can be executed using:
${HBASE_HOME}/bin/hbase org.jruby.Main rubyscript.rb 
Java API:
http://hadoop.apache.org/hbase/docs/current/api/index.html

My ruby script:

#!/usr/bin/ruby -w

include Java
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.HColumnDescriptor
import org.apache.hadoop.hbase.HTableDescriptor
import org.apache.hadoop.hbase.client.HBaseAdmin
import org.apache.hadoop.hbase.client.HTable
import org.apache.hadoop.hbase.io.BatchUpdate
import org.apache.hadoop.io.Text

pubchem_compound_fields = [
    'cid',
    'iupac_openeye_name',
    'iupac_cas_name',
    'iupac_name',
    'iupac_systematic_name',
    'iupac_traditional_name',
    'nist_inchi',
    'cactvs_xlogp',
    'cactvs_exact_mass',
    'openeye_mw',
    'openeye_can_smiles',
    'openeye_iso_smiles',
    'cactvs_tpsa',
    'total_charge',
    'heavy_atom_count']

compound_table_name = 'compound'

numfields = pubchem_compound_fields.length

path = "/home/zhguo/BigTable/BigTable-Pubchem/data/"
filename = "#{path}compound.dat"
file = File.new(filename, 'r')
counter = 0

conf = HBaseConfiguration.new
tablename = compound_table_name
tablename_text = Text.new(tablename)
desc = HTableDescriptor.new(tablename)
coltextarr = Array.new
pubchem_compound_fields.each_with_index do |v, i|
    if (i == 0) then next; end
    desc.addFamily(HColumnDescriptor.new("#{v}:"))
    coltextarr << Text.new("#{v}:")
end

admin = HBaseAdmin.new(conf)
if !admin.tableExists(tablename_text) then
    admin.createTable(desc)
=begin
    puts "deleting table #{tablename_text}"
    admin.disableTable(tablename_text)
    admin.deleteTable(tablename_text)
    puts "deleted table #{tablename_text} successfully"
=end
end

#admin.createTable(desc)
table = HTable.new(conf, tablename_text)

startind = 1641500 #from which line should we start.This
                   #is useful when you don't want to start
                   #from the beginning of the data file.

nlines = `cat #{filename} | wc -l`

logfilename = 'updatedb.log'
logfile = File.new(logfilename, "a")
while (line = file.gets) #&& (counter < 20)
    counter += 1
    if (counter < startind) then
        next
    end
    msg = "processing line #{counter}/#{nlines}"
    logfile.puts msg
    if counter%100 == 0 then
        print  msg
        STDOUT.flush
        logfile.flush
    end

    arr = line.split("\t")
    len = arr.length
        if (numfields != len) then
        next
    end
    rowindex = 0
    rowname = arr[rowindex]
    arr.delete_at(rowindex)
    row = Text.new(rowname)
    b = BatchUpdate.new(row)

    arr.each_with_index do |v, i|
        str = java.lang.String.new(v)
        b.put(coltextarr[i], str.getBytes("UTF-8"))
    end
    table.commit(b)
end

Sunday, August 24, 2008

Installation and configuration of Hadoop and Hbase

Hadoop

Installation
Hadoop installation instructions: http://hadoop.apache.org/core/docs/current/quickstart.html and http://hadoop.apache.org/core/docs/current/cluster_setup.html.
To set up hadoop cluster, generally two configuration files should be modified:
hadoop-site.xml and slaves.
(1) My hadoop-site.xml looks like:
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>pg3:9000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>pg3:9001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>
Read file hadoop-default.xml for all available options.
(2) My slaves file looks like:
localhost
pg1
pg2

I need to install Hadoop on three machines now and I use rsync to make these machines synchronized with eath other in terms of configuration.

Commands
(*) Format a new file system: hadoop namenode -format
(*) Start/stop Hadoop
start-dfs.sh/stop-dfs.sh
start up the distributed file system (HDFS)
start-mapred.sh/stop-mapred.sh
start up map reduce service.
start-all.sh/stop-all.sh
start up both HDFS and map reduce service

Hadoop reads content in file slaves to get all nodes and then starts up all these nodes.

Check status of the services
HDFS: http://domain:50070/
MapReduce: http://domain:50030/

HBase

Installation instructions: http://hadoop.apache.org/hbase/docs/current/api/overview-summary.html#overview_description
The configuration file is hbase-site.xml. My hbase-site.xml looks like
<configuration>
  <property>
    <name>hbase.master</name>
    <value>pg3:60000</value>
    <description>The host and port that the HBase master runs at.</description>
  </property>

  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://pg3.ucs.indiana.edu:9000/hbase</value>
    <description>The directory shared by region servers.</description>
  </property>
</configuration>

Commands
start-hbase.sh    starts up hbase service
stop-hbase.sh      stop hbase service

Note: hbase bases its functionalities on hadoop. Sometimes it is necessary for hbase to know the configuration of hadoop. Following statements are excerpted from hbase document which I think is important:

"Of note, if you have made HDFS client configuration on your hadoop cluster, hbase will not see this configuration unless you do one of the following:
  • Add a pointer to your HADOOP_CONF_DIR to CLASSPATH in hbase-env.sh
  • Add a copy of hadoop-site.xml to ${HBASE_HOME}/conf, or
  • If only a small set of HDFS client configurations, add them to hbase-site.xml
An example of such an HDFS client configuration is dfs.replication. If for example, you want to run with a replication factor of 5, hbase will create files with the default of 3 unless you do the above to make the configuration available to hbase. "

Monday, August 18, 2008

Mysql error "InnoDB: Unable to lock ./ibdata1, error: 11"

Recently, there was a power outage in the lab. I did not shut down my machines before the power outage. After I restarted my Ubuntu, I could not start up Mysql. The error was:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Error in opening ./ibdata1

But I am 100% sure that no another mysqld process was running. After I searched on line, I found that I was not the only one encountering this error. See this post.
This post has an insight into this problem. It seems to be caused by NFS. Mysql is not installed on a local file system. It is installed on a remote file system which is mounted to other file systems by using NFS.

Solution
make a copy of the original files (ibdata1, ib_logfile0, ib_logfile1...).

mv ibdata1 ibdata1.bak
cp -a ibdata1.bak ibdata1
......

Tuesday, July 29, 2008

Social Networking on Drupal

Previously, I tried several open source social networking frameworks including elgg, aroundme, peopleaggregator... However, few of them are mature and stable. So I also tried Content Management Systems. I delved into Drupal.
Drupal is an excellent CMS which has many powerful modules. Some modules provide social networking support. There exists a drupal group which promotes social networking support in Drupal.
Here I list some useful modules which may be used in social networking:
Organic Groups
"Enable users to create and manage their own groups. Each group can have subscribers, and maintains a group home page. Groups get their own theme, language, taxonomy, and so on."

Buddylist
Allows users to add other users to buddy list, organize buddies into buddy group and monitor buddies' content.

Invite
Adds an 'Invite a friend' feature that allows users to send and track invitations to join a web site.

FOAF
Functionalities:
"1. automatically import/synchronize profile information between any Drupal-powered sites
2. import profile information from external FOAF files
3. export a FOAF file based on a user's profile"

Privatemsg
Allows users to send private messages to each other.

User Relationships
Allow users to create named relationships between each other.

Wiki
interwiki
This filter provides a simplified, wiki-like syntax for linking to articles or search results on many commonly-used internet reference websites
Liquid Wiki
The Liquid Wiki Engine Project aims to build a module providing wiki funktionality to Drupal.
MediaWiki
Integrate Drupal with an existing MediaWiki installation.

Online Chat
Chat Room
This module enables chat rooms to be set up on Drupal sites. No additional software is required. Chat Room uses AJAX to update chats without page reloads. Chat rooms are nodes that provide access to chats and chat archives.
BoWoB
Free Chat for your website, integrated with your Drupal users and design. It is implemented in Flash.