Monday, November 29, 2010

Schedulers in supercomputers

LoadLeveler

http://www.ccs.ornl.gov/eagle/LL.html
http://www.bu.edu/tech/research/training/scv-software-packages/loadleveler/
http://kb.iu.edu/data/azvs.html

     
Cluster status llstatus  
List jobs in the queues/classes llq  
List information of classes/queues llclass  
Reservation check llqres This may or may not work. Probably you should command showres

PBS

pbsnodes

Moab

http://www.clusterresources.com/products/mwm/docs/a.gcommandoverview.shtml
http://www.clusterresources.com/products/maui/docs/16.1simulationoverview.shtml

mshow -a  # show available resources

Saturday, November 27, 2010

IOMeter

Document: http://iometer.cvs.sourceforge.net/viewvc/iometer/iometer/Docs/Iometer.pdf

Windows: install IOMeter (it includes both dynamo and iometer GUI)

Linux: the linux package only includes dynamo (the runtime engine). It does not include GUI. So you need to connect dynamo to the Windows box on which IOMeter is installed. Use following command to start dynamo

sudo ./dynamo -i iometer_computer_host -n manager_name -m manager_computer_host

Parameters:
iometer_computer_host: host/ip where IOMeter GUI is run (Windows box)
manager_computer_host: host/ip where dynamo is run
manager_name: name of the dynamo manager (can be arbitrary. used to distinguish different dynamo managers connecting to the same IOMeter server)

Several options that are confusing

  1. Disk Targets > # of Outstanding I/Os
    "specifies the maximum number of outstanding asynchronous I/O operations per disk the selected worker(s) will attempt to have active at one time."
    "This control can be overridden by the # of Outstanding I/Os control group in the Test
    Setup tab (depending on the Cycling Options)."
  2. Test Setup > Cycling Options
    "The Cycling Options control group specifies the series of tests that is run for each access specification."
    • Number of workers per manager
      Workers are added in the order they are shown in the Topology panel
    • Number of targets per worker
      selected targets are added to each worker in the order they are shown in the Disk Targets or Network Targets tab for that worker.
    • Number of outstanding I/O operations per target (disk workers only).
    Cycling Options
    1. Normal - run all selected targets for all workers.
      In all managers, all works are active and each work uses all its selected targets. "The number of outstanding I/Os per disk is specified by the # of Outstanding I/Os field in the Disk Targets tab."
    2. Cycle Workers -- add step workers using all selected targets at a time
      "This test type increases the number of workers for each manager in each test.". In each test, the first N (depending on iteration and step) workers listed in Topology panel are used.
    3. Cycle Targets -- add step targets for all workers at a time.
      This test type increases the number of targets for each worker in each test.
    4. Increment Targets Parallel -- add step targets to all managers at a time.
      Test # Manager 1  Manager 2  Targets per Manager
      Worker 1  Worker 2  Worker 1  Worker 2 
      1 1 target  0 targets  1 target  0 targets  1 target 
      2 2 targets  0 targets  2 targets  0 targets  2 targets 
      3 2 targets  1 target  2 targets  1 target  3 targets 
      4 2 targets  2 targets  2 targets  2 targets  4 targets 
    5. Increment Targets Serial -- add step targets at a time.
       Test #    Manager 1    Manager 2    Total Targets 
       Worker 1    Worker 2    Worker 1    Worker 2    
       1    1 target    0 targets    0 targets    0 targets    1 target  
       2    2 targets    0 targets    0 targets    0 targets    2 targets  
       3    2 targets    1 targets    0 targets    0 targets    3 targets  
       4    2 targets    2 targets    0 targets    0 targets    4 targets  
       5    2 targets    2 targets    1 target    0 targets    5 targets  
       6    2 targets    2 targets    2 targets    0 targets    6 targets  
       7    2 targets    2 targets    2 targets    1 target    7 targets  
       8    2 targets    2 targets    2 targets    2 targets    8 targets  
    6. Cycle Targets and Workers -- add step targets at a time spread across workers
      Test  #  Worker 1  Worker 2  Worker 3  Targets  per Manager 
      1 1 target  0 targets  0 targets  1 target 
      2 2 targets  0 targets  0 targets  2 targets 
      3 1 target  1 target  0 targets  2 targets 
      4 2 targets  2 targets  0 targets  4 targets 
      5 1 target  1 target  1 target  3 targets 
      6 2 targets  2 targets  2 targets  6 targets 
    7. Cycle # Outstanding I/Os -- run step outstanding I/Os on all disks at a time.
    8. Cycle # Outstanding I/Os and Targets -- run step outstanding I/Os on step targets at a time

Not all managers have the same number of works. Not all works have the same number of targets. Read following paragraph to see how IOMeter handles it.

"The ending value for each sequence is determined differently for each quantity. For the number of workers per manager, it is determined by the manager with the most workers in the Topology panel. For the number of targets per worker, it is determined by the worker with the most targets selected in the Disk Targets or Network Targets tab. For the outstanding I/Os per target, it is determined by the # of Outstanding I/Os control group in the Test Setup tab.
    If not all managers have the same number of workers, or not all workers have the same number of targets, those managers/workers with less than the maximum number of workers/targets will repeat at their highest value while other managers/workers continue to increase.  "

In Disk Targets Tab, "A yellow icon with a red slash through it means that the drive needs to be prepared before the test starts".

"Yellow disk icons represent logical drives (i.e. those with a drive letter). They are shown only if they are writable. Disk workers access logical drives by reading and writing a file called iobw.tst in the root directory of the drive. If this file exists, the drive is shown with a plain yellow icon; if the file does not exist, the drive is shown with a red slash through the icon. (If this file exists but is not writable, the drive is considered read-only and is not shown at all.)
    If you select a drive that does not have an iobw.tst file, Iometer will begin the test by creating this file and expanding it until the drive is full (this is shown as “Preparing Drives” in the status bar). You can change the size of this file to control how much of the disk Iometer can use by setting the Maximum Disk Size control to the desired size (in sectors).
    Blue disk icons represent physical drives. They are shown only if they contain nothing but free space (no defined partitions). Disk workers access physical drives by writing direct to the raw disk. Physical drives do not use an iobw.tst file. Running with physical drives is recommended."

Friday, November 12, 2010

JsUnit Maven Plugin

Document is at http://jsunit.berlios.de/maven2.html. It's too brief, especially following paragraph:

The type of the test suite, one of the following values:

ALLTESTS
Looks for a class AllTests derived from TestSuite and runs its suite.
TESTSUITES
Looks for all classes ending with TestSuite and that are derived from TestSuite and run their suites.
TESTCASES
Looks for all classes ending with TestCase and that are derived from TestCase and runs them (the default).

The problem is what "derived from" means and how to to that. I will show in detail how to use JsUnit plugin.

1) sample test file

Following is a dummy test file. It should be put into src/test/js.

var dummyobj = dummyobj || {};

function DummyTest(name) {
  TestCase.call(this, name);
};

DummyTest.inherits(TestCase);

DummyTest.prototype.setUp = function() {
    dummyobj.name = "Gerald";
};

DummyTest.prototype.tearDown = function() {
    delete dummyobj.name;
};

DummyTest.prototype.testDummy = function() {
  this.assertEquals('Gerald', dummyobj.name);
};

DummyTest is the test case.
It "inherits" from class TestCase. All of its functions whose name start with "test" will be tested.

2) Inherit implementation

Following is implementation of inherit borrowedfrom Shindig code. It can be put in a file inherit_implementation.js under directory src/main/js.

Function.prototype.inherits = function(parentCtor) {
    function tempCtor() {};
    tempCtor.prototype = parentCtor.prototype;
    this.superClass_ = parentCtor.prototype;
    this.prototype = new tempCtor();
    this.prototype.constructor = this;
};

3) Pom.xml

<plugin>
    <groupId>de.berlios.jsunit</groupId>
    <artifactId>jsunit-maven2-plugin</artifactId>
    <executions>
        <execution>
            <id>test</id>
            <configuration>
                <sourceDirectory>${basedir}/src/main/js</sourceDirectory>
                <sources>
                    <source>inherit_implementation.js</source> 
<source>file_to_be_tested.js</source> </sources> <testSourceDirectory>${basedir}/src/test/js</testSourceDirectory> <testSuites> <testSuite> <name>SampleSuite</name> <type>TESTCASES</type> <includes> <include>*/*test.js</include> </includes> </testSuite> </testSuites> </configuration> <goals> <goal>jsunit-test</goal> </goals> </execution> </executions> </plugin>

Resources

For some tests, you need to provide fake window object, XmlHttpRequest object, DOM objects, etc. Project env-js (site) is exactly designed for this purpose. It provides a simulated browser environment.

Monday, November 08, 2010

How to run/debug Shindig in Eclipse

Instructions here (http://shindig.apache.org/developers/java/build.html#Setting_up_an_Eclipse_project_to_build_Apache_Shindig) are for old versions of Eclipse.

I am using Eclipse Helios. You should install Maven plugin m2eclipse before following following instructions.

  1. Download or check out shindig code.
  2. File –> Import
    Maven –> Existing Maven Projects
    Specify the root directory of shindig code.
  3. Right click top level imported project (*-project):  Debug As –> Debug Configuration
    Create a new configuration for “Maven Build”.
    Base directory: root directory of shindig code
    Profiles: run
    Unselect “Skip Tests”.
    If you want to use a port number rather than 8080, add a parameter “jetty.port”.
    Clicek “Debug”.
  4. Jetty server should run successfully. Look at your console for possible error messages.

Add breakpoints, then send a request to Jetty server. Eclipse complains that it cannot find source code for the debugged app, and prompt you to add source lookup directories.