Back to Tools

About QosCosGrid (QCG)

QosCosGrid (QCG) is a two-layer architecture which is able to deal with computationally intensive, large-scale, complex and parallel simulations that require multiple computational sites. QosCosGrid is able to combine resources from different administrative domains into a single powerful machine and is tightly integrated with commonly used tools for parallel and multiscale simulations (e.g., OpenMPI, ProActive and more recently MUSCLE) using advance reservation and co-allocation features.

Currently, QosCosGrid supports a wide range of development frameworks and programming models relevant for multiscale application developers, QosCosGrid gives the ability to work across heterogeneous computing sites and hides the complexity of underlying e-Infrastructures by simplifying many complex deployment, advance reservation and access procedures.

More information available at: http://www.qoscosgrid.org

FrontPage Recent Changes All Pages Orphan Pages Draft Pages

QosCosGrid tutorial

Details Print

Running SimpleExample with QosCosGrid services #

  • Initialize QCG environment: 
[mschoolXX@grass1 ~]$ module load qcg-client/mapper
  • Generate proxy (the private key is protected using the same password as your shell account): 
[mschoolXX@grass1 ~]$ grid-proxy-init
Your identity: /O=Grid/OU=MAPPER/OU=simpleCA-MAPPER/OU=man.poznan.pl/CN=mschool01
Enter GRID pass phrase for this identity:
Creating proxy ...................................... Done
  • Look at the example JobProfile (a QCG-Broker job description file) 
vim /opt/app/mapper/qcg/SimpleExample/simple-mavrino-grass.xml
  • In this file you specify:
a) Where to run each kernel. In our case the "r" kernel would be started at mavrino and "w" at grass'es cluster: 
         <topology>
                 <processes processesId="r">
                         <processesCount>
                                 <value>1</value>
                         </processesCount>
                         <candidateHosts>
                                 <hostName>mavrino.chem.ucl.ac.uk</hostName>
                         </candidateHosts>
                 </processes>
                 <processes processesId="w">
                         <processesCount>
                                <value>1</value>
                         </processesCount>
                         <candidateHosts>
                                <hostName>grass1.man.poznan.pl</hostName>
                        </candidateHosts>
                </processes>
         </topology>
b) The first argument of the muscle application must be the CxA file name: 
         <arguments>
                 <value>SimpleExample.cxa.rb</value>
         </arguments>
c) One must specify how to stage the CxA file to the remote clusters (in this case the file is taken from examples directory located at grass1): 
          <stageInOut>
               <file name="SimpleExample.cxa.rb" type="in">
                         <location type="URL">gsiftp://grass1.man.poznan.pl///opt/app/mapper/qcg/SimpleExample

/SimpleExample.cxa.rb</location>
               </file>
          </stageInOut>
d) Also where to put output stdout and stderr files (we want to store the files in our home directory, using QCG Job ID as suffix for the file names): 
          <stdout>
               <directory>
                    <location type="URL">gsiftp://grass1.man.poznan.pl/~/stdout-${JOB_ID}</location>
               </directory>
          </stdout>
          <stderr>
               <directory>
                    <location type="URL">gsiftp://grass1.man.poznan.pl/~/stderr-${JOB_ID}</location>
              </directory>
          </stderr>
e) And finally the execution duration of the experiment: 
          <executionTime>
               <executionDuration>P0Y0M0DT0H10M</executionDuration>
          </executionTime>
  • Start the QCG-Broker client in -console mode 
[mschoolXX@grass1 ~]$ qcg-client -console
https://elder11.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /O=Grid/OU=MAPPER/OU=simpleCA-MAPPER/OU=man.poznan.pl/CN=mschool01
ProxyLifetime = 0 Days 11 Hours 11 Minutes 51 Seconds

qcg>
  • You are ready to submit a job: 
qcg> submit_job /opt/app/mapper/qcg/SimpleExample/simple-mavrino-grass.xml
jobId = J1327856866253_MAPPER_5637
  • After job is submitted you can query about it status: 
qcg> task_info J1327856866253_MAPPER_5637 muscle
TaskType: MAPPER
SubmissionTime: Sun Jan 29 18:07:46 CET 2012
FinishTime: 
ProxyLifetime: PT0S
Status: RUNNING
StatusDesc: 
ScheduledTime: Sun Jan 29 18:08:00 CET 2012 - Sun Jan 29 18:19:00 CET 2012
StartTime: Sun Jan 29 18:07:56 CET 2012

Allocation: 
UserDN: /O=Grid/OU=MAPPER/OU=simpleCA-MAPPER/OU=man.poznan.pl/CN=mschool01
HostName: grass1.man.poznan.pl
ProcessesCount: 1
ProcessesGroupId: w
Status: RUNNING
StatusDescription: 
SubmissionTime: Sun Jan 29 18:07:54 CET 2012
FinishTime: 
LocalSubmissionTime: Sun Jan 29 18:07:55 CET 2012
LocalStartTime: Sun Jan 29 18:08:06 CET 2012
LocalFinishTime: 

Allocation: 
UserDN: /O=Grid/OU=MAPPER/OU=simpleCA-MAPPER/OU=man.poznan.pl/CN=mschool01
HostName: mavrino.chem.ucl.ac.uk
ProcessesCount: 1
ProcessesGroupId: r
Status: RUNNING
StatusDescription: 
SubmissionTime: Sun Jan 29 18:07:54 CET 2012
FinishTime: 
LocalSubmissionTime: Sun Jan 29 18:07:58 CET 2012
LocalStartTime: Sun Jan 29 18:08:08 CET 2012
LocalFinishTime:
  • you can use $$ as the alias for the last submitted job id 

qcg> task_info $$ muscle                        
Command translated to:
task_info J1327856866253_MAPPER_5637 muscle
TaskType: MAPPER
SubmissionTime: Sun Jan 29 18:07:46 CET 2012
FinishTime: Sun Jan 29 18:08:59 CET 2012
ProxyLifetime: PT0S
Status: FINISHED
StatusDesc: 
ScheduledTime: Sun Jan 29 18:08:00 CET 2012 - Sun Jan 29 18:19:00 CET 2012
StartTime: Sun Jan 29 18:07:56 CET 2012

Allocation: 
UserDN: /O=Grid/OU=MAPPER/OU=simpleCA-MAPPER/OU=man.poznan.pl/CN=mschool01
HostName: grass1.man.poznan.pl
ProcessesCount: 1
ProcessesGroupId: w
Status: FINISHED
StatusDescription: 
SubmissionTime: Sun Jan 29 18:07:54 CET 2012
FinishTime: Sun Jan 29 18:08:32 CET 2012
LocalSubmissionTime: Sun Jan 29 18:07:55 CET 2012
LocalStartTime: Sun Jan 29 18:08:06 CET 2012
LocalFinishTime: Sun Jan 29 18:08:32 CET 2012

Allocation: 
UserDN: /O=Grid/OU=MAPPER/OU=simpleCA-MAPPER/OU=man.poznan.pl/CN=mschool01
HostName: mavrino.chem.ucl.ac.uk
ProcessesCount: 1
ProcessesGroupId: r
Status: FINISHED
StatusDescription: 
SubmissionTime: Sun Jan 29 18:07:54 CET 2012
FinishTime: Sun Jan 29 18:08:58 CET 2012
LocalSubmissionTime: Sun Jan 29 18:07:58 CET 2012
LocalStartTime: Sun Jan 29 18:08:08 CET 2012
LocalFinishTime: Sun Jan 29 18:08:56 CET 2012
  • The job is finished. You are now ready to quit the qcg-client and read the output of the application: 
qcg> quit
BYE ...
[mschool01@grass1 ~]$ cat stdout-J1327856866253_MAPPER_5637/stdoutmavrino.chem.ucl.ac.uk_1
...
got: 0.0
got: 1.0
got: 2.0
got: 3.0
got: 4.0

got: 0.0
got: 1.0
got: 2.0
got: 3.0
got: 4.0
...
  • Complete QCG-Broker client manual can be found here

Running Canals Application with QosCosGrid services #

  • In this example we define a workflow job that is composed of 3 tasks:
    • muscle - a task that describe muscle simulation coallocated between two clusters
    • render-images - a task depended on the muscle task that generates png images based on simulation data
    • render-video - a task that generates video from images generated by render-images task 
<qcgJob appId="MAPPER" xmlns:jxb="[http://java.sun.com/xml/ns/jaxb"] xmlns:xsi="[http://www.w3.org/2001/]
XMLSchema-instance">
	<task persistent="true" taskId="muscle">
		<requirements>
			<topology>
				<processes processesId="SW1D_1B_001:SW1D_2B_005:SW1D_1B_011">
					<processesCount>
						<value>1</value>
					</processesCount>
					<candidateHosts>
						<hostName>mavrino.chem.ucl.ac.uk</hostName>
					</candidateHosts>
				</processes>
				<processes processesId="LinearGate1B_021:SpillWay_015">
					<processesCount>
						<value>1</value>
					</processesCount>
					<candidateHosts>
						<hostName>grass1.man.poznan.pl</hostName>
					</candidateHosts>
				</processes>
			</topology>
		</requirements>
		<execution type="mapper">
			<executable>
				<application name="muscle"/>
			</executable>
			<arguments>
				<value>Canals.cxa.rb</value>
			</arguments>
			<stdout>
				<directory>
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/stdout-${JOB_ID}</location>
				</directory>
			</stdout>
			<stderr>
				<directory>
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/stderr-${JOB_ID}</location>
				</directory>
			</stderr>
			<stageInOut>
				<file name="Canals.cxa.rb" type="in">
					<location type="URL">gsiftp://grass1.man.poznan.pl///opt/app/mapper/qcg/Canals

/Canals.cxa.rb</location>
				</file>
				<directory name="outputs" type="out">
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/canals-output</location>
				</directory>
			</stageInOut>
		</execution>
		<executionTime>
			<executionDuration>P0Y0M0DT0H10M</executionDuration>
		</executionTime>
	</task>
	<task persistent="true" taskId="render-images">
		<execution type="single">
			<executable>
				<application name="CanalRenderer"/>
			</executable>
			<arguments>
				<value>canals-outputs</value>
				<value>canals-images</value>
			</arguments>
			<stdout>
				<file>
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/stdout-${JOB_ID}</location>
				</file>
			</stdout>
			<stderr>
				<file>
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/stderr-${JOB_ID}</location>
				</file>
			</stderr>
			<stageInOut>
				<directory name="canals-outputs" type="in">
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/canals-output.images</location>
				</directory>
				<directory name="canals-images" type="out">
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/canals-images.images</location>
				</directory>
			</stageInOut>
		</execution>
                <workflow>
                        <parent triggerState="FINISHED">muscle</parent>
                </workflow>
	</task>
	<task persistent="true" taskId="render-video">
		<execution type="single">
			<executable>
				<application name="Mencoder"/>
			</executable>
			<arguments>
				<value>mf://canals-images/*.png</value>
				<value>-mf</value>
				<value>w=800:h=600:fps=25:type=png</value>
				<value>-ovc</value>
				<value>lavc</value>
				<value>-lavcopts</value>
				<value>vcodec=mpeg4:mbd=2:trell</value>
				<value>-oac</value>
				<value>copy</value>
				<value>-o</value>
				<value>canals.avi</value>
			</arguments>
			<stdout>
				<file>
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/stdout-${JOB_ID}.video</location>
				</file>
			</stdout>
			<stderr>
				<file>
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/stderr-${JOB_ID}.video</location>
				</file>
			</stderr>
			<stageInOut>
				<directory name="canals-images" type="in">
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/canals-images</location>
				</directory>
				<file name="canals.avi" type="out">
					<location type="URL">gsiftp://grass1.man.poznan.pl/~/canals.avi</location>
				</file>
			</stageInOut>
		</execution>
                <workflow>
                        <parent triggerState="FINISHED">render-images</parent>
                </workflow>
	</task>
</qcgJob>
  • Submit the Canal workflow: 
qcg> submit_job /opt/app/mapper/qcg/Canals/canals-mavrino-grass-workflow.xml 
jobId = J1327864110619_MAPPER_5509
  • You can easily monitor the execution of the worklflow by using the task_statuses command: 

qcg> tasks_statuses $$
Command translated to:
tasks_statuses J1327864110619_MAPPER_5509
Number of tasks: 3
Tasks statuses: 
muscle	: RUNNING
render-video	: UNSUBMITTED
render-images	: UNSUBMITTED
------ SUMMARY --------
Number of tasks: 3
UNSUBMITTED 	: 2
RUNNING 	: 1
  • In the end you can view the canals.avi movie that was uploaded into your home directory.
0 Attachments
608543 Views
Average (0 Votes)
Comments