Basic User Guide

1 RADICAL-Pilot Setup

Documentation: http://radicalpilot.readthedocs.org/en/stable/machconf.html#preconfigured-resources

First, we will import the necessary dependencies.

import os, sys
import radical.pilot as rp
print rp.version

2. Local Pilot Example

This example shows how to execute simple tasks using a Pilot-Job.

2.1 Create a new Session and Managers.

session = rp.Session()
pmgr = rp.PilotManager(session=session)
umgr = rp.UnitManager (session=session,scheduler=rp.SCHED_ROUND_ROBIN)
print "Session id: %s \n \
 Pilot Manager: %s \n \
 Unit Manager: %s" % (session.uid, pmgr.uid, umgr.uid)

2.2 Submit Pilot and add to Unit Manager

Create a description of the Pilot, which specifies its target resource, size and runtime.

pdesc = rp.ComputePilotDescription()
pdesc.resource = "local.localhost"
pdesc.runtime  = 10 # minutes
pdesc.cores    = 2
pilot = pmgr.submit_pilots(pdesc)
umgr.add_pilots(pilot)

The umgr.add_pilot(pilot) binds a Pilot to a Unit Manager. This informs the Unit Manager that the resources of the Pilot can be used for exeuction of Units. Multiple Pilots can be bound to a Unit Manager, and later be removed via umgr.remove_pilot(). The Unit Manager will schedule incoming Compute Units across all available Pilots.

print "uid: ", pilot.uid
print "resource: ", pilot.resource
print "sandbox: ", pilot.sandbox
print "state: ", pilot.state

2.3 Submit Compute Units

Create a description of the Compute Unit, which specifies the details of the task to be executed.

cudesc1 = rp.ComputeUnitDescription()
cudesc1.environment = {'CU_NO': 1}
cudesc1.executable  = "/bin/echo"
cudesc1.arguments   = ['I am CU number $CU_NO']
cudesc1.cores       = 1

cudesc2 = rp.ComputeUnitDescription()
cudesc2.environment = {'CU_NO': 2}
cudesc2.executable  = "/bin/echo"
cudesc2.arguments   = ['I am CU number $CU_NO']
cudesc2.cores       = 1

Submit the previously created Compute Unit descriptions to the Unit Manager. This will trigger the selected scheduler (in this case the Round-Robin scheduler) to start assigning Compute Units to the Pilots.

print "Submit Compute Units to Unit Manager ..."
cu_set = umgr.submit_units([cudesc1, cudesc2])
print "Waiting for CUs to complete ..."
umgr.wait_units()
print "All CUs completed successfully!"

Inspect the output of the CUs

print cu_set[0].stdout
print cu_set[1].stdout

2.6 Close Session

The session must always be closed before exiting: this will tear down the Pilots, which would otherwise continue to consume resources until their specified runtime is up.

session.close()