Parallelism & HTCondor

I recently ran into HTCondor, which is an implementation of High Throughput Computing (HTC). It is developed and maintained by the University of Wisconsin – Madison.

HTCondor makes use of all available computational resources to get a job done. Computational resources could be processors in a single machine or a distributed computing system. It allocates jobs to different machines based on rules and has the ability to transfer jobs from 1 machine to another.

Condor generally finds its applications in servers/distributed networks/farms. However it can be used to execute jobs in parallel on a single machine too.

1. Install condor

sudo apt-get install condor

2. Run condor_status to see your processors

condor_status -available -> shows available procs

condor_status -run -> shows procs that are running a job

Condor takes an SDF file as an input. If the various tasks you are running have dependencies, you can use DAGman. DAGman uses a directional acyclic graph to inform condor about the various dependencies. In all, it takes care of generating the SDF file for you (given a graph).
A sample sdf file can be found on the HTCondor web documentation.
My SDF file can be found below:

This job can be submitted to condor, which will allocate it to a processor
Lets say, I submit the job thrice:

Now a condor_status -run will show list of procs running a condor job.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: