Shell scripts for many jobs

Okay, okay, cosmosim users out there. You’ve won. I’m finally providing here short shell-scripts for submitting many jobs at once and downloading their results via the (linux) command line. I’ve been asked repeatedly how this can be done, and it’s probably not that straightforward to figure it out using the documentation page Scripted Access (UWS). There are many ways to handle jobs, and though I would recommend using Python (UWS-client or astroquery.cosmosim package, also see astroquery post) for extensive use, it may be instructive to see how it can also be done with basic shell-scripts using some httpie, awk and sed magic.


Update: I’ve put the scripts now on GitHub as well. I will keep the links here for archiving purposes, but recommend to always use the latest version from GitHub. If the UWS-interface changes, it’s more likely that the GitHub versions will be up-to-date.


So, without further ado, here they are:

If you don’t have httpie at hand and cannot install it, then download the uws-client instead from https://github.com/adrpar/uws-client/ and use following versions:

After putting your credentials at username and password for each script, you can run them like this for testing:

bash cosmosim-multiqueries.sh jobids.txt
bash cosmosim-download.sh jobids.txt
bash cosmosim-delete.sh jobids.txt

If you use the uws-client versions, you also need to add the path to the uws-executable (in uws-client/bin/) or make a softlink to that file in your local directory.

cosmosim-multiqueries.sh

This creates several jobs, submits them to CosmoSim and stores the resulting jobids in a file.
The name of this file needs to be provided as command line argument like this:

bash cosmosim-multiqueries.sh jobids.txt

Before doing this, please adjust username and password to your credentials. Also take care to provide a new table-basename. If tables have identical names, there will be an error (which the script does not catch) and the job won’t run.

Run the script to see, if it is working. If not, please contact us.
If everything went smoothly, you can go ahead and change the query and its parameters to your needs.

The resulting jobid-file can then be used for downloading and deleting the jobs later on with the other scripts.

cosmosim-download.sh

This script downloads the results in csv-format for all jobids given in a file (see below how to get the jobids if you don’t have them anymore).
This only works if the jobs have finished already and no error occured. You can check this on the webpage or by checking the phase via UWS. If the jobs are ready, download the results with:

bash cosmosim-download.sh jobids.txt

This downloads the files to the current directory. See Access via UWS how to provide your own custom file names instead.

cosmosim-delete.sh

When you’ve downloaded your data, you probably don’t need them anymore, so clean up by deleting the jobs on the database server:

bash cosmosim-delete.sh jobids.txt

This again uses a file with jobids to delete the corresponding jobs.

cosmosim-getjobids.sh

If you want to download results or delete jobs with the scripts above, you will need a file with the necessary jobids.
You can get a full job list easy enough using http --auth : --print b GET https://www.cosmosim.org/uws/query, but then you still need to parse the output to find the ids.
This script allows you to provide a job name pattern (same as the table name) and prints the corresponding jobids and status.

bash cosmosim-getjobids.sh bdm85- newjobids.txt

The jobids will be written to the file provided here. If the file exists already, it will be overwritten.

Disclaimer

Please note that these scripts have no proper error handling and are quite basic. I’ll be happy to help, if things don’t work for you. But if you plan to run multiple queries on a regular basis, it’s probably better to put more efforts into using the UWS-client or astroquery.cosmosim package for Python.

Previous post:     Next post:
Proudly powered by Daiquiri
©2016 The CosmoSim databaseImprint and Data Protection Statement