1 / 8

HST Proposal Processing, Planning and Scheduling

HST Proposal Processing, Planning and Scheduling. Solaris Desktops talking to a Solaris Disk Server via NFS Sybase database engine running on Solaris Software (Trans, Spike, SPSS, PASS, etc) binaries are Solaris only. Mac Integration Goals.

yeva
Download Presentation

HST Proposal Processing, Planning and Scheduling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. HST Proposal Processing, Planning and Scheduling • Solaris Desktops talking to a Solaris Disk Server via NFS • Sybase database engine running on Solaris • Software (Trans, Spike, SPSS, PASS, etc) binaries are Solaris only

  2. Mac Integration Goals • Provide additional data analysis support capabilities not directly obtainable on Solaris (eg, MS Office, etc). • Provide superior single-thread performance compared to current Sun workstations. • Allows data analyses and studies to be done faster and more efficiently • Integrate in a way that ‘looks’ like the Solaris environment • Personnel familiarity • Cross-platform tools/scripts likely to work out-of-the-box • Must be careful not to remove necessary Solaris compute capability required for the process

  3. NFS • Mount the data partitions from the Solaris Server onto the Mac • Mount point setup to be /data which means the paths on the Mac are identical to those on Solaris • UIDs/GIDs on Mac synchronized with Solaris which means same file protection schemes work between Solaris and Mac OS X • Note: Solaris ACLs and Mac OS X ACLs are different and not mutually recognizable • OK to mount from Solaris to Mac OS X (read/write). Likely problems exporting from Mac OS X to Solaris (HFS+ [case insensitive/case preserving] vs. UFS [case sensitive]) Mac OS X [bifrost:~] reinhart% ls -Hlg /data/scheduling total 19 drwx------ 2 root wheel 8192 Oct 19 2004 lost+found/ drwxrwsr-x 14 planinst spb 512 Jul 27 16:34 spss_flight_data/ drwxr-sr-x 32 spssoper spb 1024 Sep 5 14:13 spss_tools/ [bifrost:~] reinhart% Solaris -4> ls -Hlg /data/scheduling total 10 drwx------ 2 root root 8192 Oct 19 2004 lost+found/ drwxrwsr-x 14 planinst spb 512 Jul 27 16:34 spss_flight_data/ drwxr-sr-x 32 spssoper spb 1024 Sep 5 14:13 spss_tools/ -4>

  4. The software setup • Majority of the operations software is Solaris only • Tools written in a combination of Python, TCL and shell scripting languages • Given data paths are the same, most of the Tools and Tool libraries work with little to no changes • Use same tool login setup scripts with minor changes so that environment variables/paths are as identical as possible (OS specific requirements excluded) • Sybase client software installed on the Mac which allows for database queries from within the Tools as on Solaris • Similar supporting software suite as on Solaris: nedit, emacs, python matplotlib package, etc (installed via Fink) • Additional analysis software available on the Mac: MS Office, sqlite, python matplotlib basemap toolkit, python pyephem package, pysqlite, etc

  5. An Example Workflow2-FGS Operations • 23 scenarios run serially that utilizes SPSS on Solaris, but does the data analysis and results storage on the Mac • Start on the Mac • Setup Sybase SPSS database for the scenario • SSH to Solaris to run the script which runs all the guide star requests for that scenario. Return to the Mac when complete. • Extract the results from the Sybase database and store in a local sqlite database keyed by the scenario • Once all the scenarios are complete, pull certain data from sqlite database, process and store back in that database. • Pull results and make plots via matplotlib (equal area sky plots can be done via the basemap toolkit)

  6. sqlite • Included in standard Tiger install (sqlite3) • ANSI SQL compliant database • see http://www.sqlite.org for all the information • No complex setups necessary • Supports several simultaneous readers but only one writer (locking is at the database level) • Databases up to 10’s of gigabytes is supported (FGS Study database was 9GB) • Writing is rather slow but reading is reasonably fast. Some queries can be very memory intensive. • Convenient in many situations compared to managing large numbers of individual files, however, populating the database can take a while • Allows interactive query of dataset without writing/modifying a tool • pysqlite is a python-to-sqlite connection package allowing queries of sqlite database from within python • see http://www.pysqlite.org for more information

  7. sqlite example [bifrost:~/gs_study] reinhart% sqlite3 gs_study.db SQLite version 3.2.8 Enter ".help" for instructions sqlite> .tables gs_summary su_target_xref wgreswnd percent_change tgm_nogs_sched_windows wguide_stars qbwindows wgacquis study_description wggs_pairs sqlite> .schema gs_summary CREATE TABLE gs_summary (study_id text, proposal_id text, obset_id text, version_num text, total_angular_coverage float, num_angle_gaps integer, total_time_coverage_3g integer, num_time_gaps_3g integer, total_time_coverage_2g integer, num_time_gaps_2g integer, num_gs_pairs integer, norm_num_gs_pairs integer, total_day_coverage_3g float, total_day_coverage_2g float); CREATE UNIQUE INDEX gs_summary_1 on gs_summary (study_id, proposal_id, obset_id, version_num); sqlite> sqlite> select study_id, count(gs_pair_id) from wggs_pairs ...> where proposal_id = "00658" ...> and obset_id = "A2" ...> group by study_id; gsc1_3fgs_def_acs_3ps|231 gsc1_3fgs_def_acs_3s|51 gsc1_3fgs_def_ota_3ps|65 gsc1_3fgs_def_ota_3s|51 gsc2_3fgs_def_acs_3ps|231 gsc2_3fgs_def_acs_3s|223 gsc2_3fgs_def_ota_3ps|75 gsc2_3fgs_def_ota_3s|57 gsc2_nofgs2_145_acs_3ps|240 gsc2_nofgs2_145_acs_3s|203 gsc2_nofgs2_145_ota_3ps|44 gsc2_nofgs2_145_ota_3s|25 gsc2_nofgs2_150_acs_3ps|264 gsc2_nofgs2_150_acs_3s|238 gsc2_nofgs2_150_ota_3ps|59 gsc2_nofgs2_150_ota_3s|44 gsc2_nofgs2_def_acs_3ps|203 gsc2_nofgs2_def_acs_3s|150 gsc2_nofgs2_def_ota_3ps|43 gsc2_nofgs2_def_ota_3s|24 gsc2_nofgs3_145_acs_3s|161 gsc2_nofgs3_145_ota_3s|34 gsc2_nofgs3_150_acs_3s|172 gsc2_nofgs3_150_ota_3s|44 gsc2_nofgs3_def_acs_3s|124 gsc2_nofgs3_def_ota_3s|33 sqlite>

More Related