1 / 7

Allison Andrews NERSC Storage Systems Group Oakland Scientific Facility Sept. 10, 2014

Automated provisioning in NGF with NIM. Allison Andrews NERSC Storage Systems Group Oakland Scientific Facility Sept. 10, 2014. Background. NGF general purpose storage resources(routinely available to nersc users and projects) Home Directories(/global/homes/<initial>/<user>)

jordan-morrow
Download Presentation

Allison Andrews NERSC Storage Systems Group Oakland Scientific Facility Sept. 10, 2014

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automated provisioning in NGF with NIM Allison AndrewsNERSC Storage Systems GroupOakland Scientific FacilitySept. 10, 2014

  2. Background • NGF general purpose storage resources(routinely available to nersc users and projects) • Home Directories(/global/homes/<initial>/<user>) • Scratch Directories (/global/scratch2/sd/<user>) • Project Directories (/project/projectdirs/<proj>) • JGI scratch directories (/global/projectb/scratch/<user>) • JGI project directories (/global/dna/projectdirs/<jgi program>/<proj>) • Give-take directories (/global/scratch/gt/<user>) • NGF Special purpose storage resources(available only by special arrangement) • “Common” directories for shared software • “Syscommon” filesystems for system management related storage • Other special purpose storage such as the seqfsfilesystem used by jgi gene sequencing machines • JGI sandbox directories (/global/projectb/sandbox/<jgi group>)

  3. Background home and scratch space is provisioned automatically from NIM Project directories, and special storage resources were provisioned manually Increasing project dir requests, and default project directories for every new project drive the automation of project directory provisioning NIM did not have mechanisms for representing storage resources like projectdirs, or project specific user dirs such as /global/projectb/scratch directories NIM did not have a mechanism to manage quota data, and transmit changes to resource providers(NGF). SSG, and the NIM team began a joint project to enhance NIM, and automate NGF provisioning tasks.

  4. Where we are today • NIM changes propagate to NGF every 10 minutes • Project directories created automatically(/project, but not /global/projectb, etc.) • Automatic updates of • Project directories owner(UID & GID) (root directory only, this is not recursive) • Project directory quotas • Home quotas • Scratch quotas

  5. What lies ahead • Replace userdir finger with the new NIM<->NGF automation framework • Add support for other general purpose NGF resources • /global/dna/projectdirs/<jgi program>/<proj> • /global/projectb/scratch/<user> • /global/scratch/gt/<user> • /your/next/filesystem • Add functionality to archive user home directories, and project directories • Add tool to periodically verify that NGF is correctly provisioned according to the data in NIM

  6. What is not in plan Automatic provisioning of common, and other specialized file systems.

  7. Thank you.

More Related