160 likes | 167 Views
Creation of an instrument maintenance program at W.M. Keck Observatory. This is the story of the IPM project which ran for FY09 and FY10: The following slides were presented at SPIE 2014 Article reprints available (just ask) Intended as advice for those attempting something similar
E N D
Creation of an instrument maintenance program at W.M. Keck Observatory
This is the story of the IPM project which ran for FY09 and FY10: • The following slides were presented at SPIE 2014 • Article reprints available (just ask) • Intended as advice for those attempting something similar • Emphasis on recounting lessons learned despite how obvious they might be in hindsight • The motivation, challenges, history, etc. are presented frankly The project team included: • OID: Grant Hill, Dwight Chan, Mike Wagner, Eric Appleby, Rich Matsuda • Software: Shui Kwok, Jeff Mader, Kevin McCann • Observing Support: Greg Wirth, Scott Dahm, Bob Goodrich, Marc Kassis, Jim Lyke
Background Instrument maintenance has historically posed a challenge at Keck • Resource constrained (lean) operations model catch-22 • Just enough resources for support & corrective maintenance • Inability to perform PM’s leads to……. corrective maintenance Instrument PM system creation was difficult at first • Early efforts suffered from……. • Attempting “as time permits” within operations (pre-project) • Resource competition from higher priority projects • Incomplete combination of skills allocations • Well resourced project was begun in 2009 • Coincident with creation of cross-disciplinary an OID • Right mix and quantity of skills with sufficient priority to retain them • Project ran for 2 years & used about ¾ FTE/year.
Project Vision & Deliverables Instrument PM tasks……. • No heritage of systematic, documented instrument maintenance • Even simple lists of PMs did not exist • Key deliverable develop, document & perform PMs for 1st time Facilitated by extensive, flexible & intuitive software • Capable of interfacing with virtually all other WMKO scheduling tools • Highly automated and intelligent (critical in a lean operations model) • Schedules, generates, and delivers work orders • Able to make decisions based on schedules and resources • Considerable interaction via email and web pages • Extensive archive to store task descriptions & performance records • Highly searchable via complex queries • Capable of storing drawings, pictures, FITS files, documents, etc.
Key planning decisions Build or buy? • Purchase commercial (turnkey) package vs. develop in-house • Allows greater control and integration Development and Deployment strategy • Broad but shallow vs. Narrow but deep • Proceed in order of cost/benefit • More immediate return frees up more time do more PMs • Charge ahead vs. wait for software • Start simple find problems pays back any missteps • Results aid software development Who will develop and perform the tasks? • Small summit instrument team stretched thin • Luxury of choice did not exist (engineer and technician too busy) • Instrument Scientist + ………
The system in use (tasks & work orders) Tasks • Procedural and scheduling information entered & viewed via web • Pictures, drawings, documents etc. can be attached • System now expanded to include facilities, safety, laser, telescopes, coatings, HQ facilities, AO, & ……….. Work orders • Notifications delivered via email • Contains links to task page and work order page • Basic interaction can be via replies to email • Work results and links to other pages (e.g. day log) can be included on work order pages • System will bug you if a work order is not responded to.
The delivered system……Success? • Competition from other projects led to some de-scope • But works very well for preventive and predictive maintenance • Rest of observatory voting with their feet (system still growing) • Through 2nd year of project and into ops…….. • Lag between task entry and activation at first • Gap between work orders issued and completed is growing • Jump in problems found after jump in task activation • Gap in problems found and fixed persists • Already seeing inability of small operations staff to keep up • What about the only metric that really matters? • Fault rate for non-AO instruments dropped • Equivalent to 3 full nights of science per year • Roughly equal to 3 papers per year • System paid for itself roughly twice over by the time of its handover to operations
Lessons learned (project phase) • Dedicated effort may be needed to create an instrument PM system • Very challenging to use operations effort • Project must have priority, resources, and right mix of skills • Competition with other projects can send you below critical mass • Leveraging time savings capabilities of software may be critical • Especially if operations model is resource constrained • For Keck it made sense to build the system software in-house • Shallow but broad approach will pay off sooner • Reduction in corrective maintenance may kick in sooner than you think • Don’t wait for software… Just start doing PMs no matter how simple • Be pragmatic. Right combination of people is better than no-one • It’s a jungle out there. Be prepared to be out-prioritized • Have viable de-scope options prepared • Emphasize user driven design and flexibility • People will vote with their feet
Lessons learned (post-project) Be prepared for turnover to operations • Invest effort (user testing and design) into making software simple & intuitive • Don’t de-scope documentation How will you run the system? • Be prepared to rationalize staffing changes to leverage the system • Make sure a system “champion” exists. Someone who…… • Monitors system usage, work order completion, looks for problems • Optimizes task parameters such as scheduling frequency • Ensures corrective maintenance leads to new PM’s entered • “Opportunistic” PMs are done • PMs are created for new instruments and upgrades
Lessons Learned (+ 2.5 years Epilog) • Make sure the system facilitates analysis, reporting and planning • Need stats and plots • Need to be able to manage PM’s in good times and especially when times are tough • Be prepared to revisit (and change) “great” ideas • Work orders auto-close and re-issue according to telescope schedule • Conversely, make sure good features are used. • Tasks are owned by aliases… not by people. • System complements (not replaces!) the day log system
Lessons Learned (+2.5 years Epilog) • Make sure there is oversight • Users may seem to know what they are doing but there can be pitfalls. • More features means more ways to trip the unwary or inexperienced. • Don’t rely on automation to warn you that things are amiss. • Be flexible in how you use the system • Leverage the natural skills and inclinations of people. Don’t square peg and round hole things. • Different approaches/methods may work best for different functional areas.