1 / 23

Modeling Long Term Care and Supportive Housing

Modeling Long Term Care and Supportive Housing. Marisela Mainegra Hing Telfer School of Management University of Ottawa. Canadian Operational Research Society, May 18, 2011. Outline. Long Term Care and Supportive Housing Queueing Models Dynamic Programming Model

dillon-arnold
Download Presentation

Modeling Long Term Care and Supportive Housing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Modeling Long Term Care and Supportive Housing Marisela Mainegra Hing Telfer School of Management University of Ottawa Canadian Operational Research Society, May 18, 2011

  2. Outline • Long Term Care and Supportive Housing • Queueing Models • Dynamic Programming Model • Approximate Dynamic Programming

  3. LTC problem λRC λC Community LTC μLTC , CLTC λH Hospital λRH Goal: Hospital level below a given threshold Community waiting times below 90 days

  4. LTC previous results • MDP model determined a threshold policy for the Hospital but it did not take into account community demands • Simulation Model determined that current capacity is insufficient to achieve the goal

  5. λRC λC Community λC-LTC λLTC LTC λH Hospital μLTC , CLTC λH-LTC λRH H_renege μRH, Queueing Model Station LTC: M/M/CLTC Station H_renege: M/M/∞

  6. steady state: <1 Queueing Model Station LTC: M/M/CLTC The probability that no patients are in the system: The average number of patients in the waiting line: The average time a client spends in the waiting line: The number of patients from the Hospital that are in the queue for LTC (LqH-LTC).

  7. Queueing Model Station H_renege: M/M/∞ • The average number of patients in the system is

  8. Queueing Model Data analysis • Data on all hospital demand arriving to the CCAC from April 1st, 2006 to May 15th, 2009. • ρLTC = 1.6269 for current capacity CLTC= 4530 • To have ρLTC < 1 we need CLTC> 7370.08, 2841 (62.71%) more beds than the current capacity. • With CLTC > 7370 we apply the formulas. • Given a threshold T for the hospital patients and the number LqLTC of total patients waiting to go to LTC, what we want is to determine the capacity CLTC in LTC such as:

  9. Queueing Model Results • 19 iterations of capacity values • Goal achieved with capacity 7389, the average waiting time is 31 days and the average amount of Hospital patients waiting in the queue is 130 ( T=134) . • This required capacity is 2859 (63.1%) more than the current capacity.

  10. λH Hospital λRH λH-SH λH-LTC μRH, LTC λSH-LTC SH μLTC , CLTC μSH , CSH λC-LTC λC-SH H_renege Community λC λRC Queueing Model with SH

  11. Queueing Model with SHResults • Required capacity in LTC is 6835, 2305 (50.883%) more beds than the current capacity (4530). • Required capacity in SH is 1169. • With capacity values at LTC: 6835 and at SH: 1169 there are 133.9943 (T= 134) Hospital Patients waiting for care (for LTC: 110.3546, reneging: 22.7475, for SH: 0.89229), and Community Patients wait for care in average (days) at LTC: 34.8799, and at SH: 3.2433.

  12. Semi-MDP Model S = {(DH_LTC, DH_SH, DC_LTC, DC­_SH, DSH_LTC, CLTC, CSH, p) } A = {0,..,max(TCLTC,TCSH)} d(s,a) = Pr(s,a,s’) = r(s,a) = State space: Action space: Transition time: Transition probabilities: Immediate reward: Optimal Criterion: Total expected discounted reward

  13. Goal find π: S A that maximizes the state-action value function γ: discount factor Bellman: there exists Q* optimal: Q* =maxQ(s,a) and the optimal policy π* Approximate Dynamic programming

  14. Reinforcement action state Reinforcement Learning

  15. RL: environment ENVIROMMENT action state transition probabilities reward function next state, immediate reward

  16. RL: Agent Knowledge: Q(s,a) exploratory Learning: update Q-values state action Behavior reward Knowledge representation (FA) Learning method • Backup table • Neural network • ... • Watkins QL • Sarsa () • ...

  17. QL: parameters • θ: number of hidden neurons. • T: number of iterations of the learning process. • 0: initial value of the learning rate. • 0: initial value of the exploration rate. • Learning-rate decreasing function. • Exploration-rate decreasing function.

  18. explorationvs/ exploitation Learning and exploration rates QL: algorithm (T, θ T

  19. QL: tuning parameters(observed regularities) • (θ, )-scheme: T= 104, 0= 10-3, 0 =1, T= 103, T=v103 , v[1,.. ]. PR(θ, ): best performance with (θ, )-scheme • PR(θ, ) monotically increase respect  until certain value  (θ) • PR(θ, ) monotically increase respect θ until certain value θ() • (θ) and θ() depend on the problem instance

  20. QL: tuning parameters(methodology: learning schedule given PRHeu) • ∆θ =50, θ =0, PRθ =0,=0, vbest=1, • While PRθ <PRheu or no-stop • θ = θ + ∆θ, PRbest=0 • While PRbest ≥ PR • = +1, T= 104, T= 103 • PR=PRbest, • For v= vbest to  • T=v103 • PR[v]=Q-Learning(T, θ, 10-3, 1, T, T) • [PRbest,vbest]=max(PR) • PRθ= PR

  21. Discussion • For given capacities solve the SMDP with QL • Model other LTC complexities: • different facilities and room accommodations, • client choice and • level of care

  22. Thank you for your attention • Questions?

  23. Neural Network for Q(s,a)

More Related