Scenario Trees and Metaheuristics for Stochastic Inventory Routing Problems

1. Scenario Trees and Metaheuristics forStochastic Inventory Routing Problems Lars Magnus Hvattum • Norwegian University of Science and Technology, Trondheim, Norway • Arne Løkketangen • Molde University College, Molde, Norway • Gilbert Laporte • HEC, Montréal, Canada DOMinant Workshop, Molde, Norway, September 21.-22., 2009

2. Outline of thepresentation

3. Inventory Routing Problems extend the VRP, and look at a larger part of a supply chain Supplier Customer VRP IRP

4. The dynamicaspect of inventoryroutingcan be handled in variousways 1) Single Periodmodels 2) Multi-Periodmodels 3) Infinitehorizonmodels (...) (...) Kleywegt, Nori, and Savelsbergh (2002, 2004) Adelman (2004)

5. The problem is modelled as a MarkovDecisionProcess, whereeachepochcorresponds to a day

6. The problem is modelled as a MarkovDecisionProcess, whereeachepochcorresponds to a day

7. The infinitehorizondiscountedrewardMarkovDecision Problem canonly be solved for tinyinstances (...) A large number of states: all possibleinventorylevels A large number of actions: all possibledeliverypatterns A large number of transitions: all possibledemandrealizations (...) (...)

8. Previoussolutionmethods for theseSIRPsarebasedonapproximatingthevaluefunctions Standard algorithms for MDPscansolvesomeinstances: 4 customers,1 vehicle, and 9 inventorylevels 5 customers, 5 vehicles, 5 inventorylevels, butonlydirectdelivery Kleywegt et al. proposeapproximations of thevaluefunction for instanceswithdirectdelivery or at most 3 deliveries per route Adelmanproposesapproximations of thevaluefunction for instanceswith an unlimitednumber of vehicles

9. To simplify, weonlylook at finite scenario trees to generate a stochastic policy The infinitehorizon is approximated by a finitetree The large number of transitions is approximated by sampledrealizations The action spaceremainsunchanged State transitionsarebasedoncurrentinventorylevel, deliveredquantities and sampleddemandrealizations

10. The problem of finding optimal actions conditional on the sampled tree is formalized as an integer program

11. We have examinedthreemethods to solvethe scenario tree problem (STP) CPLEX Solve IP Solve MIP, whereintegralityconstraintsarekeptonly for root node GRASP Constructsolutions to the STP in a randomized adaptive fashion, butwithaddedlearningmechanisms Progressive HedgingAlgorithm Decompose problem over scenarios, solveeach scenario usingmodified GRASP Graduallyenforce an implementablesolution by penalties, including a quadratic term

12. In GRASP, we start with a solutionwithoutanydeliveries, thenadd more deliveriesif profitable u = 0,2,0,0 y = 0,0,0,0 No deliveriesscheduled in any node d = 1,1,2,2 d = 1,2,2,1 u = 0,1,0,0 u = 0,0,0,0 y = 0,0,0,0 y = 0,0,0,0 (...) (...) (...) (...)

13. In GRASP, we start with a solutionwithoutanydeliveries, thenadd more deliveriesif profitable u = 0,2,0,0 y = 0,0,2,0 First iteration: add 2 units of delivery to customer 3 usingvehicle 1 in node 1 d = 1,1,2,2 d = 1,2,2,1 u = 0,1,0,0 u = 0,0,0,0 y = 0,0,0,0 y = 0,0,0,0 No increase in inventory, butreduction in stock-outs (...) (...) (...) (...)

14. In GRASP, we start with a solutionwithoutanydeliveries, thenadd more deliveriesif profitable u = 0,2,0,0 y = 0,0,2,2 Seconditeration: add 2 units of delivery to customer 4 usingvehicle 1 in node 1 d = 1,1,2,2 d = 1,2,2,1 u = 0,1,0,0 u = 0,0,0,1 y = 0,0,0,0 y = 0,0,0,0 Increase in inventory in node 3 for customer 4 (...) (...) (...) (...)

15. In GRASP, we start with a solutionwithoutanydeliveries, thenadd more deliveriesif profitable u = 0,2,0,0 y = 0,0,2,2 Continue making insertions: yadditionalunits to customeriusingvehiclek in node v d = 1,1,2,2 d = 1,2,2,1 u = 0,1,0,0 u = 0,0,0,1 y = 0,0,2,0 y = 0,0,0,0 Stopwhenno profitable, feasibleinsertionscan be made (...) (...) (...) (...)

16. Severalvariations of the GRASP areexamined, to find a robust versionthatcan handle different types of instances RestrictedCandidate List (RCL) basedoneithervalue or rank [former is more robust] Size of RCL is controlled by a parameter that is adjusteddynamically[more robust than a fixedvalue] Buildsolution node by node (recursively) or all nodes simultaneously[former is much faster] Uselearningbasedonanalysingcompletedsolutions to findpotentialimprovements [increasesrobustness of buildingsolution node by node]

17. The progressive hedgingalgorithm is basedondecomposingthe problem over scenarios

18. Single scenarios aresolvedseparately, untilimplementability is enforcedthroughpenalties Each scenario is solvedusing GRASP Objectivefunction is modified to penalizedeviations from an averagedsolution:

19. Severalvariations of the progressive hedgingalgorithmareexamined, to find a robust versionthatcan handle different types of instances A progressive hedgingheuristic is employed to generatedfeasiblesolutions to the original scenario tree problem [ensuresthatgoodfeasiblesolutionsarefoundevenifthemethod has insufficient time to converge] Penalty parameter is updateddynamically [betterbalance of progress towards an implementablesolution] Use multiple penalty parameters and weights [betterthanusing a single parameter] Useintermediateheuristicsolutions for guiding thesearch [does not work, quickerconvergencebut to worsesolutions] Lock variables for whichconcensusseems to have beenreached [does not work, inducescyclingbehavior?]

20. The search of the resulting progressive hedging algorithm has a fairly similar behavior across instances Left: theaveragedsolutiongives a heuristicupperbound, and the progressive hedgingheuristicgivesactuallowerbounds Right: wecanmeasurethedistances in thesolutionspace and the parameter spacebetweeniterations, as well as thedynamicpenalty parameter

21. Therearedifferentways of controllingthecomputationaleffort used by themethods Increasingtheeffortgivesimprovedresults Right: profit as a function of thenumber of GRASP iterations per epoch

22. The size of the scenario trees is crucialboth for thecomputational time and thesimulationresults Increasingthesize of the scenario treesincreasesthecomputationaleffort as well as theprofitsobserved

23. Several scenario tree problems werestudiedseparately to studythe GRASP, the PHA, and theothermethods

24. Simulationsare run over manyepochs (600), and quickheuristics for the scenario tree problems must be selected Ourmethods: no initial time required, butshouldallocatesome time for thedaily problem (time consumingwhenevaluating, but ok in practice) Othermethods: higheffortinitially (days), butfairlyquick for thedaily problem (ok whenevaluating, practicerequires a stable situation)

25. GRASP is quickerthan PHA, but PHA is betteronsomeinstances (producessolutionswithdifferentstructure)

26. Some of theobservationsmade during thework have potential for leading to futureresearch 1) Scenario treesare used to representthestochastic and dynamicaspects: how to incorporatethese in the most efficientway? 2) Scenario tree problems must be solved to generatedecisions: how is this best done?

27. Scenario treesare not often used in (meta-) heuristics, and severalquestionsremain as to howtheyshould be generated Wegeneratethemusingrandom sampling to cover a specifiedtreestructure The size of thetree is determined by specifyingthebranchingfactors for eachlevel of thetree

28. Canwe save computational time by using a clevertreestructure? Weuselowerbranchingfactorslower in thetree (more important to representstochasticitythat is close in time?) Weshouldvarythedepth/width of thetreebasedontheinstancesolved? Weusethe same length for every scenario Wehad to limit thesize of thetree to be able to evaluatethemethodswithsimulations

29. Wecanimproveresults for a given treesize by sampling differently? Weuserandom sampling: by usinglargertreeswegetcloser to the true distribution Wealsotried a moment matching method, but a requirement of theimplementationwasthat for a single parent node, thenumber of children must be at leastthenumber of customers (distributions) Research question: canwedetermine a suitableobjectivefunction to be used by a metaheuristicthatconstructs scenario trees?

30. We have tested GRASP and PHA for solvingthe scenario tree problems Localsearchbasedmethodsfrequentlyperformbetter and faster than GRASP Problem: difficult to findlocalmoves in a scenario tree, as theinterconnectedness of decisionscreatesfeasibilityissues (how to find a suitablemoveevaluation? how to guide thesearch back to feasiblespaceifallowingcapacity/inventoryviolations?)

31. One idea is to hybridizelocalsearch and constructionheuristics Do localsearchmovesonly in this part of thesolution (representation) To evaluateeachmove, this part of thesolution must be completedusing a constructionheuristic Solution: (Root node) (Rest of tree)

32. Concludingremarks Stochastic and dynamic problems maybecomeincreasinglyimportant (withbettertechnology and access to data) The stochasticinventoryrouting problem is an interestingplayground for testing howonecandealwithstochastic and dynamic problems Using scenario trees to representstochasticity is relativelyuntested in combination withmetaheuristics Severaldirections for futureresearch have beenfoundbut not yetpursued