DRAFT

AssessingStaffingNeedsforaSoftware

GiulianoAntoniol,AnielloCimitile,

GiuseppeA.DiLucca,MassimilianoDiPenta

DRAdipenta@unisannio.itBenevento,Italy

October17,2003

antoniol@ieee.org,cimitile@unisannio.it,dilucca@unisannio.it,

RCOST-ResearchCentreonSoftwareTechnology

UniversityofSannio,DepartmentofEngineering-PiazzaRoma,I-82100

FTSimulation

DRAFT

MaintenanceProjectthroughQueueing

2

Abstract

Wepresentanapproachbasedonqueueingtheoryandstochasticsimulationtohelpplanning,managingandcontrollingtheprojectstaffingandtheresultingservicelevelindistributedmulti-phasemaintenanceprocesses.

DatafromaY2KmassivemaintenanceinterventiononalargeCOBOL/JCLfinancialsoftwaresystemwereusedtosimulateandstudydifferentservicecenterconfigurationsforageographicallydistributedsoftwaremaintenanceproject.Inparticular,amonolithicconfigurationcorrespondingtothecustomer’spoint-of-viewandmorefine-grainedconfigurations,accountingfordifferentprocessphasesaswellasforrework,werestudied.

Thequeueingtheoryandstochasticsimulationprovidedameanstoassessstaffing,evaluateservicelevel,andassessthelikelihoodtomeettheprojectdeadlinewhileexecutingtheproject.Itturnedouttobe

aneffectivestaffingtoolformanagers,providedthatitiscomplementedwithotherproject-managementtools,inordertoprioritizeactivities,avoidconflictsandchecktheavailabilityofresources.

IndexTerms

softwaremaintenancestaffing,softwareprocessmanagement,discrete-eventsimulation,processmodeling,processsimulation,queueingtheory,scheduleestimation

DRAI.INTRODUCTION

projectlifespan.

levelasperceivedbycustomers.

October17,2003

Costandeffortestimationsareimportantaspectsofthemanagementofsoftwaremaintenance

projects.Softwaredevelopmentandmaintenancearelabor-intensiveactivities;theprojectcostisstrictlytiedtotherequiredeffort,i.e.,totheprojectstaffing.Intoday’scompetitivemarket,acompromisehastobefoundbetweentheservicelevelsexperiencedbythecustomersandestimationisnotaone-timeactivityatprojectinitiation.Initialestimatesshouldberefinedtheprojectcosts;sometimes,increasingtheprojectstaffingmaynotbeconvenient.Cost/effortthroughoutaproject[1],verifyingeachtimethelikelihoodtomeettheestablisheddeadline.

Thus,itisessentialtotracktheeffortspent,andre-estimatestaffinglevelthroughouttheentireTraditionaltoolssuchasPERT,CPM,Ganttdiagramsandearnedvalueanalysishelptoplan

andtracktheprojectactivities.However,theyplayalittleroletoassessthelikelihoodofmeetingtheprojectdeadline,tostaffmaintenanceprojects,tohelppredictingandtrackingtheservice

FTDRAFT

3

Weproposeanapproachbasedonqueueingtheoryandstochasticsimulationtodealwiththestaffing,theprocessmanagementandtheservicelevelevaluationofcooperativedistributedmaintenanceprojects.Themethodhasbeenvalidatedondatafromamassive1Y2KmaintenanceinterventiononalargeCOBOL/JCLfinancialsoftwaresystemofaEuropeanfirm.Upto80peoplewereassignedtotheproject,whichspannedacrossfourmaintenancecenters;maintainerswereorganizedintoteams;maintenanceteamsoperatedbothatthecustomersiteandhomesite.Allmaintenancecenterscooperatedunderthecoordinationofacontrolboard.Themainrespon-suitableworkload.

sibilityofthecontrolboardwastoavoidconflicts,assigningtoeachmaintenancecenter/teamaQueueingtheoryisawell-consolidatedfield.Earlystudiesonqueueingtheorydatebackto1900[3].Thetheoryhasbeensuccessfullyappliedtoalargevarietyofproblems:telephoneswitching/networkdesign,partrepairandmaintenancecenterstaffing,merchandisedistributionand,moregenerally,servicecentermanagement.Traditionalqueueingtheorycanbeadoptedtohelpprojectstaffing,restaffingandservicelevelevaluationatdifferentdates.Incomingmaintenancerequestsmaybethoughtofascustomersqueueingupatthesupermarketcheckoutcounters.Thetimespentinthequeueisrelatedtothecustomerarrivalrateandtothetypeofserviceobtained.Theperceivedservicelevels,measuredasqueueingsystemwaitingtimes,areobviouslyrelatedtothenumberofcounters,tothedistributionoftheservicetimes(i.e.,Increasingthenumberofcounters,alsocalledservers,decreasesthetimespenttoobtainaservice.However,increasingthenumberofserversmaynotbeeconomicallyconvenient:acompromisebetweencostsandcustomersatisfactionhastobepursued.

Queueingnetworkmodelsrepresenting,atdifferentlevelsofgranularity,themaintenanceprocesseswereconsidered.Amonolithicconfigurationcorrespondstothecustomer’scoarse-thenumberofitemseachcustomeracquires)andtotheconfigurationoftheservicecenters.

DRA1

grainedpoint-of-view,whilefine-grainedmodelsexplicitlyaccountfordifferentmaintenanceIntheproposedapproach,stochasticsimulationsofqueueingnetworkspermitafine-grained

processphases,projectdistributionacrossdifferentmaintenancecentersandreworkactivities.

evaluationofstaffinglevelsattheprojectstartupandclose-downwhenstatisticalequilibrium

Theterm“massive”isreferredtothosemaintenanceinterventions,suchasY2Kremediation,Euroconversionorphone

numberingchange,involvingalargenumberofapplicationssimultaneously[2].

October17,2003

FTDRAFT

4

(Seeequation(1)inSectionIII-A)isnotachieved.Theapproachalsoprovidesameanstoassessthelikelihoodofmeetingtheprojectdeadlineandbalancingtheworkloadbetweenmaintenancecenterswhileexecutingtheproject.Differentservicecenterconfigurationsmayexhibitdifferentperformancelevelsandmayincrease(withoutchangingthenumberofservers)theaverageperceivedservicelevel,aswellasincreasethelikelihoodofmeetingtheprojectdeadline.Thefeaturesdescribedabovecombinedwiththepossibilitytoviewthemaintenanceprocessfromaqueueingnetworkperspective(thusdeterminingtheoptimalnumberofservantsorrework),maketheproposedapproachavalidcomplementtothepreviouslymentionedmainstreamtechniques(PERT,CPM,etc.).

Thispaperextendsandcomplementspreviousworks[4],[5]withanewapproachtodealwithqueueingnetworks,representingdifferentprocessphasesormaintenanceteamallocations.Furthermore,byapplyingstochasticsimulation,statisticalequilibriumisnolongerrequired,thusallowinganevaluationofstaffinglevelsattheprojectstartupandclose-down,andperformdynamicrestaffing.Giventheassumptionsmade(detailedinSectionV-A),theproposedapproachisimmediatelyapplicabletomassivemaintenanceinterventions,suchascurrencyorphonenumberingchange,andeasilyextensibletomoresophisticatedmaintenanceprocesses.foreachmaintenancephase,aswellasconsideringtheeffectofphenomenasuchasabandon

DRAoffutureworks.

II.RELATEDWORK

forecastingmaintenanceresourcesandmanagingcosts.

October17,2003

Theremainderofthepaperisorganizedasfollows.Afteranoverviewontherelatedwork

(SectionII),basicnotionsonqueueingtheoryandstochasticsimulationaresummarizedinSectionIIIforthesakeofcompleteness.Then,SectionIVintroducestheproposedqueueingthecasestudymethodologyandtheadoptedtools.CasestudyresultsarepresentedinSectionVI.systemmodels.SectionVpresentsthecasestudydescription,theresearchquestionstoaddress,

Thelastsectionsarededicatedtoasummaryoflessonslearned,papercontributionsandoutline

Inthepast,severalstudiesattemptedtodrawrelationsbetweenproduct/processmetrics/analysis

andmaintenanceprocessimprovement(e.g.,[6],[7])orproductintrinsiccharacteristics(e.g.,

[8],[9],[10]),aimingatenhancingcustomersatisfactionandreducingcosts.Theseapproachesweremorefocusedonprocessorproductimprovementthandefining/assessingthestaffing,

FTDRAFT

5

In1981andthenin2000,B.W.Boehm[11],[12]proposedderivingthetestingeffortandtheannualmaintenancecostfromthedevelopmenteffort.TheCOnstructiveCOstMOdel(COCOMO)isfoundedonempiricalformulaeforsoftwarecostestimation;themodelusedinforwardengineeringisintegratedwiththeAnnualChangeTraffic(ACT)parametertoestimate,atthemacrolevel,maintenancecosts.Wieneretal.[13]presentedaformalmodel,basedontheRayleighmodel,toestimatetheneededmanpowerforprojectsintheBankTrustCompanydomain.

planningwhen/howtheywillbeservedtoreducetotaleffortandcosts.BriandandBasili[14]proposedamodelingapproach,basedonstatisticaltechniques,tosupporttheclassificationtask.LiteraturereportsseveralworksapplyingProcessSimulationModelingtoplan,manageandcontrolsoftwarelife-cycleactivities.Kellneretal.[15]describemotivationsandpurposesofsoftwaredevelopmentprocessmodelingandsimulation.Theyhighlightthefactorscharacterizingsimulationandclassifythemaintechniquesofsimulationandmodeling.AswillbedescribedinSectionIII,softwareprocesssimulationisusuallycarriedoutbyusingsystemdynamicsanddiscretemodelingparadigms.In[16]MartinandRaffo,toovercomesomeofthedisadvantagesofthetwoparadigms,proposeacombinedmodelthatrepresentsthesoftwaredevelopmentprocessasaseriesofdiscreteprocessstepsexecutedinacontinuouslyvaryingprojectenvironment.IssuesconnectedtoempiricalstudiesofsoftwareprocesssimulationmodelingarediscussedAnimportantissuewhenfacingmaintenancerequestsisthatofclassifyingthemtoimprove

DRAsimulationmodeloutputs.

October17,2003

in[17],withparticularreferencetotheestimateofsimulationparametersfromreal-worlddata,andtocompareactualresultstothemodel’sresults.Thepaperdiscussesseveralstatisticalandanalyticaltechniquestochooseamongvariousprocessalternatives,aswellasexamplesofhowutilityfunctions,financialmeasuresanddataenvelopmentanalysiscouldbeusedtoevaluateAnimportantcontributiontotheliteratureofprocesssimulationhasbeengivenbyAbdel-

Hamid[18],[19],[20].In[18]Abdel-Hamidpublishedanapproachbasedonsystemdynamicsmodelingtosimulatethestaffingofrealworldsoftwareprojects;themodelhasbeenusedtoverifythedegreeofinterchangeabilityofmenandmonths[21].ResultsfromtheanalyzedcasestudydonotfullysupportBrooks’law.Ahybridestimationmodelisproposedin[20]:themodelavarietyofalgorithmicestimators.Thisapproach,accordingtotheauthor,canbeusedtoallow

DRAFT

exploitsthesystemdynamicssimulatorofsoftwaredevelopmentpresentedin[19],coupledwith

FT6

continuous-timemodeling/estimation.Inparticular,astheprojectprogresses,themodelvariablescanbeadjustedinrealtimetoreflectimprovedknowledgeofthemanagement,whileatprojectcompletion,themodelcanbeusedforpost-mortemanalysis.

In[22]softwaremaintenancemanagementstrategiesimplementedbysomeorganizationsaredescribedandcomparedontheattributesofperformance,economics,efficiencyandend-productquality.Subsequently,somemeasurementsaredefinedforeachattribute.Itisthentheresponsibilityofthemanagertochoose/definethemostsuitablesubsetofattributesthatsupportAnimportantissueaboutanypredictionmodelistheaccuracyofitsestimates.Jorgensen[23]reportsanexperiencefromthedevelopmentanduseofelevensoftwaremaintenanceeffortpredictionmodels.Inthisstudy,themostaccuratepredictionswereobtainedbyapplyingmodelsbasedonmultipleregressionanalysisandonpatternrecognition.

Host[24],publishedastudythatshowshowdiscrete-eventsimulationcanbeusedtoexploreoverloadconditions.Inparticular,asoftwarerequirementsmanagementprocessforpackagedsoftwaresystemswasanalyzed.Thesimulationmodelwascreatedaccordingtoapreviousstudyoftheprocessandthesimulationparameterswereestimatedfrominterviewswithprocessexperts.

his/herownstrategy.

DRAdifferentprocessstrategiesratherthanstaffingthesystem.

October17,2003

TheideaofapplyingqueueingtheorytosoftwaremaintenancewasproposedbyPapapanagio-

takisandBreuerin[25].Theyproposedamodelforthemanagementofsoftwaremaintenancebasedonqueueingnetworks.Themodelaccountedfortechnicalaspectsofthesoftwaremainte-nanceprocess,formetricsobtainedontheapplicationand,finally,fortheorganization’sresourcesandrequirements.Althoughtheydevelopedatooltosimulatethebehaviorofaqueueing-basedprocessandtosupportthemanagement,nodataaboutapplicationtorealprojectswasreported.QueueingtheorywasrecentlyappliedbyRamaswamy[26]tomodelsoftwaremaintenance

requests.Asimilarapproach,tomodelaweb-centricmaintenancecenter,wasdescribedin[5].Byadoptingafastlaneapproach,authorswereabletoimprovemaintenancecenterperformances,

asexperiencedbythemajorityofcustomers.Commonalitiescanbefoundwiththesetwo

works.However,theresearchquestionsaddressedherearedifferent.SimulationsofasoftwaremaintenanceprocesswereperformedbyPodnarandMikacin[27]withthepurposeofevaluatingMostoftheworkdescribedabovedealswiththeproblemofcostandeffortestimation;the

DRAFT

FT7

problemofmeetingtheprojectdeadlineisseldomconsidered.Thelattermayrepresentacriticalissueformassivemaintenanceinterventions,oftenaddressedover-staffingtheproject(andthusincreasingthecosts).Theseissues,withsomeanalysesreportedinthepresentpaper(relatedtothesimplestmodels)wereintroducedin[4].

III.BACKGROUNDNOTIONS

Inthissectionbackgroundnotionsaboutqueueingtheory,queueingnetworksandstatistical

A.Queueingtheorynotions

λ=1/tαC 2αtαDRAtrFig.1.Queueingmodelparameters

Aqueueingsystemcanbedescribedascustomersarrivingforservice,waitingforserviceif

itisnotimmediate,andleavingthesystemafterbeingserved(byservers).Thetermcustomerisusedinageneralsenseanddoesnotnecessarilyimplyahumancustomer(e.g.,maintenancerequestscanbethoughtofascustomers).Aqueueingsystemmodelsthesteadystateoftheparametersare:

process,whiletransientsituationsarenottakenintoaccount.Theobservablequeueingsystem

1)Arrivaltrafficrate:theaveragerateofcustomersenteringthequeueingsystem,

theinterarrivaltime

),and,ifavailable,itsstatisticaldistribution;

2)Servicetime:thetimeaserverneedstoprocessarequest,distribution;

October17,2003

FTSERVERStsCs2simulationswillbesummarized.Furtherdetailscanbefoundin[3].

QUEUEtsCs2tsCs2twts(and/or

,and,ifavailable,itsstatistical

DRAFT

8

3)Queuecapacity:finiteorinfinite;and

4)Queuediscipline:FIFO,LIFO,random,withpriority,etc.

WithreferencetoFig.1,aqueuebookkeepingmayregisterthedistributionandtheaveragevaluesofwaitingtimes(),thetimesspentbycustomersinthesystem,(alsocalledresponse

times-

)and,finally,theaveragenumberanddistributionofcustomersinthesystemandin

thequeue.

Asashorthandfordescribingqueueingphenomena,anotationhasbeendefined;aqueueingarrivaltimedistribution,

istheservicetimedistribution,

capacityofqueue(infiniteifomitted),and

and

maybeMarkoviandistributions(

butions(

),orgeneraldistributions().TheErlangdistributionisaGammadistributionwith

positiveintegerparameter,anditbecomesanexponentialdistributionifisthenumberofstagescomposingtheErlangprocess.

FTsystemisdescribedbyaseriesofsymbolsandslashes

,where

istheinter-isthenumberofservers,isthe

isthequeuediscipline(FIFOifomitted).

),deterministicdistributions(),Erlangdistri-[3],where

Often,a-prioristatisticalknowledgeoftheprocessislimited;theselectionofinter-arrivalandservicetimedistributionfamiliesmaybeguidedbythesquareofthecoefficientofvariation

DRA,definedastheratioofthestandarddeviationtothemean.Availableliterature[3]suggests

anassumptionofdeterministicdistributionwhen

,anErlangdistributionwhen

,andanexponentialdistributionwhen

.Valuesof

requirea

generaldistribution.Notethat,if

,anexponentialdistributionmaybeapplied,givingrisetoanoverly

conservativeestimate.Thecasewhere1modelsclusterarrivals,suchastrainarrivals.

Goodness-of-fittestsarerequiredtosupportandvalidatetheassumptionsaboutthechosendistributionfamily.

Tomeasurequeueingsystemperformance,severalparametersmaybeconsidered;forexample,

theaveragetimespentbyacustomerwaitingforaserver(i.e.,

),theaveragequeuelength,the

averagenumberofbusyservers,theserverusecoefficient(i.e.,,seebelow),ortheprobability

thatallserversarebusy.Theserverusecoefficientisrelatedtothesystemstatistical

equilibrium:

October17,2003DRAFT

9

FT(3)

where

isthewaitingtimeaspredictedbythe

modeland

isthesquareof

DRAthecoefficientofvariationofservicetimes.B.QueueingNetworksmodel)isoftenfeasible.

statesthatinanetwork,composedof

Fig.1representsaqueueingmodelataveryhighlevelofabstraction:incomingrequests

areenqueuedandprocessedbyoneormoreservers.Morecomplexmodels,basedonqueueingQueueingnetworkanalysisiscomplexand,inmostcases(especiallywhenservicetimesare

networks[3],arerequiredtoobtaindetailedinformationondifferentprocessphases.

notexponentiallydistributedandthemodelisnottrivial),itrequiresasimulation.However,apreliminaryconservativeanalysisassumingexponentialdistributions(forallthenodesoftheQueueingnetworkparametersaredeterminedbyapplyingJackson’stheorem[3].Thetheorem

nodes,eachonecharacterizedbyparameters

(number

ofservers),(averageservicetime,exponentiallydistributed),and

(incomingarrivalrate,

October17,2003DRAFT

10

Requests exiting from the systemwRequests incoming from outside the system1− ΣpkjΛj=lkλkNk....plkΛlpklΛkRequests dispatchedto other nodes...pwkΛRequests incomingfrom other nodespkwΛwkFig.2.Queueingnetworknode

seeFig.2),arequestmayflowtonodethesystemwithprobability:

FT(

)withprobability

,ormayexitfrom

(4)

Thismeansthat,forthe

node,thetotalarrivalrateis

DRA(5)

Therefore,onceassigned

(trafficincomingfromoutsidethesystem),andtransitionprob-

abilities

,

canbedeterminedbysolvingasystemoflinearequations.Itisworthnoting

thattheJackson’stheoremappliesonlyifthesystemisatstatisticalequilibrium:evenif,duetothetaskperformedortodifferentteams’expertise,differentserversmayexhibitdifferent

performances,thenumberofserversischosentoensureflowbalance:theexitrateofanodeis

equaltoitsincomingrate.C.Simulation

Aspreviouslystated,aqueueingsystemonlymodelsthesteadystate.Therefore,toanalyze

transientanddynamicbehavior,asimulationisrequired.Simulationmodelsnon-stationaryrules(e.g.,reworks,routingproblems,etc.).

October17,2003

processes(orprocessesnotfollowingaknowndistribution),aswellascomplexdynamicsand

DRAFT

11

Simulationattemptstobuildamodelthatwillmimicarealsystemformostofitsrelevantaspects.Asimulationmaybedeterministicorstochastic.Inthefirstcase,thebehavioris“point-wise”definedbytheoutputofthemodel(apointestimateismadeforeachofthevariablesofinterest)andresultsarerepeatable.Stochasticsimulationinvolvesrandomness:multiplerunsmaygeneratedifferentvalues.Moreover,asimulationmaybestaticordynamic,dependinguponwhetherornotitinvolvestime.

Finally,simulationmaybeclassifiedassystemdynamicssimulation,discrete-eventsimula-describetheinteractionbetweenseveralprojectfactors,torepresentsituationssuchasfeedbackloops,etc.Discrete-eventsimulationdescribesprocesssteps,andiswellsuitedtomodelqueueingsystemssuchasthosedescribedinthispaper.Stochasticstate-basedsimulationmodelsexplicitlycaptureprocess-leveldetails,includingcomplexinterdependenciesamongprocesscomponents.Discrete-eventsimulationmaybeapproachedinthreedifferentways[29]:

Eventscheduling:theideaistomovealongthetimescaleuntilaneventoccursandthen,dependingonthenatureoftheevent(arrivalofarequest,finishingofaprocessphase,etc.),modifythesystemstate,andpossiblyschedulenewevents;

DRAActivityscanning:astimepasses,theactivitiesarecontinuouslymonitored,forthesatis-

factionofaparticularactivity,namedaconditionalactivity[29].SuchanactivityoccursProcess-oriented:thequeueingsimulatorprovidesthreeessentialcomponents:asourcethatgeneratescustomerarrivals,aqueuetoholdwaitingcustomers,andafacilitytoservecustomers.Onceprocess-orientedcomponentsareavailableasbuildingblocks,themodelerneedonlyprovideprocessparameters.

whentherearerequestsinqueueandatleastoneserverisidle;or

Thispaperaddressestheanalysisofthedynamicbehaviorofqueueingnetworksviastochastic,

dynamic,discrete-eventsimulations.Thesimulatorhasbeenimplemented(seeSectionV-E)

adoptingaprocess-orientedapproach.Thechoiceofthisapproachwasmotivatedbydesignconsiderations,i.e.,wedecidedtocreatealibraryofclassesrepresentingthebasicbuildingblocksofthequeueingsystem.

October17,2003

FTtion[15],[16]andstate-basedsimulation[28].Systemdynamicssimulationiswell-suitedto

DRAFT

12

IV.THEMODEL

Tomodelamaintenanceprocessusingqueues,differentlevelsofabstraction,correspondingtodifferentgranularityandapproximationlevelscanbeconsidered:

Inthesimplestmodel(Fig.1),maintenanceactivitiesandcentersarenotdistinguished;asinglequeue(i.e.,node)modelsthesystem;

Atafinerlevelofdetail,differentmaintenancephasesaremodeledbydifferentnodes,assigningresources(i.e.,servers)toeachmodeledactivity(Fig.3-a);

(Fig.3-b);and

Finally,thepossibilityofreworkamongdifferentmaintenancephasesisconsidered(Fig.3-c).

Thesingle-queuemodelmayberegardedasthecustomerview:requestsenterthesystemand,eventually,amaintenanceactivityisbilled;suchacoarse-grainedviewmaybeextremelyusefultoeasilyassesstheoverallprobabilityofsuccesswithatractablemathematicalmodel.However,formorecomplextopologies,stochasticsimulationisneeded.

Fig.3-ashowsahighlevelviewofamulti-center,multi-stagemaintenanceprocess,modeled

DRAof

byacascadeofqueues.Eachstagemayinvolveseveralmaintenancecenterswithdifferent

requestsperday,andareprocessedsequentially(i.e.,FIFO)byoneormorenodes2.Each

numbersofassignedprogrammers.Maintenancerequestsenterthesystemwithaconstantrate

node,composedofaqueueandoneormoreservers,representsaphaseofthewholemaintenanceprocess.Thereisnolimitationonthegeographicallocalizationoftheserversinthatthemodelabstractsunnecessarydetails.Communicationandcoordinationactivitiesareaccountedforbytheestimatedmodelparameters.Thiscorrespondstothesoftwaremaintenancecompanyinternalview:requestsenterthesystem,undergoasequenceofactivities,andfinallyleavethesystem.Fig.3-bshowsamodelwherearequestmayexitafteracertainnodeofthe“chain”.Consider,

forexample,acomplexsystem:theremayexistcomponentsthatdonotneedanymaintenancemodification.

2

interventions.Thus,afterthecomponentshavebeenanalyzed,theyleavethesystemwithout

Ifthearrivalrate

isnotconstant,simulationisnecessarytostudysystembehavior.

DRAFT

October17,2003

FTThepossibilitythatarequestmayleavethesystemafteracertainphaseistakenintoaccount

13

Stage 1λλStage 2λStage nλts1C2s1ts2C2s2a)tsnC2snStage 1λλp1,2Stage n−1n−2Stage nλpΠj=1j,j+1n−2ts1Cs12λ(1−p )1,2Stage 1λDRAts1Cs12Fig.3.a)Queueingmodelofamultistage,multicentermaintenanceprocess;b)Modelwithexitfromthesystem;c)Model

withrework

Finally,Fig.3-cshowsanexampleofasysteminwhichthereisthepossibilityofrework:

somephasesmaybeperformedmorethanonce.Inamaintenanceprocess,forexample,after

unittesting,onemayrealizethatitisnecessarytore-modifythecode.Inthiscase,requeststhatenterintoeachnodemayarrivefromthepreviousnode(orfromoutsidethesystemforthefirstnode)orfromoneofthesubsequentnodes.Arrivalrates,asstatedinSectionIII-BandalsoshowninFig.2,havebeencomputedusingtheJackson’stheorem.Forsakeofsimplicity,formulaehavebeenomittedinFig.3-c.

October17,2003

FTtsn−1Csn−12λpΠj=1j,j+1tsnCsn2pλ (1− )j,j+1n−2Πj=1b)Stage 2Stage nts2Cs22tsnCsn2c)DRAFT

14

Maintenancecentersmaybedescribedbyacombinationof

models.However,while

theoreticalas-inthecasestudyreportedhere,requestinter-arrivaltimesmaybedescribedbyexponentialprobabilitydistribution,servicetimessometimesgrosslydeviatefrom

sumptions.Inotherwords,when

,the

modelwasreplacedbyan.

In

,thegeneraldistribution

accountsfortheincreaseinwaitingtimesduetohigher

servicetimevariability.

V.CASESTUDY

Thecasestudyreportedhereisamassivemaintenanceproject,relatedtofixingtheY2KprobleminalargefinancialsoftwaresystemofaEuropeanbusinessfirm.TheprojectstartedonJanuary2,1999andfinished(formally)onJanuary14,2000.ThecontractrequiredthatallmaintenanceactivitieswerecompletedbyDecember1999.levels:

Theproject,managedandcoordinatedbyaProductionManager,wasorganizedintothree

Area:anaggregationofoneormoreapplications,managedbyanAreaManager;

Application:asetoffunctionsrelatedtoaparticularpartofthebusiness,managedbyanApplicationLeader;eachapplicationiscomposedofoneormoreWorkPackets(WPs);(WPs):looselycoupled,elementaryunits(fromonetonineforeachapplication)subjecttomaintenanceactivities;eacharrivingWPwasmanagedbyaWPleaderandassignedtoamaintenanceteam(theaverageteamsizewasaboutfourprogrammers).

DRAThesystemwascomposedof84WPs,eachonecomposed,onaverage,of300COBOLand

JCLfiles.TheseWPswereproducedbytheInventoryphase(seebelow)fromJanuarytotheend

ofJuly,andallmaintenancedirectlyrelatedactivitiesforthoseWPsendedinOctober.AboutinthemaintenanceoftheseWPs.AsdetailedinSectionV-C,oneofthecaseobjectivesistotheprojectwithintheestablisheddeadline.

80people(i.e.,maintainers,managers,techniciansandsecretaries)wereinvolved(notfull-time)determinethenumberofpeoplethat,workingfull-time,wouldallowsuccessfulcompletionofTheprojectfollowedaphasedmaintenanceprocess(similartowhatdefinedin[30]),defined

insidethemaintenanceorganization,encompassingfivemacro-phases:

1)Inventory:dealswiththedecompositionoftheapplicationportfoliointoindependentapplications,andsuccessivedecompositionofeachapplicationintoWPs.Theactivitywas

DRAFT

October17,2003

FT15

performedinconjunctionwithdomainandapplicationexperts.ConsistencychecksensuredthatalltherequiredcomponentswereincludedineachWP.Forsuchalargeproject,theidentificationofWPsproceededincrementally,whileprocessingalreadyidentifiedWPs:thiscreated,asstatedinSectionVI-A,aPoissonprocessofincomingmaintenancerequests.Finally,duringtheInventoryphase,managerscarriedoutearlyeffortestimationandpreliminaryprojectstaffing.Theseactivitieswereperformedbyapplyinganalogy-basedeffortestimation[31],usinginformationgatheredduringtheassessmentphaseonEuroprojects.

about50%ofthe(previouslyarrived)WPs,andondataavailablefromsimilarY2Kand2)Assessment:identifies,foreachWP,candidateimpacteditems,usingautomatictools.Typically,astartingimpactsetisidentified(i.e.,programpointsdirectlyimpacted),andthenitisaugmentedbythesetofpointsimpactedbyrippleeffects[32];

3)TechnicalAnalysis(TA):dealswiththeanalysisofimpacteditemsandidentifiesacan-didatesolutionamongasetofpre-definedsolutionpatterns.Theidentifiedsolutionwasdocumentedbyaddingastandard-formatcommenttotheitem;

4)Enactment(Enact):anautomatictoolwasusedtoapplypatchestotheproblemsidentified

DRAtemplate.

October17,2003

andanalyzedinthepreviousphases.Thesolutionwasoftenbasedonwindowing[33];

5)UnitTesting(UT)wasperformedoneachimpactedWP;toolswereusedforautomaticgenerationoftestcases.Testingwaslimited,intheabsenceofrippleeffects,tothecodeisolationtesting,reducestestingcosts,inthatdriverscanbeconstructedfromapre-definedsurroundingtheimpactedpoint(extractedusingprogramslicing):thistechnique,called

Thefirsttwophaseswereperformedonthecustomersitesbyadedicatedteamofsenior

programmers.Theteamresponsibilitiesincludedinventory,assessment,workloaddispatchingandtofourdifferentsites.Thephasesdescribedabovewerefollowedbyintegrationanddeliveryrelatedactivities,whichwerenotsubjectofourcasestudy.InordertointegrateanddeliverthetheendofOctober.Conflictresolutionandschedulingwereundercontroloftheteammanager,systemwithinthecontractualdeadlines,itwasrequiredthatallWPsunderwentmaintenancebywhiletheApplicationLeaderhandleddependenciesamongfiles.Becausetheinventorywas

conflictresolution;alltheremainingactivitieswerecarriedoutbymaintenanceteamsassigned

FTDRAFT

16

preliminaryperformedoff-linewithrespecttotheproject,thatactivitywasnotincludedinthemodeledmaintenanceprocess.A.Assumptions

ThissectionsummarizesalltheassumptionsmadetomodelthesoftwaremaintenanceprojectdescribedaboveusingthequeueingmodelsdetailedinSectionIV.

AsstatedinSectionV-B,thispaperassumesrequestsgeneratedbyaPoissonstochasticprocess(assumptionsupportedbytestsdetailedinSectionV-D.1),anunlimitedqueuesizeandaFIFOqueuediscipline.However,differentsoftwaremaintenanceprojectsmayormaynotfollowtheaboveassumptions:theremaybebatcharrivals,ordifferentmaintenancerequestsmayhavedifferentpriorities.Detailsonhowqueueingmodelsaddressthesefactorsarebeyondthescopeofthispaper,andcanbefoundin[3].Limited-capacityqueueswerenotconsideredbecause,intheorganizationsubjectofourcasestudy,therewerenocasesinwhicharequestwasnotaccepted(andsentbacktothecustomerortothepreviousnode)becausethequeuewasfull.Aspreviouslyhighlighted,themaintenanceinterventionswerecarriedoutbyapplyingastandardizedprocedure;inmostcasesproblemidentification,patchapplicationandunittestingweresupportedbyautomatictools.Thefollowingassumptionweremade:

NoparticularprioritywasrequestedforanyWP.Theproject(excepttheintegrationandsystemtestingphases,notconsideredinourcasestudy)dealtwiththeapplicationodpatchestoCOBOLandJCLsourcesaffectedbytheY2Kproblem:integrationwasperformedonlyattheendoftheproject;

Becausetheinterventionwashighlystandardized,differentprogrammerskillsandexpe-riencesplayedlittle(orno)role.Themaintenanceprocesswasconsideredasapoolofincomingrequestsenqueuedanddispatchedtothefirstavailablemaintenanceteam,therebyjustifyingtheFIFOqueueingdiscipline;

Forthesamereason,andfollowingin[18],Brooks’lawwillbeassumednotapplyinthecasestudy.Itwas,infact,assumedthatinterchangeabilitybetweenmenandmonthsispossible,hence:

17

approximationwasconsideredreasonable.Furthermore,themodelcanbegeneralizedtocasesforwhichtheBrooks’lawdoesapply,e.g.,introducinganon-linearityfactorin(6).B.InstantiatedModels

All PhasesAssessmentTechnical Analysis/Enactment/Unit Testinga)b)AssessmentFig.4.Queueingnetworkmodelsinstantiatedinthecasestudy

ModelsdescribedinFig.3wereinstantiatedaccordingtotheadoptedmaintenanceprocess.

DRAInparticular,threedifferentmodelswereappliedtotheprocess:

Asingle-queuesystem(Fig.4-a);

Asystemcomposedofatandemoftwoqueues(Fig.4-b),onefortheAssessmentphaseandanotherforallotherphases(thismodeltakesintoaccountthegeographicaldistributionoftheproject);and

Asystemcomposedofthreenodes(Fig.4-c),oneforAssessment,oneforTechnicalAnalysisandanotherforEnactmentandUnitTesting.ItisworthnotingthatnotalltheonEnactmentandUnitTesting.

requestsgobeyondtheTechnicalAnalysisphase,andtheremaybeapercentageofrework

Thoughprojectdatarevealedanegligiblelevelofrework,investigationontheeffectsofdifferent

reworklevelsispresentedinSectionVI-C.C.ResearchQuestions

Queueingsystembasicparameterscanbeeitherestimatedbyanalogyonpastprojectsor

derivedfromthelogofanongoingactivity.However,astimepasses,availableinformation

DRAFT

October17,2003

FTReworkTechnical AnalysisEnactment/Unit TestingAbandonc)18

increases,andrefinedestimatescanbeobtained.Inotherwords,theresearchquestionsaddressedinthispaperaretosomeextenttimedependent,anddifferentmilestoneswereestablishedtotrackprojectevolution.Thefollowingresearchquestionswereinvestigated:

1)Howcanthecollectedinformationonongoingactivitiesbeusedtoreadjustprojectstaffing?(SectionsV-D.1andV-D.2,seeresultsinTableIV)2)Foragivenstaffinglevel,howaccuratearethe

estimatesobtainedbydifferentmodels,

e.g.,singlenodemodelcomparedtoasystemexplicitlymodelingthedifferentphases?3)Howdoesthelikelihoodofmeetingtheprojectdeadlinechangeastheprojectprogresses?(SectionV-D.3,resultsareplottedinFig.9)

4)Howdoreworkandabandonmentaffecttheresults?(ReworkeffectsareshowninSec-tionVI-C,whileabandonmenteffectsareshown,forthethree-queuemodels,inTableIVanddiscussedinSectionVI-B)And,finallyinSectionVI-E)D.TheMethod

5)Mayaperiodicrestaffingreducepersonnelcosts?(SectionsV-D.5,resultsarediscussed

DRAwereusedtotrainthemodelwhiletheremainingtoassessmodelaccuracy.

Anapproachinspiredbytheleave-one-outcross-validationprocedure[34]andbytimeseries

(past)points,thetrainingset,

analysis[35]wasusedtomeasuremodelperformances.Collecteddatapointswereconsideredasatimeserieswithapast,apresentandafuture.Thefirst

(future)points,thetestset,wereused

AssketchedinFig.5,eachcheckwasarticulatedinthefollowingsteps:1)Datadistributionanalysisandmodelparametersestimation;2)Projectstaffing;

3)Likelihoodofmeetingtheprojectdeadlineestimation;and

4)Testsetmodelassessment.

Inaddition,simulationwasperformed(seeSectionV-D.5)toanalyzethebehaviorofthemodel1)DataDistributionAnalysisandEstimationofModelParameters:Inordertoverifythatcase

incaseofperiodicalre-staffing.

studydata(interarrivaltimesandservicetimes)followastatisticalpattern,twotaskswerecarried

October17,2003

DRAFT

FT(SectionV-D.2,resultsareshowninFig.6-b,7and8)

19

Model ParametersContractualDeadlineModel ParametersRequiredwaiting timesEstimatedStaffingPARAMETERSESTIMATIONPassedDATADISTRIBUTIONANALYSISPROJECTSTAFFINGLIKELIHOODESTIMATIONLikelihoodof FinishingInformationFailedElapsedFinishingDeadlineEstimatedResponse timesand accuracyMODELASSESSMENTTEST SETDATAAccuracy ofDeadlineEstimationENDTRAINING SETDATAFig.5.Informationflowamongthemethod’ssteps

out.First,histogramsofthedatawereplottedandtheirshapesanalyzedtoselectacandidatedistribution.Then,theKolmogorov-Smirnov(KS)goodness-of-fittest[36]andtheChi-squaregoodness-of-fittest[37]wereperformed,toassessthesimilarityofthedatadistributionwiththecandidatedistribution.andthedistributionfunction

TheKStestcomputesthelargestdistance

DRAthatthedatafittheexpecteddistribution)isrejectedif

KStestvalues,aretabulated[36]).

TheChi-squaredtestmeasurestheerrorbetweenacandidatedistribution’sdensityfunction

,where

andthedatahistogram.Thetest(i.e.,thesamenullhypothesisastheKStest)isrejectedif

istheChi-squaredstatisticand

functionwith

degreesoffreedomandsignificancelevel.

Toselectthefamilyofdistributions,testsontheinterarrivaltimedistributionandontheservice

modeldoesnotrequireanyhypothesisonservicetimedistribution.

timedistributionswereperformed,forthedifferentnodesofthequeueingsystem.Itisworthnotingthatthe

Onceithasbeenverifiedthattheincomingdatafollowawell-knownstatisticalpattern,the

interarrivalratewasestimatedusingtheMaximumLikelihoodEstimator(MLE).Itcanbe

shownthat,forexponentialdistributions,theMLEisequaltothearithmeticmean.Servicetime

descriptivestatistics(meanandstandarddeviation)werecomputedoveratrainingset:according

October17,2003

FTbetweenacandidatedistributionfunction

(

computedfromthedata.Thetest(i.e.,thenullhypothesis

values,saidcritical

istheChi-squaredpercent-point

DRAFT

20

to[3],noparticularestimatorwasneededinthiscase.

2)Projectstaffing:Onceparameterswereestimated,stochasticsimulationwasusedtocomputeteamsizes(forthedifferentnodesofeachmodel)undertheconstrainttocompletemaintenanceactivities,onaverage,withinagivendate.Severalsimulationswerecarriedoutwithincreasingteamsizeforeachnodeofthemodel,untilalltheexpectedWPswereprocessedbythechosendeadline.Thenumberofservers(maintenanceteams3foreachnode),waschosensufficientlyhightoensurethataddingfurtherresourcesdidnotsubstantiallymodifythesimulatedprojectlife-span.

Projectstaffinglevelsarethenrefinedtoreachacompromisebetweenpersonnelcostandwaitingtime.Ifthenumberofteamspernode(servers)istoosmallthenthismayproducesysteminstability(therequestarrivalrateisgreaterthanrequestdispatchrate),whileincreasingthenumberofteamsincreasestheprojectcost.

Responsetimesevaluatedfromtrainingsetparameterswerecomparedwiththoseevaluatedfromthesamemodel,with

andservicetimesofthetestset,tomeasuretheerroroccurred

estimatingtheresponsetimes.

3)LikelihoodtomeettheProjectDeadlineEstimation:Theprojectstaffinglevelcannotbeconsideredwithouttakingintoaccounttherisksrelatedtoanexcessiveslacktimeandtherelatedcontractualpenalties.Themassivemaintenanceprojectwasdevotedtomodifymission/businesscriticalsoftware:remediationcostswerenotconsideredtheprincipalissue.

Thelikelihoodofmeetingtheprojectdeadlinewasestimatedbynumericalstochasticsim-ulationbasedontherefinedstaffinglevel(inthatitguaranteessmallresponsetimesandthestatisticalequilibriumofthesystem).Fortheexpectedworkloadandanygivenpossibleprojecttime-to-finish,inarangeofinterestaroundtherequiredprojectdeadline,iterativesimulations(thenumberofsimulationswasdoubledeachtime,untiltheresultsfortwosuccessiveiterationsmatched)wereperformedtocomputethelikelihoodastheratio:

21

actualarrivaldatesandservicetimes(testset).Inotherwords,adeterministicsimulationwasexecutedwithWPsbelongingtothetestset.

5)AnalysisofTransientandAdaptiveRestaffing:Themaindrawbackoftheprocessdescribedaboveisthatthestaffinglevelisconstantacrosstheentireproject,therefore:

Projectinitialandfinaltransientsarenothandled:althoughatthebeginningandattheendoftheproject,thenumberofWPstoprocessconcurrentlyisconsiderablylower,thestaffinglevelconsideredisthesameofthesteadystate;and

projectwhenevertherateofarrivalsandtheaverageservicetimesexceedcertainthresholds.Tostudytheeffectofperiodicalrestaffing,newsimulationswereperformedasfollows:

1)OnceasmallnumberofWPs(say10),necessaryforapreliminaryestimateofmodelparametersarrived,theprojectwasstaffedtomeetthedeadline.Inparticular,givena

deadline,theaveragefinishingdateofthesimulation,andanacceptabledelay(positive

ornegative),asimulationisconsideredsuccessfulif

FTThepossibilityofan“adaptivestaffing”isnotconsidered.Itmaybeusefultorestaffthe

(7)

DRAInthecasestudyreportedhere,thedelayweeks.E.ToolSupport

1)Aqueueingsystemsimulator;and

2)Atoolforcomputingqueueingtheoryparameters.

October17,2003

2)Periodically(sayevery10days),parametersarere-estimatedondatacollectedfromarrivedWPs,simulationwasexecuted,andprojectstaffingaccordinglymodified(ifneeded).

waschosenequalto10days,correspondingtotwo

Totrainqueueingmodels(i.e.,toestimatemodelparameters),tosimulatesystemevolutions

andcomputethelikelihoodpresentedinSectionVI,twodifferenttoolswereused:

Thequeueingsystemsimulatorwasdevelopedtosupportthedeterminationoftheteamsize

requiredtocompletetheprojectwithinagivendeadline.Inthemeantime,thelikelihoodofmeetingthedeadlineisobtained.Thesoftwareallowstheexecutionofbothdeterministicandstochasticsimulations.Inthelattercase,interarrivaltimesandservicetimesarerandomly

DRAFT

22

generated,andthesimulationisiteratedforasufficientlyhighnumberoftimessothatresultsobtaineddonotsignificantlyvarybetweenexperiments.Fordeterministicsimulations,interarrivaltimesandservicetimesarereadfromfiles.

ThequeueingsystemsimulatorconsistsofalibraryofC++classes,implementingobjectsofaqueueingsystem:

Sourcesofservicerequests:AnASCIIdatafile(usedfordeterministicsimulations:requestsarereadfromafile,containingthearrivaldatesoftherequests)orrandomsourcesQueueingnodes:composedofaqueueandoneormoreservers,thatserverequests

enqueuedfromasource.Servicetimesmaybebothreadfromafileorrandomlyproducedwithanexponentialdistribution(ifthedistributionofservicetimesisgeneral,thenthePollaczek-Khintchineformulaisusedtoapproximateresponsetimes);and

Dispatchers:usedforqueueingnetworks,thissystemrandomlydecides(givenatransitionprobabilitymatrix)towhichnodesarequestgoes.

Thequeueing-theoryparameterscomputingtoolisaPerlscriptthat,givenasetofparameters

,,

andnumberofservers

,computesthequeueingmodelrelevantparametersintroduced

,thatindicatestheprobability

DRAarequesthastomovefromnode

inSectionIII(e.g.,waitingtime,serverusecoefficient,probabilitythatallserversarebusy,etc.).

Moreover,italsohandlesqueueingnetworks,giventhematrixof

tonode.

VI.CASESTUDYRESULTS

ThemethoddescribedinSectionV-Dhasbeenappliedtotheempiricaldataobtainedby

monitoringtheprojectdescribedinSectionV.Datawereinitiallyscrutinizedwithaseniormanagertoassessdataqualityandtoavoidduplication.Theprojectplan,assignedbythecustomer,wasassumedastimereferenceforanyfurtheractivity.Accordingtotheproject

milestones,atargetfinishingdatewasfixedattheendofSeptember.Thisallowedtominimize

therisksoffinishingaftertheestablisheddeadline,i.e.,theendofOctober.

Theexperimentsdescribedinthissectionwereperformedasfollows:

1)Theentiredataset(composedofarrivaldateandeffortforall84WPs)wassplit,atdifferentdates,intoatrainingset(WPsarrivedbeforetheconsidereddate)andatestset

DRAFT

October17,2003

FT(usedforstochasticsimulations:requestsareproducedwithanexponentialdistribution);

23

DateSet

Distribution

KSTest

Chi-SquareTest

Passed

26.51

14.97

No

27.33

6.83

Yes

15.38

16.14

Yes

26.51

12.14

No

27.33

8.37

Yes

15.38

YesYesYesYes

Trainingset

0.19

(overall)

0.69

Yes

0.200.20

(TA)

0.19

(TA+Enact+UT)

0.20

0.20

0.20

0.58

0.20

Testset

(overall)(TA)

(TA+Enact+UT)

KS-TESTANDCHI-SQUARETESTRESULTS

(WPsarrivedafterthatdate).

2)Then,themodelwasbuiltusingtheparametersestimatedfromthetrainingset,and

DRAOctober17,2003

assessedusingthetestsetdata.Inparticular,thethreeprojectmilestones(endofMarch,i.e.,threepairsoftrainingsetandtestsetwereconsidered.Ontheavailabledataset,fixingarestaffingmilestoneattheendofMarchdidnotallowefficientprojectrestaffing;itwastooearlyanddataavailablerepresentedonlytheinitialtransient,ratherthentheprojectitself,theyrepresentedtheinitialtransient.Similarly,restaffingattheendofMayproducesoverstaffingbecauseofthefinaltransient.Asreportedin[4],arestaffingperformedtoo

endofAprilandendofMay)wereconsideredasreferencepointsforsplittingthedataset:

earlyortoolatealsoproducedapoorresponsetimeestimate.AgoodsolutionwouldbefixingamilestoneafterhalfoftheWPswerereceived(i.e.,attheendofApril).Thisdatecorrespondstotheprojectmid-termthat,accordingtothecompanyqualityguidelines,correspondstotheidealprojectmilestonetocheckproject-scheduleand/orperformrestaffing.Fromthispoint,the42WPsthatarrivedfromJanuarytoAprilwillbereferredtoasthetrainingset,andtheremaining42,arrivedfromMaytoJuly,astestset.periodicallyestimated,andanewstaffingwasdeterminedaccordingtothetargetdate.

DRAFT

3)Inadifferentapproach,asdescribedattheendofthissection,themodelparameterswere

FT0.32

No

0.20

14.41

Yes

TABLEI

24

Parameter

Singlequeue

Tandemqueue

TA+Enact+UT

420.0761.38400.611.16

420.0761.38389.931.20

Queuewithexit

TA420.0761.38263.90.91

25

Model

20Sept22Sept

TA+Enact+UT

Threequeuesnetwork

Assessment

3

Enact+UT

31

26

(

predictionerrorsbelow10%,see[4]).

120

9080

100

70

80

6050403020

20

10

0

0

tw [h]60

40

Servers (Assessment)

Fig.7.Queueseries:estimatedwaitingtimes

Fig.7,comparedwithFig.6,alsohighlightsthat,ontheavailabledata,aseparateassessmentteamdoesnotamelioratetheoverall

phase)withacorresponding

andthenumberofservers

centerperformingTechnicalAnalysis,EnactmentandUnitTesting,asinglemaintenancecenter(responsiblefortheentireprocess)with

FTServers (TA+Enact+UT)

123

tw [h]1112131415

.Infact,oncefixedthenumberofservers

(assessment

,with

,fortheservice

.

maintainersobtains

DRAdifferentphases.Fig.8reportsthetotal

Thefactisnotsurprising;assessmentteamshavealowutilizationcoefficients,thismeansthathavingexplicitresources(choicemadebecauseofthedifferentexpertiseneeded)devotedtoThethirdmodelanalyzedcomprisesthreedifferentnodes:oneforAssessment,onefor

assessmentpreventsthefullresourceutilizationwithintheproject.

TechnicalAnalysis,andoneforEnactmentandUnitTesting.Furthermore,themodelaccounts

forthefactthatnotallrequestsneedstheEnactment/UnitTesting(inparticular,asshown64%).Afterperforming(similarlytothepreviousmodels)analysisof

inTableII,thepercentageofWPsfromthetrainingsetrequiringEnactment/UnitTestingis

and

errorgraphs,

thecompromiseconfigurationwaschosenconsideringthebestcombinationofstaffingforthe

(consideredasthesumofwaitingtimesforthethreedifferentnodes,

takingintoaccountthefactthatonlyacertainpercentageofrequestsarriveatthefinalnode)and

thepercentageerrorsfor

,varyingwithdifferentservercombinations.Itisworthnotinghow,

forexample,47differentmaintainersmaybeassignedtodifferentphases,aconfigurationof

3-10-7(Assessment-TechnicalAnalysis-Enactment+UnitTesting)performsbetterthana2-9-9.

October17,2003

DRAFT

27

ThisisnotsurprisingsinceapercentageoftherequestsdonotrequireEnactmentandUnitTesting.Eveninthiscaseareasonablecompromisebetweencostandperformanceoccursnearthekneeofthecurves.ThefiguresuggestsaconfigurationofthreeserversforAssessment,9forTechnicalAnalysisand8forEnactmentandUnitTesting.

10095908580tw [h]7570656055

2-10-7 (46)3-10-7 (47)2-10-8 (48)3-10-8 (49)2-10-9 (50)3-10-9 (51)502-9-7 (43)3-9-7 (44)2-9-8 (45)3-9-8 (46)02-9-7 (43)3-9-7 (44)2-9-8 (45)3-9-8 (46)206080100

tr Error (%)40

2-10-7 (46)3-10-7 (47)2-10-8 (48)3-10-8 (49)2-10-9 (50)# of servers for each phase (Total # of persons involved)

# of servers for each phase (Total # of persons involved)

Fig.8.Threenodesnetwork:overallestimatedwaitingtimesanderrorsonresponsetimesforsomecombinationsofservers

Model

AllphasesAssessment

Teamsize

[h]

20Sept130

15452

3

Totalstaffing:

Threequeuesnetwork

TA

2

Totalstaffing:

1

9

98.0810.72

3-10-9 (51)2-9-9 (47)3-9-9 (48)2-9-9 (47)3-9-9 (48)28

estimatethelikelihoodofendingtheprojectonacertaindatewerecarriedoutforthethreemodels,withthetotalnumberofprogrammersinvolvedrangingfrom44to46.Fig.9-ashowstheresultsfordifferentdates,rangingfromSeptember30ththroughtoDecember15th.

100

10095

9090

80Likelihood (%)Likelihood (%)85

80

7075

70

65

Single QueueQueue SeriesThree Nodes

60

sep-30

oct-15

oct-31nov-15Estimated Deadline

a)

Fig.9.a)Likelihoodoffinishingmaintenancewithinacertaindate-b)Impactofrework

C.AnalysisofDelaysduetoRework

DRAD.ModelAssessment

October17,2003

Althoughdatacollectedfromthepresentcasestudyrevealedanegligiblelevelofrework,itis

interestingtoinvestigatehowreworkcouldaffecttheprojectcompletiondateandthelikelihood

ofmeetingthedeadline.SimulationwasthereforeperformedassumingthatacertainpercentageofWPswererandomlysubjecttoreworkfortheEnact-UTphase.Itwasassumedthatthedistributionwasthesame.Itisworthnotingthat,ifarequestwaspreviouslysubjecttorework,offinishingwithinacertaindeadlineasfunctionofthepercentageofrework.

reworkeffortwasindependentfromtheeffortpreviouslyspentforEnact-UT,eventhoughtheitwouldstillhavethesameprobabilityofbeingreworkedagain.Fig.9-breportsthelikelihood

Asafinalstep,thetestsetdatawereseededintothemodelstaffedaccordingtoTableIV,

simulatingthebehaviorontheactualdata.Results,reportedinTableV,showthatforboth

thesinglequeuemodelandqueueseriesmodels,theactualfinishingdateisveryclosetotheestimateddeadline(September30th).Inthethreequeuesnetwork,theerroroccurringinthe

DRAFT

FT6050nov-30

dec-15

sep-30oct-15oct-31nov-15Estimated Deadline

nov-30b)

0%5%10%15%20%25%dec-1529

estimationwashigher.However,ashighlightedfromFig.9-a,astimeadvances,thelikelihoodoffinishingwithathree-queuesmodelisalwayshigherthanthoseofothermodels.

Model

FinishingDate

Singlequeue

Sep-22

Threequeuesnetwork

Oct-7Sep-28

6.54%

Actual

%estimation

30

Similarly,attheendoftheprojectthenumberofpendingrequestsdecreasedsothatasmallernumberofserverssufficed.Afewdaysbeforethedeadline,onlyoneWPremainedand,asshowninthefigure,asingleserverforeachphasewasrequired;and

Finally,therearecasesforwhichthearrivalrateortheaverageservicetimesmaybesubjecttorelevantvariationsduringtheproject,sothatrestaffingtomeetthedeadlineortoavoidwastingresourcesisnecessary.

Fig.10-aalsoreportstheactualprojectstaffing.Theseactualstaffinglevelswereobtainedoptimisticestimate,notaccountingforoverheadssuchastrainingonthejob,administrativedutiesorabsencessuchasvacations.Thesimulatedcurvecorrespondstoanaveragestaffingof36peopleperday,whiletheactualstaffingwas,onaverage,29peopleperday.Inotherwords,thesimulationtendstoproduceanaverageoverstaffingofabout20%.Thefigurealsohighlightsaninitialoverstaffing(theleftmostpointonthesimulatedcurve),duetotheabsenceofsufficienttrainingmaterial.TheMann-Whitneytest(p-value1.4e-5forasignificancelevelof95%)confirmedthatthedifferencebetweenthetwocurveswassignificant.Thus,thesimulationtendstogiveusapessimistic,conservativeestimateofthestaffing.However,theoverallaccuracy

DRAVII.LESSONSLEARNED

deadline,whilemonitoringtheprojectstaffinglevel.givingrelevantinsightstothemanagers.

October17,2003

ishighersincetheactualstaffingshowninFig.10-aisthetheoreticalequivalentactualstaffingcorrespondingtoanidealsituationwherepeoplework20daysamonth.

Massivemaintenanceinterventions,aswellasmanyothersoftwareengineeringprojects,

arefirmdeadlineprojects.ThusakeyparametertomonitoristhelikelihoodofmeetingthesimulationisaneffectivetooltomonitorandassessthelikelihoodofmeetingagivenprojectAnotheradvantageoftheproposedapproachisthatreworkimpactmaybeeasilyestimated.

deadline,inthatdelaysmaysubstantiallyinfluencetheprojectcost.Weexperiencedthatqueuing

Differentsimulationswithdifferentteamsizesandreworklevelswillleadtodifferentestimates,Modelparameterscouldbeadjustedasnewdataarrives,thusrefiningearlyestimates.Queuing

simulationand,moregenerally,queueingtheory,hasfurtheradvantageswhenappliedtosoftware

maintenanceprojects.Indeed,itisessentialthatthenumberofmodelinputparametersiskept

DRAFT

FTbydividingthedaily-recordedeffortby8(thenumberofworkinghoursperday).Thisisan

31

lowand,furthermore,thatthoseinputparameterscanbeeasilyestimatedandunderstoodbymanagers.Thoughthemodelper-secanbehighlycomplex,thekeyobservationisthatitsinterfacecanbekeptrelativelysimpleandintuitive.Webelievethattheparametersrequiredtorunasimulationarereadilyunderstandabletopeopleworkinginthesoftwareindustry.Evenwithacomplexmodelaccountingformultiplemaintenancecentersandmulti-phasemaintenanceprocesses,therequiredinputparametersaresimply:themaintenancerequestinter-arrivaltimes,themeaneffortrequiredbyamaintenancetask,andatargetdeadline(seeFig.5).Thusqueuebaselinesbycomputingsimpledescriptivestatistics.

parametershaveintuitivephysicalmeaningandtheymaybeeasilyobtainedfromexistingprojectWhentalkingtoprojectmanagersoftheY2Kproject,wenoticedthattheyhadnodifficultyinprovidingestimatesforthemodelrequiredparameters.However,theyoftenbasedthoseestimatesonahybridapproach,usinganalogyandalgorithmicmethods.Basically,inter-arrivaltimesandeffortrequiredinamaintenanceinterventionwerecomputedastheaveragevalueoverpastprojectdata.Figureswerethenvalidatedandadjustedaccordingtothemanagerexperience.Thisempiricalstudyconfirmedthatqueueingsimulationrequiresfinegrainrecordeddata,dailyorhalf-dailybased;weeklybasedrecordingofeffortandinter-arrivaltimesproducesmodelswhicharetoocoarse,evenworse,morecomplexmodelsaccountingforbatcharrivalsmaybeSoftwareprojectsexhibitcharacteristicsquitedifferentfromotherphenomenathatcanbe

DRArequired.

October17,2003

modeledbyclassic(nonsimulation-based)queueingtheory.Inourcasestudy,stochasticsimu-

lationsrevealedthenecessitytostudythedynamicaspectsoftheprocess(i.e.,projectstartupstudy,projectlifespanwasaboutoneyear,projectstartupandclose-downcoveredabout30%projectandnon-simulationbasedapproachesshouldonlybeconsideredasafirstapproximation.Queueing-basedmodelstendtoproduceastaffinglevelthatoverestimatestheactualstaffing.Thisisespeciallytrueforthetransientphases.Thus,forafirmdeadlineproject,intheworst-casescenario,aqueue-basedmodelwillleadtoaminimizationoftheprojectmanagerriskofexcessivedelays.Inthecasestudyreportedhere,itwasalsoobservedthatthetotaleffortspentontheprojectandtheareaunderthesimulatedcurveshowninFig.10donotappeartobeclose.Adeeperanalysisrevealedthattheaccuracyofthemethodishigherthanthe20%

DRAFT

andclose-down,averagefinishingdate,likelihoodoffinishingwithinacertaindate).Inthiscaseoftheentireproject:inotherwords,thetransientpartwasasubstantialportionoftheentire

FT32

averageerrorshowninthefigure.Discussionwithcompanymanagersrevealedthat,accordingtoacompanyguideline,theaveragenumberofworkingdaysinamonthwasestimatedtobebetween17and17.5insteadof20.Thisfigureaccountsforthedifferentsourceofoverheadsandabsences.ThustheactualstaffingofFig.10mustbeincreasedbyabout13-15%.Inotherwords,theaveragestaffinglevelanymanagerofthecompanywouldhaveacceptedwasnot29but33/34maintainerswithacorrespondingerrorofabout6-9%.

Intheexperienceoftheauthors,theproposedapproachhastobecomplementedandintegratedquests.Giventhecurrentstateoftheartinsoftwareengineering,weperceivehumaninterventiontobeessentialinordertominimizeconflictingmaintenanceinterventions,toevaluatetheimpactofcommunicationoverheadinsideandamongmaintenanceteams,etc.

Afinalissueistheapplicabilityandgeneralityoftheapproach.AspointedoutintherelatedworkSection,simulationbasedapproacheswerepreviouslyproposedintheliterature;inparticularin[18]theauthorobservedthatthereareprojectswhereBrooks’lawseemsnottobecompletelyapplicable.Thisisalsocentraltoourapproach;werelyonthehypothesisthatmaintenanceinterventionsmaybestandardized.Inotherwords,increasingthenumberofpeople,decreasestheprojectduration.Othermassivemaintenanceprojects,suchastheEurowithadetailedwork-breakdownstructureandaplanningprocess,todispatchmaintenancere-

DRAmassivemaintenanceintervention[2].

VIII.CONCLUSIONS

andassessmentofmaintenanceprojects.

October17,2003

conversionforthenewEuropeanUnionmembers,orchangingthephonenumberingsystem

(tobeperformedinUSAby2010),arelikelytoexhibitcharacteristicsverysimilartoaY2K

Datafromamassive,correctivemaintenanceprojecthavebeenpresentedtogetherwithanap-

proachbasedonqueueingtheoryandstochasticsimulationtodealwiththestaffing,managementQueueingtheoryandstochasticsimulationsarehighlyeffectiveatevaluatingstaffinglevels,

aswellassupportingandassessingrestaffingdecisions.Thepapershowshowsimulationscanbecarriedouttomodelprojectstartup,close-down,andtoevaluatethelikelihoodofmeetingtheprojectdeadline.Thelatterfact,evaluatedfordifferentnetworktopologiesandreworklevels,givestothemanagementabroaderviewofthecomparisonbetweentheactualprojectestimatedstatusandtheprojectplan.Thelikelihoodofsuccesscanbeusedtoestablishatrade-offbetween

DRAFT

FT33

staffing/restaffing,acceptedrisks,projectdelaysandcustomerexpectations.Themainadvantagesoftheproposedapproachcanbesummarizedasfollows:

Itishighlyeffectiveatevaluatingthelikelihoodofmeetingtheprojectdeadline;

Rework,aswellasabandonment,canbeexplicitlymodeled,andtheireffectsonprojectstaffingandprojectmilestonesassessed;and

Projectstart-upandclose-downcanbeexplicitlymodeled.

Asafinalremark,ithasbeenshownhowdifferentprojectmanagementapproachesimpactonresourceutilization,therebyallowingeffectivecomparisonofdifferenttopologieswithrespecttothelikelihoodofmeetingtheprojectdeadline.

IX.ACKNOWLEDGEMENT

Thisresearchhasbeensupportedbytheproject”VirtualSoftwareFactory”,fundedbyMin-isterodellaRicercaScientificaeTecnologica(MURST),andjointlycarriedoutbyEDSItaliaSoftware,UniversityofSannio,UniversityofNaples”FedericoII”andUniversityofBari.

REFERENCES

[1]T.DeMarco,ControllingSoftwareProjects,YourdonPress,1982.

[2]C.Jones,“Mass-updatesandsoftwareprojectmanagementinisorganizations-http://www.artemis.it/artemis/artemis/lang

34

[10]T.PearseandP.Oman,“Maintainabilitymeasurementsonindustrialsourcecodemaintenanceactivities,”inProceedings

ofIEEEInternationalConferenceonSoftwareMaintenance,Opio-NiceFrance,Oct1995,pp.295–303.[11]B.W.Boehm,SoftwareEngineeringEconomics,Prentice-Hall,EnglewoodCliffs,NJ,1981.

[12]BoehmB.W.,HorowitzE.,MadachyR.,ReiferD.,ClarkB.K.,SteeceB.,WinsorBrownA.,ChulaniS.,andC.Abts,

SoftwareCostEstimationwithCocomoII,Prentice-Hall,EnglewoodCliffs,NJ,2000.

[13]W.K.Wiener-Ehrlich,J.R.Hamrick,andV.F.Rupolo,“Modelingsoftwarebehaviourintermsofaformallifecyclecurve:

Implicationsforsoftwaremaintenance,”IEEETransactionsonSoftwareEngineering,vol.10,no.4,pp.376–383,1984.[14]L.C.BriandandV.R.Basili,“Aclassificationprocedureforaneffectivemanagementofchangesduringthesoftware

maintenanceprocess,”inProceedingsofIEEEInternationalConferenceonSoftwareMaintenance,Orlando,FL,1992.SystemsandSoftware,vol.46,no.2/3,pp.91–106,1999.

[15]M.I.Kellner,R.J.Madachy,andD.M.Raffo,“Softwareprocesssimulationmodelling:Why?what?how?,”Journalof[16]R.H.MartinandD.M.Raffo,“Amodelofthesoftwaredevelopmentprocessusingbothcontinuousanddiscretemodels,”

InternationalJournalofSoftwareProcessImprovementandPractice,vol.5,no.2/3,pp.147–157,2000.Software,vol.47,no.9,2000.

[17]D.M.RaffoandM.I.Kellner,“Empiricalanalysisinsoftwareprocesssimulationmodeling,”JournalofSystemsand[18]T.K.Abdel-Hamid,“Thedynamicsofsoftwareprojectstaffing:asystemdynamicsbasedsimulationapproach,”IEEE

TransactionsonSoftwareEngineering,vol.15,no.2,pp.109–119,1989.Cliffs,N.J.,1991.

vol.26,no.3,pp.20–29,1993.1995.

[19]T.K.Abdel-HamidandS.E.Madnick,SoftwareProjectDynamics:AnIntegratedApproach,Prentice-Hall,Englewood[20]T.K.Abdel-Hamid,“Adapting,correcting,andperfectingsoftwareestimates:amaintenancemetaphor,”IEEEComputer,[21]F.Brooks,TheMythicalMan-Month,20thanniversaryedition,Addison-WesleyPublishingCompany,Reading,MA,[22]G.E.StarkandP.W.Oman,“Softwaremaintenancemanagementstrategies;observationsfromthefield,”Journalof

SoftwareMaintenance-ResearchandPractice,vol.9,pp.365–378,December1997.

[23]M.Jorgensen,“Experiencewiththeaccuracyofsoftwaremaintenancetaskeffortpredictionmodels,”IEEETransactions

onSoftwareEngineering,vol.21,no.8,pp.674–681,1995.

DRA[30]IEEEstd1219:StandardforSoftwaremaintenance,1998.

Engineering,vol.23,no.11,pp.736–743,1997.

October17,2003

[24]M.H¨ost,B.Regnell,J.NattochDag,J.Nedstam,andC.Nyberg,“Exploringbottlenecksinmarket-drivenrequirements

managementprocesseswithdiscreteeventsimulation,”inProsim,London,UK,July1999.

[25]G.PapapanagiotakisandP.Breuer,“Asoftwaremaintenancemanagementmodelbasedonqueueingnetworks,”Journal

ofSoftwareMaintenance-ResearchandPractice,vol.6,no.1,pp.73–97,1994.

[26]R.Ramaswamy,“Howtostaffbusinesscriticalmaintenanceprojects,”IEEESoftware,vol.7,no.7,pp.90–95,May2000.[27]I.PodnarandB.Mikac,“Softwaremaintenanceprocessanalysisusingdiscrete-eventsimulation,”inConferenceon

SoftwareMaintenanceandReengineering,Lisbon,Portugal,March2001,pp.192–195,IEEESocietyPress.

[28]D.M.Raffo,W.Harrison,andJ.Vandeville,“Coordinatingmodelsandmetricstomanagesoftwareprojects,”International

JournalofSoftwareProcessImprovementandPractice,vol.5,no.2/3,pp.159–168,2000.

[29]JerryBanks,JohnCarson,andBarryL.Nelson,Discrete-EventSystemSimulation,PrenticeHall,1998.

[31]M.J.ShepperdandC.Schofield,“Estimatingsoftwareprojecteffortusinganalogies,”IEEETransactionsonSoftware

FTDRAFT

35

[32]R.Arnold,“Softwarereengineering,”1993.

[33]E.C.Lynd,“Livingwiththe2-digityear,year2000maintenanceusingaproceduralsolution,”inProceedingsofIEEE

InternationalConferenceonSoftwareMaintenance,Bari,Italy,1997,pp.206–212.

[34]M.Stone,“Cross-validatorychoiceandassessmentofstatisticalpredictions(withdiscussion),”JournaloftheRoyal

StatisticalSocietyB,vol.36,pp.111–147,1974.

[35]G.BoxandM.Jenkins,TimeSeriesForecastingAnalysisandControl,HoldenDay,SanFrancisco(USA),1970.[36]MylesHollanderandDouglasA.Wolfe,NonparametricStatisticalMethods,2ndEdition,Wiley-Interscience,1999.[37]JayL.Devore,ProbabilityandStatisticsforEngineeringandtheScience,DuxburyPress,1999.

GiulianoAntoniolreceivedhisdoctoraldegreeinElectronicEngineeringfromtheUniversityofPaduain1982.HeworkedatIrstfortenyearswereheledthetheIrstProgramUnderstandingandReverse

PLACEPHOTOHERE

Engineering(PURE)Projectteam.GiulianoAntoniolpublishedmorethan60papersinjournalsandinternationalconferences.HeservedasamemberoftheProgramCommitteeofinternationalconferencesandworkshopssuchastheInternationalConferenceonSoftwareMaintenance,theInternationalWorkshoponProgramComprehension,theInternationalSymposiumonSoftwareMetrics.Heispresentlymemberof

theEditorialBoardoftheJournalSoftwareTestingVerification&Reliability,theJournalInformationandSoftwareTechnology,theEmpiricalSoftwareEngineeringandtheJournalofSoftwareQuality.HeiscurrentlyAssociateProfessortheUniversityofSannio,FacultyofEngineering,whereheworksintheareaofsoftwaremetrics,processmodeling,softwareevolutionandmaintenance.

DRAPLACEHEREPHOTO

October17,2003

AnielloCimitilereceivedtheLaureadegreeinElectronicEngineeringfromtheUniversityofNaples,Italy,in1973.HeiscurrentlytheChancelloroftheUniversityofSannioinBenevento,Italy,whereheisafullProfessorofComputerScience.Previously,hewaswiththeDepartmentof’InformaticaeSistemistica’attheUniversityofNaplesFedericoII.Since1973hehasbeenaresearcherinthefieldofsoftwareengineeringandhislistofpublicationscontainsmorethan100paperspublishedinjournalsandconferenceproceedings.Heservesintheprogramandorganizingcommitteesofseveralinternational

conferencesandintheeditorialandreviewercommitteesofseveralinternationalscientificjournalsinthefieldsofsoftwareengineeringandsoftwaremaintenance.Heisaco-editorinchiefoftheJournalofSoftwareMaintenanceandEvolution:ResearchandPractice.Hehasbeenresponsibleformanyinternationalandnationalappliedresearchprojects.Hisresearchinterestsincludesoftwaremaintenanceandtesting,softwarequality,reverseengineering,andreusereengineering.

FTDRAFT

36

GiuseppeA.DiLuccareceivedtheLaureadegreeinElectronicEngineeringfromtheUniversityofNaples”FedericoII”,Italy,in1987andthePh.D.degreeinElectronicEngineeringandComputerSciencefrom

PLACEPHOTOHERE

thesameUniversityin1992.HeiscurrentlyanassociateprofessorofComputerScienceattheDepartmentof”Ingegneria”oftheUniversityofSannio.Previously,hewaswiththeDepartmentof’InformaticaeSistemistica’attheUniversityofNaples’FedericoII’.Since1987hehasbeenaresearcherinthefieldofsoftwareengineeringandhislistofpublicationscontainsmorethan50paperspublishedinjournalsand

conferenceproceedings.Heservesintheprogramandorganizingcommitteesofconferencesinthefieldofsoftwaremaintenanceandprogramcomprehension.Hisresearchinterestsincludesoftwareengineering,softwaremaintenance,reverseengineering,softwarereuse,softwarereengineering,webengineering,andsoftwaremigration.

DRAPLACEHEREPHOTOmaintenancefield.

October17,2003

MassimilianoDiPentareceivedhislaureadegreeinComputerEngineeringin1999andhisPh.D.inComputerScienceEngineeringin2003attheUniversityofSannioinBenevento,Italy.Currentlyhe

iswithRCOSTResearchCentreOnSoftwareTechnologyinthesameUniversity.Hismainresearch

interestsincludesoftwaremaintenance,reverseengineering,programcomprehensionandsearch-basedandworkshops.Heservestheorganizingcommitteesofworkshopsandconferencesinthesoftware

softwareengineering.Heisauthorofabout25papersappearedininternationaljournals,conferences

FTDRAFT