the SAS® System
Cheryl Doninger, Manager of SAS/CONNECT® Software R & D
Randy Tobias, Manager of Linear Models R & D
Introduction
Parallel computation allows large data processing tasks critical for science, business, andindustry to be accomplished in a fraction of the usual time. The new MP CONNECT tools inSAS/CONNECT software enable you to perform parallel computation by coordinating the dataprocessing power of the SAS System running simultaneously on multiple servers.
The %Distribute macro is the workhorse of a system of SAS macros using the new MP
CONNECT capabilities to distribute a fairly general class of computational problems across anarbitrarily large number of SAS/CONNECT servers. Features of the system include
• a load balancing scheme to improve the scaling efficiency of the parallel computation• a simple but informative real-time display of the state of all the servers
• automatic generation and management of independent random number streams on the
servers
This paper describes the underlying approach used in %Distribute and demonstrates itsapplication for Monte Carlo simulation. The approach has also been used to address openproblems in statistical analysis of variance (Westfall et al. 2001, O'Brien and Tobias 2001). Theaccuracy and completeness of these simulation approaches would have been infeasible withoutthe distributed parallel processing capabilities in MP CONNECT. For the work of Westfall et al. inparticular, MP CONNECT made it possible to condense three years' worth of computing time intojust a month's worth of desktop time.
Distributing SAS with MP CONNECT
To begin, let's step through what's required for distributing a large SAS job with MP CONNECT.In outline, all you need to do is• Connect to the hosts.• •
Give them tasks to do in parallel.Wait for them to complete.
To be more specific, suppose you have a macro
%Simulate(Output,1000,seed=64382);
that simulates a complicated random process 1000 times and puts the resulting sample in thedata set Output. In theory, you could do this in half the time if you did it in parallel on twomachines instead of one. What are the bare-bones steps you need to take to do this with MPCONNECT? First, you need to sign on to the hosts
%letHost1=< filenameScript\"< signonremote=Host1script=Script; %letHost2=< filenameScript\"< and remotely submit the code to do half the problem on each one: rsubmitremote=Host1wait=no; %Simulate(Output,500,seed=64382);endrsubmit; rsubmitremote=Host2wait=no; %Simulate(Output,500,seed=31584);endrsubmit; The WAIT=NO option on the RSUBMIT statements is crucial. It specifies that the two halves ofthe problem are to run simultaneously, and thus enables the anticipated speed-up. So finally, youjust wait for the jobs to finish: waitfor_all_Host1Host2;But this leaves out some crucial details:• How do you get the definition for the %Simulate macro from the client to the hosts? Youneed to run the code to define it on each of the hosts. One way to do this is to nest thedefinition within another macro that remotely submits it: %macroRInit; rsubmitremote=Host&iHost; %macroSimulate(ds,n,seed=0);< and run this for each host %letiHost=1;%RInit;%letiHost=2;%RInit; • How do you do you retrieve the results on the hosts and put them back together on theclient? If the results can be condensed to a few macro variables, then you can pass thesemacro variable values back through the MP CONNECT-ion using %sysrput. However, ifyour results are more complicated, you'll need to use data sets. SAS/CONNECT RemoteLibrary Services enable you to connect the client to the WORK library of each host libnameRWork1slibref=WORKserver=Host1;libnameRWork2slibref=WORKserver=Host2; and you can use these to combine the two host Output data sets into a single one on theclient: dataOutput;setRWork1.OutputRWork2.Output;run; Load Balancing The last section presented a general scheme for distributing a task of size N across k hosts(N=1000 and k=2 above). If your hosts are all guaranteed to have equivalent horsepower, thenthis scheme will likely work well enough. However, if some of the k hosts are faster than others,then simply giving each of them a job of size N/k and waiting for them all to complete will make itseem like they are all as slow as the slowest one. What you need to do is to balance the load,giving faster hosts more of the job and slower ones less. If you know the relative speeds of your hosts, then you can determine the proper load balance upfront in such a way that all hosts finish their part of the job in about the same amount of time.However, when you are accessing many hosts on a network, you often cannot determine hostspeed ahead of time, since it is a function of jobs other users are running in real time. In thiscase, a reasonable approach is to break the size N task into small chunks of size n < N/K. Ashosts finish up their chunk, give them another until the entire problem has been parceled out.This interactive form of load balancing will give faster hosts more chunks and thus more of theentire problem. What is the optimal choice for the size n of the chunks? There are two things to consider: howmuch time it takes on the hosts to run each chunk, and how much time it takes on the client tomanage submitting chunks. The distributed processing will be efficient when the hosts are givenenough work so that they don't spend too much time waiting for the client to manage them, butnot so much that the entire process ends up waiting for slow hosts to finish their task. The %Distribute Macro System The previous two sections discuss how to distribute a large SAS job efficiently over several hosts.Using MP CONNECT, you connect to and initialize your hosts, remotely submit pieces of the job,and then collect the results. Furthermore, you should balance the load so that fast hosts do moreof the work; breaking the problems into more chunks than there are hosts and giving fast hostsnew chunks to work on as they finish can be effective. The system built around the %Distribute macro implements the approach to distributed SASprocessing outlined above. In general, %Distribute addresses problems that consist of manyreplicate runs of a fundamental task. For example,• • • Global integer optimization problems are often solved by steepest ascent (the fundamentaltask) with random restarts (the replicate runs). Monte Carlo methods in statistics rely on a set of pseudo-random observations (the replicateruns) of one or more difficult-to-compute test statistics (the fundamental task). Mining web data for e-intelligence applications often involves running queries and summaries(the fundamental task) over a massive number of different groups of data (the replicate runs). If your problem can be factored in this way, then %Distribute may offer a useful approach.Besides running the specified fundamental task the required replicate number of times,%Distribute offers two additional services:• Applications of %Distribute often require a stream of random numbers, and %Distributecreates and manages such a stream for each host. • In order to monitor your program, you would like to know where your fundamental processingtime is spent. So %Distribute keeps track of how much time each host spends working on itsfundamental tasks and waiting for the client to give it more work. In order to use the %Distribute system, you must first tell it how to connect to and initialize thehosts and what fundamental task to perform. In order to do this, you need to create the followinginput for the system: 1. _Hosts data set - contains information about the hosts you want to run on2. %RInit macro - defines the programs to be run on each hostDetailed descriptions of the input follow: _Hosts Data Set Define a data set named _Hosts in the current WORK library with an observation for eachhost you want to distribute the computation to. For each observation, set two (character)variables: VariableNameHostScriptContents Name of the hostSignon script for host These variables are used on the client, so make sure they are valid names and paths inthis environment. The value of the Host variable should be either a name defined in yourHOSTS file, a name defined to your name server, or the actual IP address of a machineto which you want to connect. The value of the Script variable is the unquoted fullpathname of a script file to be used in a SIGNON statement for the specified HOST. Ifthe script file exists in the directory from which SAS is invoked, you can specify only thefile name instead of the full path. %RInit Macro This is what really defines the distributed computation. It needs to wrap (at least) twomacro definitions %FirstRSub - initialization on that host %TaskRSub - the fundamental unit of distributed computation within an RSUBMIT block, like so %macroRInit; rsubmitremote=Host&iHostwait=yesmacvar=Status&iHost; %macroFirstRSub; < %macroTaskRSub; < < endrsubmit;%mend; It is your job to write the macros %FirstRSub and %TaskRSub so that they define thefundamental task. Note that while anything between %macro…; and %mend; in theRSUBMIT block will be submitted on the host as expected, additional macro initializationson the host that are not between %macro…; and %mend; may be intercepted by theclient's macro processor by default. For example, if you want to initialize the macrovariable HostMac to 1 for the host inside the RSUBMIT block but outside of %FirstRSub, you need to use the macro quoting function %nrstr() to keep the %letstatement from being intercepted by the client, like so: %macroRInit; rsubmitremote=Host&iHostwait=yesmacvar=Status&iHost; %macroFirstRSub;…;%mend;%macroTaskRSub;…;%mend; %nrstr(%%)letHostMac=1;... endrsubmit;%mend; For more information on interaction between remote submitting and the macro language,see Doninger (2001). As parameters for these macros, you can assume the following global macro variables tobe available within the host SAS session: Predefined hostglobal macroRem_HostRem_iHostRem_NIterRem_SeedContents Name of this host Index of this host in _Host data setSize of chunk Random seed, rerandomized for each chunk If you need other macro values (say, &Mac1, &Mac2, and &Mac3) to be passed from theclient to the host, you should specify them in a %Macros macro, like so: %macroMacros; %syslputrem_Mac1=&Mac1;%syslputrem_Mac2=&Mac2;%syslputrem_Mac3=&Mac3;%mend; The %Distribute macro will take care of running this macro for each of the hosts, so thatyou can use the macros &rem_Mac1, &rem_Mac2, and &rem_Mac3 in your definitions of%FirstRSub and %TaskRSub. Once you have created the _Hosts data set and the %RInit macro, all you need to do to distributethe fundamental task across all the hosts is: 1. Use the %SignOn macro to sign on to the hosts. 2. Set the macro variables &NiterAll and &Niter to the size of the entire problem and the size of each chunk, respectively.3. Use the %Distribute macro to perform the distribution. As chunks of the fundamental task are parceled out to the hosts, a status line will be printed tothe SAS log on the client, displaying '.' for hosts that are currently working, '!' for hosts that are currently waiting to be assigned work, and'X' for hosts that are unable to do work for some reason. Additional status information includes the number of fundamental tasks submitted and completedso far. When the distribution is complete, the system will display a breakdown of how much workwas done on each host, together with an over-all summary of the efficiency of the distribution.Finally, you can4. use the %RCollect or %RCollectView macros to collect the results from each host into a data set or DATA step view that concatenates them all. Further analysis of the collected results can be performed on the client. Example: Distributing a Monte Carlo Simulation In linear models, an experimental design can be concisely represented as a matrix X, and theeigenvalues of X'X give information about the efficiency of the design. In particular, the so-calledE-efficiency of a design is related to the smallest eigenvalue of X'X. Suppose you are interested k in the E-efficiency of a set of k normally distributed points in R, for different values of k. Creatingrandom normal data and computing the minimum eigenvalue requires only a couple of lines ofSAS/IML code x=rannor((1:&k)`*(1:&k));e=eigval(x`*x)[><]; but it takes a large sample to estimate the expected value of e with precision, and if you want tolook at the expected value for many values of k, it can take a while. In order to distribute the job,you first need to wrap the above code so that it can be run by %Distribute on the hosts. %macroRInit; rsubmitremote=Host&iHostwait=yesmacvar=Status&iHost; /* /Thismacroinitializesthehost,preparingittorun/thefundamentaltaskmultipletimes.Inthiscase,/allyouneedtodoisinitializethedatasetinwhich/alltheresultsforthishostaretobecollected. /--------------------------------------------------------*/%macroFirstRSub; optionsnonotes;dataSample;/*Createanemptydataset*/ if(0);run;%mend; /* /Thismacrodefinesthefundamentaltask.SAS/IMLis/usedtoperformthebasicsimulationthenumberof/times(&rem_NIter)required.Eachchunkofresults/iscreatedinaworkdatasetcalledSample,andall/theresultsforthishostarecollectedin/Results.Sample&rem_iHost. /--------------------------------------------------------*/%macroTaskRSub; prociml; /* /Initializerandomnumbers /--------------------------------------------------*/x=rannor(&rem_Seed); /* /Foreachjobinthischunk,createrandomdata,/computemineigenvalue,andsaveit. /--------------------------------------------------*/freedata; doi=1to&rem_Niter; x=rannor((1:&rem_Dimen)`*(1:&rem_Dimen));e=eigval(x`*x)[><];data=data//e;end;/* /Savewholesampleforthischunk. /--------------------------------------------------*/createiSamplevar{E1};appendfromdata;quit; /* /Appendthissampletotheaccumulatedsetofall/samplescreatedonthishost. /--------------------------------------------------*/dataSample;setSampleiSample;run;%mend; endrsubmit;%mend; Define this macro on the client and make a _Hosts data set as described above. One final bit ofset-up is required: you need to tell %Distribute to pass the dimension of X down to the hosts.Note that we have assumed in %TaskRSub above that this is available in a macro &rem_Dimen.Assuming its value is associated with a macro variable &Dimen on the client, the followingdefinition for %Macros will do the trick. %macroMacros; %syslputrem_Dimen=&Dimen;%mend; Once these three components (_Hosts, %Rinit, and %Macros) have been set up, the followingcode will run a 10-dimensional simulation 1,000,000 times over the hosts in chunks of 2,000. %inc\"Distribute.sas\"; /* /Argumentsforthedistribution. /--------------------------------------------------------------*/%letNIter=2000;%letNIterAll=1000000;%SignOn; /* /Thisperformsthedistributedcomputationfora /10-dimensionalproblem. /--------------------------------------------------------------*/%letDimen=10;%Distribute; For 25 hosts, the first few status lines look like this: Stat:Stat:Stat:Stat:Stat: .!!!!!!!!!!!!!!!!!!!!!!!!:..!!!!!!!!!!!!!!!!!!!!!!!:...!!!!!!!!!!!!!!!!!!!!!!:....!!!!!!!!!!!!!!!!!!!!!:.....!!!!!!!!!!!!!!!!!!!!: (2000,0)/1000000(4000,0)/1000000(6000,0)/1000000(8000,0)/1000000(10000,0)/1000000 The bottom line here, for example, indicates that the first five hosts are working on their tasks andthe others are waiting to be given tasks. After a while, the tasks on the first hosts will begin tofinish up, and they'll be given more to do: Stat:!..........!!!!!!!!!!!!!!:(22000,2000)/1000000Stat:.!..........!!!!!!!!!!!!!:(26000,4000)/1000000Stat:..!!.........!!!!!!!!!!!!:(30000,8000)/1000000 Eventually, all hosts will have been given tasks and the status line will indicate which ones finishtheirs while other hosts are being serviced by the client: Stat:Stat:Stat:Stat: ..!..........!!.....!!...:....!!...!............!..:...!...!........!......!.:..........!!...!.........: (682000,642000)/1000000(692000,650000)/1000000(702000,660000)/1000000(712000,668000)/1000000 Toward the end of the run, most hosts will be waiting while the last chunks to finish up, and astatus line will be printed as each one does: Stat:Stat:Stat:Stat:Stat: !.!!!!.!!!!!!!.!!!!!..!.!:!.!!!!.!!!!!!!!!!!!!!!!.!:!.!!!!.!!!!!!!!!!!!!!!!!!:!!!!!!.!!!!!!!!!!!!!!!!!!:!!!!!!!!!!!!!!!!!!!!!!!!!: (1000000,988000)/1000000(1000000,994000)/1000000(1000000,996000)/1000000(1000000,998000)/1000000(1000000,1000000)/1000000 For the 25 UNIX hosts that the above test was run on, the timing results look like this: WorkTime/2000Iter0:00:040:00:030:00:030:00:03 ... 0:00:030:00:03 EstimatedTimeforEntireProblem0:35:280:27:500:27:190:27:08 ... 0:25:050:25:02 WaitTime/2000Iter0:00:010:00:010:00:000:00:01 ... 0:00:010:00:01 iHost20815...2212 HostUNIX68UNIX73UNIX49UNIX16 ... UNIX71UNIX79 No.Iter30000420004800044000 ...3600040000 DistributionEfficiency 92%72%71%70%...65%65%0:01:33 Totalelapsedtime: Cumulativeworkingtime:Cumulativewaitingtime:0:26:220:07:59 (The names of the machines have been changed to protect the innocent.) Most of the machinestook about three seconds for each chunk of 2000 samples, but UNIX68 (in bold) took about fourseconds and was consequently given fewer chunks to do. The bottom line is that a job that wouldhave taken about a half-hour on one machine took just a minute and a half distributed across 25machines. You can use a DATA step view to collect the results from all 25 hosts into one data set andanalyze them: %RCollect(Sample,Sample/view=Sample); procmeansdata=Samplenmeanstderrlclmuclm; vare1;run; The results indicate that the minimum eigenvalue for a 10-dimensional, 10-point random normaldesign is about 0.0725 with about four digits of precision. TheMEANSProcedureAnalysisVariable:E1 Lower95%Upper95% NMeanStdErrorCLforMeanCLforMean --------------------------------------------------------------10000000.07245930.0001068800.07224980.0726688--------------------------------------------------------------The original problem was to examine the relationship of the minimum eigenvalue to the dimension for a range of dimensions. Once all the set-up has been done, it is a simple matter towrap a macro around the distribution and collection code: %macroMinEVal; optionsnonotes; dataMinEval;if(0);run;%doDimen=1%to30; %Distribute; %RCollectView(Sample,Sample);procsummarydata=Sample; vare1; outputout=_tempmean=MeanStdErr=StdErr LCLM=LCLMUCLM=UCLM; data_temp;set_temp;Dimen=&Dimen;dataMinEval;setMinEval_temp;run;%end;optionsnotes;%mend;%MinEval; This simulation would require 40 hours to run on a single machine, but it took less than two hourswhen distributed across 25 UNIX hosts. The results indicate that the expected value of theminimum eigenvalue is inversely related to the dimension. Conclusion The %Distribute system described in this paper, for distributing many replicates of a fundamentaltask across multiple hosts, has been applied to a variety of real-world problems. It enabledWestfall et al. (2001) and O'Brien and Tobias (2001) to use simulation to address importantproblems in statistical testing for analysis of variance for which analytical results were difficult oreven impossible. The accuracy and completeness of these simulation approaches would havebeen infeasible without the distributed parallel processing capabilities in MP CONNECT. Inparticular, for the work of Westfall et al. (2001), MP CONNECT made it possible to condensethree years' worth of computing time into just a month's worth of desktop time. Global optimization using steepest ascent with random restarts is also amenable to distribution bythis method. Thus, for example, it is straightforward to use this system with the OPTEXprocedure in SAS/QC™ software to perform extensive searches for optimal experimentaldesigns. The system has been tested on a variety of configurations, using PCs running Windows NT andHP/UX UNIX boxes for both the client and the hosts. The system successfully used MP CONNECT to distribute a single computation between the U.S. and Germany, connecting throughthe Internet using only the IP addresses of the German machines. As many as 150 servers havebeen employed simultaneously. The %Distribute system is very much a work in progress; we are still working to make it easier toconfigure as well as more robust. For example, although the current version of the system workswell as long as all server connections stay active, handling for connection problems needs to beimproved, to ensure that work is salvaged when some of the servers go down. Moreover, developing the %Distribute system has inspired areas for future work in MP CONNECT, including• remote library services that make it easy for the servers to transfer data from and back to the client • easy ways to pass macro definitions from the client down to the servers The goal is to make it as easy as possible for you to use MP CONNECT to multiply by manytimes the power of SAS for solving complex problems. Acknowledgment Dr. Frank Bretz of the University of Hannover implemented and tested various versions of %Distribute and also made his machines available to us for testing. His insights on issues (i.e.bugs) have significantly improved both the software and this article, and we heartily thank him forhis encouraging help. References Doninger, C. (2001), \"Interaction Between Macro Facility and RSUBMIT,\" unpublished manuscript.O'Brien, R., and Tobias, R. (2001), \" Improving the Levene/Brown-Forsythe Test for Heterogeneity of Spread,\" presented at the Spring Meeting of the International BiometricSociety, Eastern North American Region, March 25–28,2001, Charlotte, NC. Westfall, P., Tobias, R., and Bretz, F. (2000), \"Estimating Directional Error Rates of Step-wise Multiple Comparisons Methods Using Distributed SAS Computing and Variance Reduction,\"submitted for publication; draft available athttp://www.sas.com/rnd/app/papers/directional.pdf.Appendix - The %Distribute System /****************************************************************//**//*NAME:DISTRIBUTE*//*TITLE:DistributedparallelprocessinginSASusingMP*//*CONNECT*//*SYSTEM:ALL*//*PROCS:TEMPLATE*//*PRODUCT:SAS/CONNECT*//*DATA:*//**//*SUPPORT:sasrdtUPDATE:26FEB01*//*REF:See\"Large-ScaleParallelNumericalComputationin*//*theSASSystem\CherylDoninger&RandyTobias.*//****************************************************************//*--------------------------------------------------------------------Themacrosdefinedinthisfilearecenteredaroundthe%Distributemacro,whichusesMPCONNECTtoolsfromSAS/CONNECTsoftwaretoemploydistributedprocessingforproblemsthatconsistofmanyreplicaterunsofafundamentaltask.Forexample,globalintegeroptimizationproblemsareoftensolvedbysteepestascent(thefundamentaltask)withrandomrestarts(thereplicateruns).Similarly,MonteCarlo methodsinstatisticsrelyonasetofpseudo-randomobservations(thereplicateruns)ofoneormoredifficult-to-computeteststatistics(thefundamentaltask).Ifyourproblemcanbefactoredinthisway,then%Distributemayofferausefulapproach. Besidesrunningthefundamentaltaskyouspecifytherequiredreplicatenumberoftimes,%Distributeofferstwoadditionalservices: -Problemsofthetypethat%Distributecanhandleoftenrequireastreamofrandomnumbers,and%Distributecreatesandmanagessuchastreamforeachhost.-Inordertomonitoryourprogram,youwouldliketoknowwhereyourfundamentalprocessingtimeisspent.So %Distributekeepstrackofhowmuchtimeeachhostspendsworkingonitsfundamentaltasksandwaitingfortheclienttogiveitmorework.Inordertouse%Distributesystem,youmustfirstcreatethefollowing: 1._Hostsdataset-containinginformationaboutthehostsyou wanttorunon 2.%RInitmacro-definingtheprogramstoberunoneachhost(1)sayshowtoconnecttoandinitializetothehosts;(2)definesthefundamentaltasktobeperformedoneachhost.Detaileddescriptionsoftheinputisasfollows: _Hostsdataset Defineadatasetnamed_HostsinthecurrentWORKlibrarywithanobservationforeachhostyouwanttodistributethecomputationto.Foreachobservation,settwo(character)variables: VariableName Contents HostScriptNameofthehost signonscriptforhost Thesevariablesareusedontheclient,somakesuretheyarevalidnamesandpathsinthisenvironment %RInitmacro Thisiswhatreallydefinesthedistributedcomputation.needstowrapof(atleast)twomacrodefinitions It %FirstRSub-initializationonthathost %TaskRSub-thefundamentalunitofdistributedcomputationwithinanRSUBMITblock,likeso %macroRInit; rsubmitremote=Host&iHostwait=yesmacvar=Status&iHost; %macroFirstRSub; < %macroTaskRSub; < < endrsubmit;%mend; Itisyourjobtowritethemacros%FirstRSuband%TaskRSubsothattheydefinethefundamentaltask.Asparametersforthesemacros,youcanassumethefollowingglobalmacrovariablestobeavailablewithinthehostSASsession: PredefinedhostglobalmacroRem_HostRem_iHostRem_NIterRem_Seed Contents Nameofthishost Indexofthishostin_HostdatasetSizeofchunk Randomseed,rerandomizedforeachchunk Ifyouneedothermacrovalues(say,&Mac1,&Mac2,and&Mac3)tobepassedfromtheclienttothehost,youshouldspecifythemina%Macrosmacro,likeso: %macroMacros; %syslputrem_Mac1=&Mac1;%syslputrem_Mac2=&Mac2;%syslputrem_Mac3=&Mac3;%mend; The%Distributemacrowilltakecareofrunningthismacroforeachofthehosts,sothatyoucanusethemacros&rem_Mac1,&rem_Mac2,and&rem_Mac3inyourdefinitionsof%FirstRSuband%TaskRSub. Onceyouhavecreatedthe_Hostsdatasetandthe%RInitmacro,youcandistributethefundamentaltaskacrossallthehostswithjustthefollowingfourlinesofcode: %SignOn; %letNIterAll= '.'forhoststhatarecurrentlyworking, '!'forhoststhatarecurrentlywaitingtobeassignedwork,and'X'forhoststhatareunabletodoworkforsomereason. Additionalstatusinformationincludesthenumberoffundamentaltasks submittedandcompletedsofar.Whenthedistributioniscomplete,thesystemwilldisplayabreakdownofhowmuchworkwasdoneoneachhost,togetherwithanover-allsummaryoftheefficiencyofthedistribution.Finally,youcan -usethe%RCollector%RCollectViewmacrostocollectthe resultsfromeachhostintoadatasetorDATAstepviewthatcatenatesallofthem. Furtheranalysisofthecollectedresultscanbeperformedontheclient. -----------------------------------------------------------------------DISCLAIMER: THISINFORMATIONISPROVIDEDBYSASINSTITUTEINC.ASASERVICE TOITSUSERS.ITISPROVIDED\"ASIS\".THEREARENOWARRANTIES,EXPRESSEDORIMPLIED,ASTOMERCHANTABILITYORFITNESSFORA PARTICULARPURPOSEREGARDINGTHEACCURACYOFTHEMATERIALSORCODECONTAINEDHEREIN. ---------------------------------------------------------------------*//* /Namethefilestowhichtowriteanylogandlistingproducedby/theservers. /---------------------------------------------------------------------*/%letDistLog=distribute.log;%letDistLst=distribute.lst; /* /Optionstocontroldistribution-monitoringinformation: /---------------------------------------------------------------------*/%letDIST_DETAIL=1;/*Printtiminginfofromservers?*/%letDIST_STATUS=1;/*Printperiodicstatuslinesforservers?*/%letDIST_DEBUG=0;/*Printdebugginginformation*/ %*********************************************************************;%*MACRO:SignOn*;%*USAGE:%SignOn;*;%*DESCRIPTION:*;%*Thismacrosignsontothehosts,usingtheinformationinthe*;%*_Hostsdataset.Seecommentsaboveforadescription.*;%*NOTES:*;%*Assumes_Hostsdatasethasbeencreatedcorrectly.*;%**;%*********************************************************************;%macroSignOn; %localnotesopt;%letnotesopt=%sysfunc(getoption(notes));optionsnonotes; /* /Retrieveglobalhostinformationfromthe_Hostsdatasetandset/upSeedandStatusvariablesforeachhost. /---------------------------------------------------------------------*/%globalNumHosts; data_Hosts;set_Hosts; Status='!';iHost=_N_; callsymput('NumHosts',trim(left(_N_)));run; %doiHost=1%to&NumHosts; %globalHost&iHostStatus&iHost;%end; data_null_;set_Hosts; callsymput('Host'||trim(left(iHost)),trim(left(Host)));callsymput('Seed'||trim(left(iHost)),round(100000*ranuni(1)));run; data_null_;set_Hosts; select(Status); when('!')callsymput('Status'||trim(left(iHost)),0);when('X')callsymput('Status'||trim(left(iHost)),1);when('.')callsymput('Status'||trim(left(iHost)),2);end; run; /* /Foreachhost,retrievescript,signon,andsetupaRLSlibref/tothehost'sworklibrary. /---------------------------------------------------------------------*/%doiHost=1%to&NumHosts; %putStartingupHost&iHost=&&host&iHost; data_null_;set_Hosts(where=(iHost=&iHost)); callsymput('Script',trim(left(Script)));run; optionsnotes; procprinttolog=\"&DistLog\"print=\"&DistLst\";run;filenameScript\"&Script\"; signonremote=Host&iHostscript=Script;filenameScript; libnameRWork&iHostslibref=WORKserver=Host&iHost;%letStatus&iHost=-1;%RInit; %letstimeropt=%sysfunc(getoption(stimer));optionsnostimer;procprintto;run;options&stimeropt;optionsnonotes;%end; /* /Printtheinitialstatusofeachhost. /---------------------------------------------------------------------*/ %letStatLine=; data_null_;set_Hosts; callsymput('StatLine',symget('StatLine')||trim(left(Status)));run; %putStat:&StatLine;options¬esopt;%mend; %*********************************************************************;%*MACRO:SignOff*;%*USAGE:%SignOff;*;%*DESCRIPTION:*;%*Thismacrosignsoffthehosts.*;%*********************************************************************;%macroSignOff; %localnotesopt;%letnotesopt=%sysfunc(getoption(notes));optionsnonotes; data_null_;set_Hosts; callsymput('Host'||trim(left(iHost)),trim(left(Host))callsymput('NumHosts',trim(left(_n_))select(Status); when('!')callsymput('Status'||trim(left(iHost)),0);when('X')callsymput('Status'||trim(left(iHost)),1);when('.','?')callsymput('Status'||trim(left(iHost)),2);end; run; %doiHost=1%to&NumHosts; %putStoppingHost&iHost=&&host&iHost; optionsnotes; procprinttolog=\"&DistLog\"print=\"&DistLst\";run;%letStatus&iHost=-1; signoffremote=Host&iHostmacvar=Status&iHost;/* rsubmitremote=Host&iHostwait=yesmacvar=Status&iHost;; endrsubmit;*/ %letstimeropt=%sysfunc(getoption(stimer));optionsnostimer;procprintto;run;options&stimeropt;optionsnonotes;%end; %PrintStatus;options¬esopt;%mend; );); %*********************************************************************;%*MACRO:_TaskRSub*;%*USAGE:Internalmacroonly*;%*DESCRIPTION:*;%*Thismacrosubmitsachunkofthefundamentaltasktothehost*;%*withindexnHost.*;%*NOTES:*;%*Assumesthemacro%TaskRSubhasbeenpreviouslydefinedonthe*;%*host.DothisintheRInitmacrodefinedontheclient.*;%*********************************************************************;%macro_TaskRSub;%globalDIST_DEBUG; %if(^&DIST_DEBUG)%then%do; procprinttolog=\"&DistLog\"print=\"&DistLst\";run;%end;%letStatus&nHost=-1; /* /Keeptrackofwhat'sbeensubmittedtoeachhost*on*the/hoststhemselves,toensurewiresdon'tgetcrossed. /------------------------------------------------------------------*/rsubmitremote=Host&nHostwait=yes; %nrstr(%%)letTaskNSubm=%eval(&TaskNSubm+&rem_niter); %nrstr(%%)sysrputNSubm&rem_iHost=&TaskNSubm;endrsubmit; %syslputGlobalNSubm=&NSubm; /* /Submitnotonlythetasktothishost,butalsotherandom/seedandtimemanagementcode. /------------------------------------------------------------------*/rsubmitremote=Host&nHostwait=nomacvar=Status&nHost SYSRPUTSYNC=yes;data_null_; old_Seed=&rem_Seed; new_Seed=round(100000*ranuni(&rem_Seed));callsymput('rem_Seed',trim(left(new_Seed)));run; data_TimeBetween; Host=\"&rem_Host\";now=datetime(); callsymput('_TimeStart_',now); TimeBetween=now-symget('_TimeEnd_');run; data_null_;callsymput('_TimeStart_',datetime());run;%TaskRSub; data_TimeEnd; Host=\"&rem_Host\";now=datetime(); callsymput('_TimeEnd_',now); Time=now-symget('_TimeStart_');run; data_Time;merge_TimeBetween_TimeEnd;data_TimeAll;set_TimeAll_Time;run; %nrstr(%%)letTaskNDone=%eval(&TaskNDone+&rem_niter);%nrstr(%%)sysrputNDone&rem_iHost=&TaskNDone;endrsubmit; %letNSubm=%eval(&NSubm+&NIter); %if(^&DIST_DEBUG)%then%do; optionsnostimer;procprintto;run;optionsstimer;%end; %mend; %*********************************************************************;%*MACRO:PrintStatus*;%*USAGE:Internalmacroonly*;%*DESCRIPTION:*;%*Thismacroprintsalinesummarizingthecurrentstatusof*;%*eachserver:*;%*'.'forhoststhatarecurrentlyworking,*;%*'!'forhoststhatarecurrentlywaitingtobeassigned*;%*work,and*;%*'X'forhoststhatareunabletodoworkforsomereason.*;%*Additionalstatusinformationincludesthenumberof*;%*fundamentaltaskssubmittedandcompletedsofar.*;%*********************************************************************;%macroPrintStatus; %localnotesopt;%letnotesopt=%sysfunc(getoption(notes));optionsnonotes; data_Hosts;set_Hosts; xStatus=1*symget('Status'||trim(left(iHost))); select(xStatus); when(0)Status='!';when(1)Status='X';when(2)Status='.'; otherwisedo;Status='?';putiHost=xStatus=;end;end; NSubm=1*symget('NSubm'||trim(left(iHost)));NDone=1*symget('NDone'||trim(left(iHost)));run; %letStatLine=; data_null_;set_Hosts; callsymput('StatLine',symget('StatLine')||trim(left(Status)));run; procsummarydata=_Hosts; varNSubmNDone; outputout=_SHostssum=NSubmNDone;data_null_;set_SHosts; callsymput('StatLine',symget('StatLine') ||':('||trim(left(put(NSubm,best20.)))||','||trim(left(put(NDone,best20.)))||')/'||trim(left(put(&NIterAll,best20.)))); run; %putStat:&StatLine;options¬esopt;%mend; %*********************************************************************;%*MACRO:Distribute*;%*USAGE:%Distribute*;%*DESCRIPTION:*;%*Thismacroperformstheactualdistribution,assumingthatyou*;%*havefirstcreatedthe_Hostsdatasetandthe%RInitmacro,*;%*asdescribedintheheadercommentsabove.*;%*********************************************************************;%macroDistribute; %globalDIST_DETAILDIST_STATUSDIST_DEBUG; %doiHost=1%to&NumHosts; %globalHost&iHostSeed&iHostStatus&iHostNSubm&iHostNDone&iHost TimeWork&iHostTimeWait&iHostTimeFreq&iHost_ElapsedTime; %end; %if(^&DIST_DEBUG)%then%do;%localnotesopt;%letnotesopt=%sysfunc(getoption(notes)); optionsnonotes;%end;%else%do; optionsmprintmtrace;%end; /* /Startthetimer. /---------------------------------------------------------------------*/data_null_;callsymput('TimeStart',trim(left(datetime())));run;/* /Setupmacrovariableswiththenamesandrandomnumberseedsfor/allthehosts,andinitializemonitoringinformation. /---------------------------------------------------------------------*/data_null_;set_Hosts; callsymput('Host'||trim(left(iHost)), trim(left(Host))); callsymput('Seed'||trim(left(iHost)), trim(left(round(100000*ranuni(1))))); callsymput('NumHosts', trim(left(_N_))); select(Status); when('!')callsymput('Status'||trim(left(iHost)),0);when('X')callsymput('Status'||trim(left(iHost)),1);when('.')callsymput('Status'||trim(left(iHost)),2);end; callsymput('NSubm'||trim(left(iHost)),'0');callsymput('NDone'||trim(left(iHost)),'0');run; %letNSubm=0; /* /Start-upphase:submittheinitializationtasktoeachhost,and/alsothefirstfundamentaltask. /---------------------------------------------------------------------*/%doiHost=1%to&NumHosts; %if(^&DIST_DEBUG)%then%do; procprinttolog=\"&DistLog\"print=\"&DistLst\";run;%end; optionsremote=Host&iHost;%syslput%syslput%syslput%syslput%Macros; rsubmitremote=Host&iHostwait=yes; %FirstRSub; data_null_; now=datetime(); callsymput('_TimeEnd_',now);run; data_TimeAll;if(0);run; %nrstr(%%)letTaskNSubm=0; %nrstr(%%)sysrputNSubm&rem_iHost=&TaskNSubm;%nrstr(%%)letTaskNDone=0; %nrstr(%%)sysrputNDone&rem_iHost=&TaskNDone;endrsubmit; %letnHost=&iHost;%_TaskRSub; %if(^&DIST_DEBUG)%then%do; optionsnostimer;procprintto;run;optionsstimer;%end;%if(&DIST_STATUS)%then%PrintStatus;%letNSubm=0;%letNDone=0; %dojHost=1%to%eval(&iHost); %letNSubm=%eval(&NSubm+&&NSubm&jHost);%letNDone=%eval(&NDone+&&NDone&jHost);%end;%dojHost=1%to%eval(&iHost); %letLastStat&jHost=&&Status&jHost;%end; /* /Wealsorecyclethroughprevioushostshere,resubmittingtasks/tothemifthey'rereadyformore:thissavesalittletime/whensomehostsfinishtheirfirsttaskbeforethestart-up/phaseiscomplete. /------------------------------------------------------------------*/%dojHost=1%to%eval(&iHost); %if((&&LastStat&jHost=0)&(&NDone<&niterall))%then%do; %letnHost=&jHost;%_TaskRSub; rem_Hostrem_Seedrem_niterrem_iHost =&&host&iHost;=&&seed&iHost;=&niter;=&iHost; %end;%end;%end; %if(&DIST_STATUS)%then%PrintStatus;%letNSubm=0;%letNDone=0; %dojHost=1%to%eval(&NumHosts); %letNSubm=%eval(&NSubm+&&NSubm&jHost);%letNDone=%eval(&NDone+&&NDone&jHost);%end; /* /MonitortheMACVARsStatus1,Status2,etc.towatchthejobs /finish.Astheydo,resubmitnewtasksforfreehoststoworkon./---------------------------------------------------------------------*/%letRunning=1; %do%while(%length(&Running)); waitfor_any_ %doiHost=1%to&NumHosts; %if(&&LastStat&iHost=2)%then%do; Host&iHost%end;%end;; %if(&DIST_STATUS)%then%PrintStatus;%letNSubm=0;%letNDone=0; %dojHost=1%to%eval(&NumHosts); %letNSubm=%eval(&NSubm+&&NSubm&jHost);%letNDone=%eval(&NDone+&&NDone&jHost);%end;%letRunning=; %doiHost=1%to&NumHosts; %letLastStat&iHost=&&Status&iHost; %if(&&LastStat&iHost=2)%then%letRunning%end; =&Running&iHost; %doiHost=1%to&NumHosts; %if((&&LastStat&iHost=0)&(&NSubm<&niterall))%then%do; %letnHost=&iHost;%_TaskRSub; %letLastStat&iHost=&&Status&iHost; %if(&&LastStat&iHost=2)%then%letRunning %letNSubm=0;%letNDone=0; %dojHost=1%to%eval(&NumHosts); %letNSubm=%eval(&NSubm+&&NSubm&jHost);%letNDone=%eval(&NDone+&&NDone&jHost);%end;%end;%end;%end; /* /Alltasksarefinished:retrievethetiminginformationfromeach/ofthehosts,... /---------------------------------------------------------------------*/%doiHost=1%to&NumHosts; %letLastStat&iHost=&&Status&iHost;%if(&&LastStat&iHost=0)%then%do; %if(^&DIST_DEBUG)%then%do; procprinttolog=\"&DistLog\"print=\"&DistLst\";run;%end; =&Running&iHost; rsubmitremote=Host&iHostwait=yes; procsummarydata=_TimeAll; varTimeTimeBetween; outputout=_TimeSummmean=WorkWait;run; data_null_;set_TimeSumm; callsymput('_TimeWork',trim(left(Work)));callsymput('_TimeWait',trim(left(Wait)));callsymput('_TimeFreq',trim(left(_FREQ_)));run; %nrstr(%%)sysrputTimeWork&rem_iHost=&_TimeWork;%nrstr(%%)sysrputTimeWait&rem_iHost=&_TimeWait;%nrstr(%%)sysrputTimeFreq&rem_iHost=&_TimeFreq;endrsubmit; %if(^&DIST_DEBUG)%then%do; optionsnostimer;procprintto;run;optionsstimer;%end;%end;%end; /* /...addittothe_Hostsdataset,... /---------------------------------------------------------------------*/ data_null_; Elapse=datetime()-&TimeStart; callsymput('_ElapsedTime',trim(left(put(Elapse,best.))));run; dataTimeSumm;set_Hosts(keep=HostiHost); keepiHostHostNIterTimeWorkEstElapsedEffTimeWaitTotalWorkTotalWait; NIter=&NIter*symget('TimeFreq'||trim(left(iHost)));TimeWork=1*symget('TimeWork'||trim(left(iHost)));TimeWait=1*symget('TimeWait'||trim(left(iHost)));EstElapsedEff TotalWorkTotalWaitrun; ==== (&NIterAll/&NIter)*TimeWork; (EstElapsed/&_ElapsedTime)/&NumHosts;(NIter/&NIter)*TimeWork;(NIter/&NIter)*TimeWait; /* /...andreportit. /---------------------------------------------------------------------*/ %if(&DIST_DETAIL)%then%do; procsortdata=TimeSummout=TimeSumm;bydescendingTimeWork;procprintdata=TimeSummlabelnoobs; formatTimeWorktime.;formatTimeWaittime.;formatEstElapsedtime.;formatEffpercent.;labelNIter=\"No.Iter\";labelTimeWork=\"WorkTime/&NiterIter\";labelTimeWait=\"WaitTime/&NiterIter\"; labelEstElapsed=\"EstimatedTimeforEntireProblem\";labelEff=\"DistributionEfficiency\"; variHostHostNIterTimeWorkEstElapsedEffTimeWait;run;%end; procsummarydata=TimeSumm; varTotalWorkTotalWait; outputout=TotalTimeSum=TotalWorkTotalWait;data_null_;setTotalTime; Elapsed=&_ElapsedTime;Eff=(TotalWork/Elapsed)/&NumHosts;put\"\"; putputputputputputrun; \"\"\"\"\"\";\"\"; Totalelapsedtime: Cumulativeworkingtime:Cumulativewaitingtime:Scalingefficiency:\"\"\"\"Elapsedtime.;TotalWorktime.;TotalWaittime.;Effpercent8.2; %if(^&DIST_DEBUG)%then%do; options¬esopt;%end;%mend; %*********************************************************************;%*MACRO:RCollect*;%*USAGE:%RCollect(< %if(^%length(%left(%trim(&n))))%then%letn=&NumHosts;%doiHost=1%to&NumHosts; %globalHost&iHostStatus&iHost;%end;data&to; set %doi=1%to&N; %if(&&Status&i=0)%then%do; RWork&i..&from%end;%end;; run;%mend;/* Example Inlinearmodels,anexperimentaldesigncanbeconciselyrepresentedasamatrixX,andtheeigenvaluesofX'Xgiveinformationabouttheefficiencyofthedesign.Inparticular,theso-calledE-efficiencyofadesignisrelatedtothesmallesteigenvalueofX'X.SupposeyouareinterestedintheE-efficiencyofasetofknormallydistributedpointsinR^k,fordifferentvaluesofk.CreatingrandomnormaldataandcomputingtheminimumeigenvaluerequiresonlyacoupleoflinesofSAS/IMLcode x=rannor((1:&k)`*(1:&k));e=eigval(x`*x)[><]; butittakesalargesampletoestimatetheexpectedvalueofewithprecision,andifyouwanttolookattheexpectedvalueformany valuesofk,itcantakeawhile.Inordertodistributethejob,youfirstneedtowraptheabovecodesothatitcanberunby%Distributeonthehosts. %macroRInit; rsubmitremote=Host&iHostwait=yesmacvar=Status&iHost; %macroFirstRSub; optionsnonotes; dataResults.Sample&rem_iHost; if(0);run;%mend; %macroTaskRSub; prociml; x=rannor(&rem_Seed); freedata; doi=1to&rem_Niter; x=rannor((1:&rem_Dimen)`*(1:&rem_Dimen));e=eigval(x`*x)[><];data=data//e;end;createSamplevar{E1};appendfromdata;quit; dataResults.Sample&rem_iHost; setResults.Sample&rem_iHostSample;run;%mend; endrsubmit;%mend; Definethismacroontheclientandmakea_Hostsdatasetas describedabove.Onefinalbitofset-upisrequired:youneedtotell%DistributetopassthedimensionofXdowntothehosts.Notethatwehaveassumedin%TaskRSubabovethatthisisavailableinamacro&rem_Dimen.Assumingitsvalueisassociatedwithamacrovariable&Dimenontheclient,thefollowingdefinitionfor%Macroswilldothetrick. %macroMacros; %syslputrem_Dimen=&Dimen;%mend; Oncethesethreecomponents(_Hosts,%Rinit,and%Macros)havebeensetup,thefollowingcodewillruna10-dimensionalsimulation1,000,000timesoverthehostsinchunksof2,000. %inc\"Distribute.sas\";%letNIter=2000;%letNIterAll=1000000;%SignOn; %letDimen=10;%Distribute; For25hosts,thefirstfewstatuslineslooklikethis: Stat:Stat:Stat:Stat:Stat: .!!!!!!!!!!!!!!!!!!!!!!!!:..!!!!!!!!!!!!!!!!!!!!!!!:...!!!!!!!!!!!!!!!!!!!!!!:....!!!!!!!!!!!!!!!!!!!!!:.....!!!!!!!!!!!!!!!!!!!!: (2000,0)/1000000(4000,0)/1000000(6000,0)/1000000(8000,0)/1000000(10000,0)/1000000 Thebottomlinehere,forexample,indicatesthatthefirstfivehostsareworkingontheirtasksandtheothersarewaitingtobegiventasks.Afterawhile,thetasksonthefirsthostswillbegintofinishup,andthey'llbegivenmoretodo: Stat:!..........!!!!!!!!!!!!!!:(22000,2000)/1000000Stat:.!..........!!!!!!!!!!!!!:(26000,4000)/1000000Stat:..!!.........!!!!!!!!!!!!:(30000,8000)/1000000 Eventually,allhostswillhavebeengiventasksandthestatuslinewillindicatewhichonesfinishtheirswhileotherhostsarebeingservicedbytheclient: Stat:Stat:Stat:Stat: ..!..........!!.....!!...:....!!...!............!..:...!...!........!......!.:..........!!...!.........: (682000,642000)/1000000(692000,650000)/1000000(702000,660000)/1000000(712000,668000)/1000000 Towardstheendoftherun,mosthostswillbewaitingwhilethelastchunkstofinishup,andastatuslinewillbeprintedaseachonedoes: Stat:Stat:Stat:Stat:Stat: !.!!!!.!!!!!!!.!!!!!..!.!:!.!!!!.!!!!!!!!!!!!!!!!.!:!.!!!!.!!!!!!!!!!!!!!!!!!:!!!!!!.!!!!!!!!!!!!!!!!!!:!!!!!!!!!!!!!!!!!!!!!!!!!: (1000000,988000)/1000000(1000000,994000)/1000000(1000000,996000)/1000000(1000000,998000)/1000000(1000000,1000000)/1000000 Forthe25UNIXhoststhattheabovetestwasrunon,thetimingresultslooklikethis: WorkTime/2000Iter0:00:040:00:030:00:030:00:03 ... 0:00:030:00:03 EstimatedTimeforEntireProblem0:35:280:27:500:27:190:27:08 ... 0:25:050:25:02 WaitTime/2000Iter0:00:010:00:010:00:000:00:01 ... 0:00:010:00:01 iHost20815...2212 HostUNIX68UNIX73UNIX49UNIX16 ...UNIX71UNIX79 No.Iter30000420004800044000 ...3600040000 DistributionEfficiency 92%72%71%70%...65%65%0:01:330:26:220:07:59 Totalelapsedtime: Cumulativeworkingtime:Cumulativewaitingtime: (Thenamesofthemachineshavebeenchangedtoprotecttheinnocent.)Mostofthemachinestookabout3secondsforeachchunkof2000 samples,butUNIX68tookabitlongerandwasconsequentlygivenfewerchunkstodo.Thebottomlineisthatajobthatwouldhavetakenaboutahalf-hourononemachinetookjustaminuteandahalfdistributedacross25machines. YoucanuseaDATAstepviewtocollecttheresultsfromall25hostsintoonedatasetandanalyzethem: %RCollect(Sample,Sample/view=Sample); procmeansdata=Samplenmeanstderrlclmuclm; vare1;run; Theresultsindicatethattheminimumeigenvaluefora10-dimensional,10-pointrandomnormaldesignisabout0.0725withabout4digitsofprecision. TheMEANSProcedureAnalysisVariable:E1 Lower95%Upper95% NMeanStdErrorCLforMeanCLforMean --------------------------------------------------------------10000000.07245930.0001068800.07224980.0726688--------------------------------------------------------------Theoriginalproblemwastoexaminetherelationshipoftheminimumeigenvaluetothedimensionforarangeofdimensions.Onceallthe set-uphasbeendone,itisasimplemattertowrapamacroaroundthedistributionandcollectioncode: %macroMinEVal; optionsnonotes; dataMinEval;if(0);run;%doDimen=1%to30; %Distribute; %RCollect(Sample,Sample/view=Sample);procsummarydata=Sample; vare1; outputout=_tempmean=MeanStdErr=StdErr LCLM=LCLMUCLM=UCLM; data_temp;set_temp;Dimen=&Dimen;dataMinEval;setMinEval_temp;run;%end;optionsnotes;%mend;%MinEval; Thissimulationwouldrequire40hourstorunonasinglemachine,butittooklessthantwohourswhendistributedacross25UNIXhosts. Theresultsindicatethattheexpectedvalueoftheminimumeigenvalueisinverselyrelatedtothedimension.*/ 因篇幅问题不能全部显示，请点此查看更多更全内容