CCPR Computing Services Workshop 1: Programming Basics, Unix, Remote Computing October 13, 2004

1. CCPR Computing ServicesWorkshop 1:Programming Basics, Unix, Remote ComputingOctober 13, 2004

2. Part 1: Programming Basics • Motivation • Before you start coding • Programming Conventions • Documentation • Names, comments • Directory Structure • Basic Constructs • Miscellaneous (debugging, cross-checking results)

3. Motivation • Facilitate research • Save time • Cleaner code • Easily share programs • Basic Concepts MUCH better programming

4. Programming Conventions • What are conventions? • Examples • Who cares? • Readability of code • Organization • Transferring code to others • Apply conventions consistentlyto: • variable names and function names • comments • directory structure

5. Before you start coding… • THINK • WRITE down the problem • WRITE down the algorithm in English (not code) • Modularity • Comments • Create test (if reasonable) • TRANSLATE one section to code • TEST the SECTION thoroughly • Translate/Test next section, etc.

6. Documentation - File Header #Laura Piersol (laurapiersol@ccpr.ucla.edu) #HRS project #/u/socio/laurapiersol/HRS/ #October 11, 2004 #Python version 2.4 #Stata version 8 #Purpose: Create and merge two datasets in Stata, # then convert data to SAS #Input programs: # HRS/staprog/H2002.do, # HRS/staprog/x2002.do, # HRS/staprog/mergeFiles.do #Output: # HRS/stalog/H2002.log, # HRS/stalog/x2002.log, # HRS/stalog/mergeFiles.log # HRS/stadata/Hx2002.dta # HRS/sasdata/Hx2002.sas #Special instructions: Check log files for errors # check for duplicates upon new data release • File header includes: • Name • Project • Project location • Date • Version of software • Purpose • Inputs • Outputs • Special Instructions

7. Naming Conventions • Not a detail! • Good names clarify your code • Portray meaning/purpose • Adopt a convention and BECONSISTENT

8. Naming Conventions, cont. • Use language standard (if it exists) • If no standard, pick one and BE CONSISTENT Functions: getStats, calcBetas, showResults Scalar variables: scPi, scGravity, scWorkHours String variables: stName, stCareer Global variables: _Country, _Nbhd • Be aware of language-specific rules • Max length, underscore, case, reserved words

9. Naming Conventions, cont. • Differentiating log files: • Programs MergeHH.sas, MergeHH.do • Log files MergeHHsas.log, MergeHHsta.log • Meaningful variable names: • LogWt vs. var1 • AgeLt30 vs. x • Procedure that cleans missing values of Age: • fixMissingAge • Matrix multiplication X transpose times X • matXX

10. Commenting Code • Good code is SELF-COMMENTING • Naming conventions, structure, header explain 95% • Comments explain • PURPOSE, not every detail • TRICKS • (good) reasons for unusual coding • CommentsDO NOT • fix sloppy code • translatesyntax

11. Commenting Code - Stata example SAMPLE 1 program def function1 foreach v of varlist _all { local x = lower("`v'") if `"`v'"' != `"`x'"' { rename `v' `=lower("`v'")' } } end No conventions, comments, structure SAMPLE 2 *Convert names in dataset to lowercase. program deflowerVarNames foreach v of varlist _all { local LowName = lower("`v'") *If variable is already lowercase, *rename statement throws error. if `"`v'"' != `"`LowName'"' { rename `v' `=lower("`v'")' } } end Comments: succinct and not overdone Names: lowerVarNames -action word for program -distinct use of case LowName -descriptive -distinct use of case v -looping variable and short scope -non-descriptive, but does not detract from meaning! Structure- indentations, parentheses lined up!

12. Directory Structure • A project consists of many different types of files • Use folders toSEPARATE filesin a logical way • Be consistent across projects if possible • ATTIC folder for older versions

13. Miscellaneous Tips • BACKUP! Weekly zip file stored externally • README.txt file to describe folder • BE ORGANIZED • CROSS-VERIFY results • Something not working? • Remember the computer is following YOUR directions… go back to your code

14. Programming Constructs • Tools to simplify and clarify your coding • Available in virtually all languages

15. Constructs - Looping • Repeat section of code • START value, INCREMENT, STOP value • Example- • convert uppercase to lowercase for each variable in a dataset

16. Constructs – Looping Examples C for loop: Start with x=1, Increment = x+1, Stop when x==10 for(x=1; x<10; x++) { …code… } PERL while loop: Start with count= 1, Increment= count+1, Stop when count==11 $count=1; while ($count<11) { print "$count\n"; $count++; } STATA foreach loop: Start = first variable in varlist, Increment = next variable in varlist, Stop =last variable in varlist foreach v of varlist _all { local LowName = lower("`v'") if `"`v'"' != `"`LowName'"' { rename `v' `=lower("`v'")' } }

17. Constructs - If/then/else • Execute section of code if condition is true: ifconditionthen {execute this code if condition true} end • Execute one of two sections of code: ifconditionthen {execute this code if condition true} else {execute this code if condition false} end

18. Constructs - Elseif/case • Elseif - Execute one of many sections of code: ifcondition1then {execute this code if condition1 true} elseifcondition2 then {execute this code if condition2 true} else {execute this code if condition1, condition2, condition3 are all false} end • Case- same idea, different name casecondition1 then {execute this code if condition1 true} case condition2 then {execute this code if condition2 true} etc.

19. Constructs - And, or, xor AND - BOTH conditions must be true results in True OR - AT LEAST ONE condition must be true results in True XOR - EXACTLY ONE condition must be true for statement to be true

20. Constructs - Break • Stop execution of program • Examples: • Debugging. If particular error occurs then break. • Parameters in function call are nonsensical. Print error and break.

21. Constructs - keywords • Looping - for, foreach, do, while • If statements – if, then, else, case • And/Or/Xor – logical, and, or, xor, &, | • Break – exit, break

22. PART 2: Unix • Motivation • Basic Commands • Job submission and management • Pipes • Unix Shell • Script files

23. Unix • Motivation • A quick history • Unix variants • (AIX, Solaris, FreeBSD, Linux) • Where? • Nicco (SSC’s server) • CCPR’s linux cluster (coming soon) • CCPR’s data server (coming soon)

24. Unix – Basic Commands Getting Help

25. Unix – Basic Commands File Management

26. Unix – Basic Commands Directory Management

27. Unix – Basic Commands Using Previous Commands

28. Unix – Basic Commands Other Useful Unix Tools

29. Unix – Basic Commands Disk Usage

30. Editing Files in Unix • Vi Editor and Emacs • Neither are user-friendly for starters • Look at CCPR internet when you start • Best way to learn is to start editing a document • Once you get used to them, they’re easy and fast to use.

31. Being nice • Always run your jobs “nicely” • Prevents interfering with other users • Precede command with “nice +19” (no quotes) user@nicco%nice +19 stata –b jobfile.do

32. Job Submission in Unix • Interactive user@nicco%stata • Foreground jobs user@nicco%nice +19 stata –b do jobfile.do user@nicco%nice +19 sas jobfile.sas • Background jobs – & user@nicco%nice +19 stata –b do jobfile.do& user@nicco%nice +19 sas jobfile.sas& • Background jobs with logoff – nohup user@nicco%nohup nice +19 stata –b do jobfile.do& user@nicco%nohup nice +19 sas jobfile.sas&

33. Job Management

34. Pipes • “|” redirects command 1 outputcommand 2 input • Extends to more than 2 commands

35. Unix Shell • What’s a Unix Shell? • The Unix shell is the program that provides the interface between the user and the kernel • What’s a shell script? • A list of commands put into a file that can be interpreted by the Unix Shell. • What are scripting languages? • Generally easier to code but less efficient • Shell scripts, Perl, Python

36. Remote Computing *See http://www.ccpr.ucla.edu/asp/compserv.asp, XP users get latest version

37. Remote Computing * http://computing.sscnet.ucla.edu/training/tutorial_SSH.htm ** http://computing.sscnet.ucla.edu/training/tutorial_samba.htm

38. Finally… • Questions and Feedback