Model Checking an Entire Linux Distribution for Security Violations

1. ACSAC 2005 Model Checking an Entire Linux Distribution for Security Violations Jacob West,Security Research Group, Fortify Software Work by Benjamin Schwarz, Hao Chen, David Wagner, Geoff Morrison, Jacob West, Jeremy Lin and Wei Tu

2. Outline • Introduction • MOPS Background • Analyzing Red Hat 9 • Tool performance • Human performance • Security properties • Vulnerability Examples • TOCTTOU • Standard File Descriptors • Temporary Files • strncpy() • Results

3. Introduction • Over 50% of security vulnerabilities caused by coding errors • Automated detection possible • Rapidly expanding field • Academic and commercial • Feasible at large scale

4. MOPS(MOdelchecking Programs for Security properties) • Static analysis for security • C programs • Enforce temporal safety rules

5. Analyzing Red Hat 9:Overview • Tool performance • Analysis of large code base feasible • Compaction improves performance • Reasonable resource requirements • Human performance • Integration with existing build processes • False positives • Easy-to-review error traces • Grouped error traces • Security properties • Temporal safety properties • Employable by other tools • Iteratively refined for low false positives

6. Analyzing Red Hat 9:Tool Performance • Red Hat 9: 839 packages, 60 million TLOC • 732 packages (87%) • 107 failures caused by parse errors • 73 packages contained C++ code • 34 packages used unsupported C99 constructs • Compaction improves performance • Only consider relevant operations • Reasonable resource requirements • TOCTTOU takes about 10 hours on P4 1.5 GHZ / 1GB

7. Analyzing Red Hat 9:Human Performance • Integration with existing build processes • Integrated with rpmbuild, make • Interposed on gcc • Analyze multiple packages easily • False positives • Relatively low, permits human review • Easy-to-review error traces • Navigate code quickly to verify error traces • Grouped error traces • Understand multiple traces through representative samples

8. Analyzing Red Hat 9:Security Properties • Temporal safety properties • Security properties expressed as Finite State Automata (FSA) • Pattern variables • e.g. foo(x); bar(x); where x is the same • Iteratively refined to reduce false positives • Employable by other tools • Properties include • TOCTTOU: Time-of-check, to time-of-use race conditions • Standard File Descriptors: Vulnerable uses of stdin, stdout and stderr • Temporary Files: Insecure creation of temporary files • strncpy(): Dangerous uses of strncpy()

9. Security Properties :TOCTTOU • Time-of-check to time-of-use race conditions occur when a program checks the access permission of an object and, if the check succeeds, makes a privileged system call on the object. • Example: if (access(pathname, R_OK) == 0) fd = open(pathname, O_RDONLY);

10. Security Properties :TOCTTOU • Checks: access(), stat(), etc. • Uses: creat(), open(), unlink(), etc.

11. Vulnerability Example:TOCTTOU – binutils :: ar exists = lstat (to, &s) == 0; /* Use rename only if TO is not a symbolic link and has only one hard link. */ if (! exists || (!S_ISLNK (s.st_mode) && s.st_nlink == 1)){ ret = rename (from, to); if (ret == 0) { if (exists) { chmod (to, s.st_mode & 0777); if (chown (to, s.st_uid, s.st_gid) >= 0) { chmod (to, s.st_mode & 07777); } ...

12. Security Properties: Standard File Descriptors • Since the kernel does require that stdin, stdout and stderr point to terminal devices, an attacker may cause a victim program open one of them to a sensitive file. • Example /* victim.c */ fd = open("/etc/passwd", O_RDWR); if (!process_ok(argv[0])) perror(argv[0]); /* attack.c */ int main(void) { close(2); execl("victim", "foo:<pw>:0:1:Super-User-2:...", NULL); }

13. Security Properties: Standard File Descriptors • States correspond to the status of the three standard file descriptors and transitions occur on a "safe" open (/dev/null and /dev/tty). open(…) open(…)

14. Vulnerability Example: Standard File Descriptors - gnuchess void BookBuilder(short depth, ...){ FILE *wfp,*rfp; if (depth == -1 && score == -1) { if ((rfp = fopen(BOOKRUN,"r+b")) != NULL) { printf("Opened existing book!\n"); } else { printf("Created new book!\n"); wfp = fopen(BOOKRUN,"w+b"); fclose(wfp); if ((rfp = fopen(BOOKRUN,"r+b")) == NULL) { printf("Could not create %s file\n", BOOKRUN); return; } ...

15. Security Properties:Temporary Files • Because many of the functions in the C standard library that create temporary files are insecure an adversary that is able to predict the filename can gain control of the file by precreating it. • Example fd = mkstemp(action_file_name); ... unlink(action_file_name);

16. Security Properties:Temporary Files • tmpnam(), tempnam(), mktemp() and tmpfile() are always unsafe • mkstemp() is safe if the generated filename is not used

17. Vulnerability Example:Temporary Files - yacc static void open_files() { ... fd = mkstemp(action_file_name); if (fd < 0 || (action_file = fdopen(fd, "w")) == NULL){ ...open_error(action_file_name); } } void open_error(char *filename) { warnx("f - cannot open \"%s\"", filename); done(2); } void done(int k) { ... if (action_file_name[0]) unlink(action_file_name);

18. Security Properties:strncpy() • First strncpy() encourages off-by-one errors if the programmer is not careful to compute the value of n precisely. Secondly, because the function does not automatically null-terminate a string in all cases it is a common mistake for a program to create unterminated strings during its execution. • Example buf[sizeof(buf)-1] = '\0'; strncpy(buf, ..., sizeof(buf));

19. Security Properties:strncpy()

20. Vulnerability Example:strncpy() - xloadimage newopt->info.dump.type = argv[++a]; ... dumpImage(dispimage, dump->info.dump.type, dump->info.dump.file, verbose); void dumpImage(Image *image, char *type, char *filename, int verbose) { int a; char typename[32]; char *optptr; optptr = index(type, ','); if (optptr) { strncpy(typename, type, optptr - type); typename[optptr - type] = '\0'; ... }

21. Results • 1358 strncpy() warnings; 53 audited; 11 real bugs* • 200 human hours found 108 real bugs in 50 million lines of code • Order of magnitude larger in scale than previous academic work • Static analysis will be feasible and integral part of building systems

22. Questions? Want to talk more about software security? jwest@fortifysoftware.com