32-bit Virtual Memory Constraints in Windows: an Update

1. 32-bit Virtual Memory Constraints in Windows: an Update Mark Friedman Demand Technology 1020 Eighth Avenue South, Suite 6, Naples, FL 34102 USA phone: (239) 261-8945 fax: (239) 261-5456 e-mail: markf@demandtech.com http://www.demandtech.com

2. Outline • Topics • Virtual Memory concepts • Virtual Memory constraints in 32-bit Windows • /3 GB option • Physical Address Extension (PAE) • Address Windowing Extensions (AWE) • 64-bit relief

3. Intel x86

4. Intel x86 in PAE mode

5. Virtual Memory Manager • 4 GB virtual per process address space • Lower 2 GB - Private • Page 0 reserved • Code pages • Heap • Upper 2 GB - System • System code • Shared dlls • System cache

6. Virtual Memory Manager • All address spaces share the virtual addresses from the same upper 2 GB of the System area • All virtual addresses in the lower 2 GB User area are unique • Shared memory in the System range can be used for IPC

7. Virtual Memory map • Paged • Memory mapped files • Shared DLLs • File Server • IIS (html, jpg, gif) • Page tables; kernel thread stacks • nonPaged: • I/O buffers used by device drivers • TCP Session data

8. Virtual Memory Manager • Commit Limit • A hard limit on the number of Virtual Memory pages that the system will allocate • Sizeof (RAM) + paging file(s) • Memory allocations start to fail as % Committed Bytes in Use 100% • For smaller systems (< 2 GB RAM), Committed Bytes tends to become exhausted prior to running out of VM

9. Monitoring Committed Bytes

10. Monitoring Process\(*)Virtual Bytes

11. Virtual Memory Manager • Commit Limit • But page files are extendible! • Should you or shouldn’t you?

12. Paging file extension

13. System memory pools • No hard limits on the sizes of the system pools • One reserved area of virtual memory can fill up faster than the others;

14. Monitoring the System Pools

15. Virtual Memory map • Virtual memory map is dynamic, subject to adjustment by the OS • One reserved area of virtual memory can fill up faster than the others • When any one area fills, VMM routines can empty the file cache to acquire more vm for the paged/nonpaged pools Paged Pool Cache PTEs nonPaged Pool

16. Default Paged and Nonpaged pool size can be overridden:

17. Default Paged and Nonpaged pool sizes can be overridden: • Dynamic adjustment of system virtual memory when one pool is exhausted. • Workloads most vulnerable: • Terminal server with an excess of processes & threads • Servers booted with the /3 GB option • NonPagedPoolSize, NonPagedPoolSize, and SystemPagescan be set explicitly. • Max pool size is only available from the debugger!

19. Extended Virtual Addressing • Boot.ini /3 GB switch • Virtual storage constraint relief for some applications • Squeezes the System VM into 1 GB • Physical Address Extension (PAE) • Supports 37-bit real addresses on Xeon processors • Address Windowing Extensions (AWE) • Permits processes to address real memory > 4 GB

20. Virtual Memory Manager • Boot.ini /3 GB switch • Virtual storage constraint relief for some server applications • SQL Server • Exchange 2000 • Etc. • /userva=SizeInMB subparameter, where SizeinMB can be any value between 2048 and 3072

21. Exchange 2003 memory tuning • Exchange Server 2003 recommendations /3 GB /Userva = 3030 SystemPages = -1 HeapDeCommitFreeBlockThreshold = 0x00040000 msExchESEParamCacheSizeMax = 311296 (1.2 GB) • See KB #815372

22. Virtual Memory constraints • Try the /3 GB switch • 1 GB system area is not enough virtual memory for some applications • Possible shortage of Free System Page Table Entries • Possible shortage of Nonpaged Pool • Where Session data from TCP connections is stored • Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e.g., SQL Server, MS Exchange database – store.exe) using Microsoft Cluster Server (MCS) • During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time

23. Physical Address Extension • PAE is supported on most recent Intel servers • Physical addresses are 37-bits • Up to 128 GB of RAM can be installed • New format Page Table Entries (PTEs) are 64-bits • Process address spaces are still limited to 4 GB. How can they exploit 64 GB of RAM? • Expand sideways, using multiple address spaces • Large Memory Enabled (LME) device drivers • Address Windowing Extensions (AWE) • Boot.ini /pae switch • Limitation: system addresses can be no higher than 16 GB

24. Intel x86 in PAE mode

25. Physical Address Extension • Expand sideways, using multiple address spaces • e.g., MS SQL Server

26. IIS 6.0 (expand sideways)

27. Configuring IIS 6.0 Application Pools

28. Address Windowing Extensions (AWE) • API that allows processes to address real memory locations outside their 4 GB virtual addressing range • Create and manage memory Overlays • Used in conjunction with PAE AllocateUserPhysicalPages Virtual Alloc MapUserPhysicalPages

29. AWE Note: Frequent unmapping and remapping of Physical Memory blocks is expensive!

30. AWE support • MS SQL Server 2000: /3 GB, PAE, & AWE • Multiple process instances • awe enabled = 1 • AWE activity counters in the Buffer Manager object • Oracle: /3 GB, PAE, & AWE • AWE_WINDOW_MEMORY • Also Set max server Oraclememory • SAS: /3 GB, PAE • Work library can be placed in PAE memory

31. System Virtual Memory tuning

32. Default Paged and Nonpaged pool size can be overridden:

33. Default Paged and Nonpaged pool size can be overridden (Win 2003): • System Pages = x’ffff ffff’16 or -1 • Maximizes the number of PTEs that can be built • Potentially useful with /PAE or Terminal Services • PagedPoolSize = x’ffff ffff’16 or -1 • Allows the OS maximum flexibility to determine the size of the Paged Pool • Or PagedPoolSize and NonPagedPoolSize can be set explicitly • LargeSystemCache controls the maximum virtual address range of the file cache • 512 MB – 1 GB

34. !vm Debugger command runs the Page Frame Number (PFN) database Available Pages: 4920 ( 19680 Kb) ResAvail Pages: 358 ( 1432 Kb) Locked IO Pages: 251 ( 1004 Kb) Free System PTEs: 204387 ( 817548 Kb) Free NP PTEs: 28645 ( 114580 Kb) Free Special NP: 0 ( 0 Kb) Modified Pages: 596 ( 2384 Kb) Modified PF Pages: 660 ( 2640 Kb) NonPagedPool Usage: 2750 ( 11000 Kb) NonPagedPool Max: 33768 ( 135072 Kb) PagedPool 0 Usage: 3544 ( 14176 Kb) PagedPool 1 Usage: 1359 ( 5436 Kb) PagedPool 2 Usage: 1340 ( 5360 Kb) PagedPool Usage: 6243 ( 24972 Kb) PagedPool Maximum: 138240 ( 552960 Kb) Shared Commit: 6842 ( 27368 Kb) Special Pool: 0 ( 0 Kb) Shared Process: 3688 ( 14752 Kb) PagedPool Commit: 6398 ( 25592 Kb) Driver Commit: 1630 ( 6520 Kb) Committed pages: 211846 ( 847384 Kb) Commit limit: 320257 ( 1281028 Kb)

35. !poolused Debugger command accounts for all system pool allocations (plus pooltags.txt documentation) lkd> !poolused 2 Sorting by NonPaged Pool Consumed Pool Used: NonPaged Paged Tag Allocs Used Allocs Used LSwi 1 2576384 0 0 NV 287 1379120 14 55272 File 2983 504920 0 0 MmCm 16 435248 0 0 LSwr 128 406528 0 0 Devi 267 377472 0 0 Thre 452 296512 0 0 PcNw 12 278880 0 0 Irp 669 222304 0 0

36. Poolmon utility (from DDK)

37. 64-bit Virtual Memory Relief

38. 64-bit Relief • Windows supports two flavors of 64-bit hardware • IA-64 (Itanium) • AMD64 and Intel x64 (aka EMT) • Runs 32-bit applications with almost no performance penalty • 32-bit applications can allocate their entire 4 GB virtual memory range

39. Questions ?