웹서비스 장애 관련
This presentation is the property of its rightful owner.
Sponsored Links
1 / 12

웹서비스 장애 관련 PowerPoint PPT Presentation


  • 145 Views
  • Uploaded on
  • Presentation posted in: General

웹서비스 장애 관련. 강호성. 목 차. 서버 시스템 구성도 서버 다운 증상 서버 정보 (vmstat ) 서버 정보 (netstat ) 서버 정보 (netstat ) 서버 다운 정보 (startup.sh 옵션 : GC 정보 ) 서버 정보 ( ps -aux ) 서버 정보 ( 메모리 heap 변화 모니터링 ) 4. 질문사항 5. 예상 장애 요인. 1. 서버 시스템 구성도. 연동방법

Download Presentation

웹서비스 장애 관련

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


5754463

웹서비스 장애 관련

강호성


5754463

목 차

  • 서버 시스템 구성도

  • 서버 다운 증상

  • 서버 정보 (vmstat )

  • 서버 정보 (netstat )

  • 서버 정보 (netstat )

  • 서버 다운 정보 (startup.sh 옵션 : GC 정보)

  • 서버 정보 (ps -aux )

  • 서버 정보 (메모리 heap 변화 모니터링)

  • 4. 질문사항

  • 5. 예상 장애 요인


5754463

1. 서버 시스템 구성도

연동방법

Thumnail List출력 : URLConnection 이용

Apache : Redirect 를 이용

무룡이미지다운서비스 : Socket통신,다운로드

File Server (웹용 썸네일 이미지 파일들)

Env : Windows

Tomcat 4.1.18

Apache2.0.43

IP : 생략

위치 : 생략

사양 : CUP 2.4 G / 550MHZ

2G Memory

실제 원본 Data Server

사용용도 : Web Server와 암호모듈

통해서 다운로딩 서비스

 기존부터잘사용됨

Web Server

Env : Linux Redhat 9.0

Tomcat 4.1.18

Apache2.0.43

IP : 생략

위치 : 생략

상세내용 :

FileServer와연동(URLConnection 이용)

사양 : LGIBM

intel pentium 3 Xeon 550 Dual

1G Memory

장애발생서버

DataBase

Env : windows + Mssql2000

IP : 생략

위치 : 생략

사양 : HP DL380 모델

CUP 3.06 G / 550MHZ

2G Memory

사용자

연동방법

DNS : search.XXX.co.kr

Apache : Redirect 를 이용

Search Server

Env : Linux Redhat 9.0

Tomcat 4.1.18

Apache2.0.43

IP : 생략

위치 : 생략

상세내용 :

JSP  Servlet  C 검색모듈

사양 : CPU - Xeon 2.6 GHz * EA

Board - Intel SE7501BR2

Mem - 1G SDRAM * 4 EA

10 M 단독라인


5754463

* Tomcat 로그 (catalina) May 19, 2004 7:02:28 PM org.apache.tomcat.util.log.CommonLogHandler log INFO: All threads are busy, waiting. Please increase maxThreads or check the servlet status150 150 TOP 명령어를 이용한 총 프로세스는 평상시 200개 이하.서버 장애 시점에는 갑자기 8개이상씩 증가하면서 5~10분여 만에 총 프로세스 400개이상.서버 장애 시점에서의 초기에는 JSP 페이지는 안뜨나, html 페이지는 열립니다.서버 장애 후반에서는 JSP및html 페이지가 뜨지 않는 현상.

2.서버 다운 증상


5754463

* vmstat (평상시)procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 1360 127668 45360 263056 0 0 3 7 9 10 4 2 18 0 0 0 1360 127668 45360 263056 0 0 0 0 728 390 3 1 96 0 0 0 1360 127668 45360 263060 0 0 0 0 877 523 5 2 93 0 0 0 1360 127668 45360 263060 0 0 0 0 656 303 1 1 98 0 0 0 1360 127668 45360 263060 0 0 0 0 727 402 6 1 93 1 0 0 1360 127668 45360 263060 0 0 0 100 835 472 6 2 92 0 0 0 1360 127668 45360 263064 0 0 0 0 860 440 3 4 93 0 0 0 1360 127660 45360 263064 0 0 0 0 883 456 5 2 93 0 0 0 1360 127660 45360 263064 0 0 0 0 658 283 2 0 98 0 0 0 1360 127660 45360 263064 0 0 0 0 779 441 1 2 96 0 0 0 1360 127660 45360 263064 0 0 0 104 697 352 1 1 98 0 0 0 1360 127668 45360 263064 0 0 0 0 573 190 0 4 95 0 0 0 1360 127660 45360 263064 0 0 0 0 873 477 8 2 90 0 0 0 1360 127660 45360 263068 0 0 0 0 943 479 1 2 97 1 0 0 1360 127660 45360 263068 0 0 0 0 942 716 8 2 89 0 0 0 1360 127660 45364 263068 0 0 0 132 732 393 2 2 96 0 0 0 1360 127660 45364 263068 0 0 0 0 562 179 1 0 99 1 0 0 1360 127660 45364 263080 0 0 12 0 567 192 2 1 96 0 0 0 1360 127660 45364 263080 0 0 0 0 779 389 8 1 90 0 0 0 1360 127660 45364 263080 0 0 0 0 740 413 1 6 93 0 0 0 1360 127660 45364 263080 0 0 0 76 690 327 5 2 92 0 0 0 1360 127660 45364 263084 0 0 0 0 790 492 4 1 95 0 0 0 1360 127660 45364 263084 0 0 0 0 689 372 4 3 93 0 0 0 1360 127652 45364 263084 0 0 0 0 962 646 6 6 88 0 0 0 1360 127652 45364 263088 0 0 0 0 877 529 2 1 97 0 0 0 1360 127652 45364 263088 0 0 0 108 768 450 4 2 94 0 0 0 1360 127644 45364 263088 0 0 0 0 669 287 1 1 98 0 0 0 1360 127644 45364 263088 0 0 0 0 680 644 8 1 90 0 0 0 1360 127636 45364 263088 0 0 0 0 803 510 4 2 94 0 0 0 1360 127636 45364 263088 0 0 0 0 647 301 3 1 95 0 0 0 1360 127644 45368 263152 0 0 68 96 745 460 6 2 92 1 0 0 1360 127652 45368 263164 0 0 12 0 782 428 3 2 95

3.서버 정보 (vmstat )

* vmstat (서버장애시)

procs memory swap io system cpu

r b w swpd free buff cache si so bi bo in cs us sy id

0 0 0 1360 92372 44664 278980 0 0 0 0 644 327 2 1 97

0 0 0 1360 92096 44664 278980 0 0 0 0 584 177 1 1 98

0 0 0 1360 92064 44668 278980 0 0 0 16 563 178 4 2 94

0 0 0 1360 91752 44668 278980 0 0 0 0 601 228 0 2 98

0 0 0 1360 91384 44668 278980 0 0 0 0 578 191 1 1 97

0 0 0 1360 90500 44668 278980 0 0 0 0 593 241 2 3 95

0 0 0 1360 89320 44668 278980 0 0 0 0 572 212 1 4 95

0 0 0 1360 89188 44672 278976 0 0 0 44 589 208 2 3 94

0 0 0 1360 89064 44672 278976 0 0 0 0 580 181 0 2 97

0 0 0 1360 88860 44672 278976 0 0 0 0 582 207 2 3 95

0 0 0 1360 88792 44672 278976 0 0 0 0 565 173 1 4 94

0 0 0 1360 88728 44672 278976 0 0 0 32 594 186 0 1 98

0 0 0 1360 88552 44672 278976 0 0 0 0 607 216 1 2 96

0 0 0 1360 88164 44672 278976 0 0 0 0 566 170 1 1 98

0 0 0 1360 88156 44672 278976 0 0 0 0 580 172 0 1 98

0 0 0 1360 87848 44672 278976 0 0 0 0 618 213 1 2 97

0 0 0 1360 87356 44672 278976 0 0 0 28 615 266 4 3 93

0 0 0 1360 86848 44672 278976 0 0 0 0 575 180 1 2 97

0 0 0 1360 86848 44672 278976 0 0 0 0 570 166 1 5 94

0 0 0 1360 86756 44672 278976 0 0 0 0 587 193 1 2 97

0 0 0 1360 86700 44672 278976 0 0 0 0 587 173 1 2 97

0 0 0 1360 86492 44672 278976 0 0 0 24 577 205 2 2 95

0 0 0 1360 86300 44672 278976 0 0 0 0 569 179 2 6 92

0 0 0 1360 85988 44672 278976 0 0 0 0 582 188 1 1 98

0 0 0 1360 85540 44672 278976 0 0 0 0 606 260 3 2 95

0 0 0 1360 85416 44672 278976 0 0 0 0 571 170 1 2 97

0 0 0 1360 84944 44672 278976 0 0 0 40 609 270 2 2 96

0 0 0 1360 83992 44672 278976 0 0 0 0 653 289 2 3 95

1 0 0 1360 82340 44672 278980 0 0 0 0 640 294 3 2 94

0 0 0 1360 80124 44672 278980 0 0 0 0 590 250 1 4 95

0 0 0 1360 79724 44672 278980 0 0 0 48 605 202 2 6 92


5754463

* netstat -n | grep EST (평상시)tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52493 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52492 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52486 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:22 xxx.xxx.29.4:4586 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52575 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52571 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:80 xxx.xxx.152.9:4608 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52567 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52553 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52493 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52492 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52486 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52571 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52575 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52567 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52553 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52475 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52478 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:52464 xxx.xxx.xx.xx:8009 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52478 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52475 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52464 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:80 xxx.xxx.9.59:1625 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:51512 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:51513 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:51778 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:51778 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:50330 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:50742 database-ip:1433 ESTABLISHED tcp 0 68 xxx.xxx.xx.xx:50914 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:50913 database-ip:1433 ESTABLISHED tcp 0 0 xxx.xxx.xx.xx:50952 database-ip:1433 ESTABLISHED tcp 0 904 xxx.xxx.xx.xx:22 xxx.xxx.xx.x:1112 ESTABLISHED-------------------------------------------------------------------------------------------------총 ESTABLISHED 개수 : 33개Database 연동관련 : 9개8009 포트  기타 포트 : 10 개기타 포트  8009 포트 : 10 개80 포트  client : 2 개

3.서버 정보 (netstat : Established)

* netstat -n | grep EST (서버장애시)

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 webserver-ip:80 xxx..xxx.49.174:19848 ESTABLISHED

tcp 0 0 webserver-ip:8009 webserver-ip:36670 ESTABLISHED

tcp 0 0 webserver-ip:80 xxx.xxx.242.71:2976 ESTABLISHED

tcp 0 0 webserver-ip:22 xxx.xxx.29.4:4586 ESTABLISHED

tcp 0 0 webserver-ip:8009 webserver-ip:36703 ESTABLISHED

tcp 0 0 webserver-ip:80 xxx.xxx.197.146:1682 ESTABLISHED

tcp 0 0 webserver-ip:8009 webserver-ip:36696 ESTABLISHED

: : : : 기타 15개 동일 내용 (상동)

tcp 0 0 webserver-ip:80 xxx.xxx.xxx.xx:2177 ESTABLISHED

tcp 0 0 webserver-ip:8009 webserver-ip:36717 ESTABLISHED

: : : : 기타 28개 동일 내용 (상동)

tcp 0 0 webserver-ip:80 xxx.xxx.158.13:3051 ESTABLISHED

tcp 0 0 webserver-ip:8009 webserver-ip:36744 ESTABLISHED

: : : : 기타 36개 동일 내용 (상동)

tcp 0 0 webserver-ip:80 xxx.xxx.xxx.107:2604 ESTABLISHED

tcp 0 0 webserver-ip:8009 webserver-ip:36774 ESTABLISHED

: : : : 기타 6개 동일 내용 (상동)

tcp 0 0 webserver-ip:36817 webserver-ip:8009 ESTABLISHED

: : : : 기타 106개 동일 내용 (상동)

tcp 0 0 webserver-ip:8009 webserver-ip:36822 ESTABLISHED

: : : : 기타 20개 동일 내용 (상동)

tcp 0 0 webserver-ip:80 xx8.xxx.31.12:3735 ESTABLISHED

tcp 0 0 webserver-ip:80 xxx.xxx.2.102:2354 ESTABLISHED

tcp 0 0 webserver-ip:35038 database-ip:1433 ESTABLISHED

tcp 0 0 webserver-ip:35213 database-ip:1433 ESTABLISHED

tcp 0 0 webserver-ip:35214 database-ip:1433 ESTABLISHED

tcp 0 0 webserver-ip:33816 database-ip:1433 ESTABLISHED

tcp 0 0 webserver-ip:80 xxx.215.xx5.55:3877 ESTABLISHED

: : : : 기타 23개 동일 내용 (상동)

-------------------------------------------------------------------------------------------------

총 ESTABLISHED 개수 : 256개

Database 연동관련 : 4개

8009 포트  기타 포트 : 102 개

기타 포트  8009 포트 : 107 개

80 포트  client : 32 개


5754463

* GC (평상시)[GC 39885K->37285K(40124K), 0.0071930 secs][GC 39887K->37605K(40324K), 0.0100080 secs][Full GC 37605K->16401K(40324K), 0.7960270 secs][GC 19328K->17062K(40124K), 0.0126660 secs][GC 19892K->17599K(40124K), 0.0111760 secs][GC 20432K->18012K(40124K), 0.0408080 secs][GC 20850K->18626K(40124K), 0.0109500 secs][GC 21463K->18830K(40124K), 0.0118570 secs][GC 21451K->19196K(40124K), 0.0095420 secs][GC 22037K->19555K(40124K), 0.0093910 secs][GC 22399K->20149K(40124K), 0.0098520 secs][GC 22854K->20379K(40124K), 0.0083420 secs][GC 22996K->20612K(40124K), 0.0076090 secs][GC 23455K->20845K(40124K), 0.0063710 secs][GC 23655K->21308K(40124K), 0.0075850 secs][GC 24370K->21892K(40124K), 0.0072240 secs][GC 24732K->22138K(40124K), 0.0060080 secs][GC 24949K->22380K(40124K), 0.0061300 secs][GC 39762K->37170K(40124K), 0.0058780 secs][GC 39727K->37429K(40300K), 0.0068420 secs][Full GC 37429K->16191K(40300K), 0.7953360 secs][GC 39679K->37276K(40124K), 0.0069350 secs][GC 39925K->37610K(40300K), 0.0076660 secs][Full GC 37610K->14276K(40300K), 0.8748960 secs][GC 17581K->15373K(40124K), 0.0095520 secs][GC 18449K->15861K(40124K), 0.0104900 secs][GC 18483K->16526K(40124K), 0.0137420 secs][GC 19146K->17209K(40124K), 0.0346310 secs][GC 19832K->17766K(40124K), 0.0206310 secs][GC 20389K->17794K(40124K), 0.0103710 secs][GC 20391K->17813K(40124K), 0.0061950 secs][GC 20584K->17906K(40124K), 0.0054300 secs][GC 20498K->17998K(40124K), 0.0061370 secs][GC 20617K->18008K(40124K), 0.0054560 secs][GC 20730K->18018K(40124K), 0.0055230 secs][GC 20619K->18137K(40124K), 0.0055160 secs][GC 20735K->18145K(40124K), 0.0058770 secs]

3.서버 다운 정보 (startup.sh 옵션 : GC 정보)

  • GC (서버장애시)

  • [GC 35807K->32869K(44368K), 0.0074160 secs]

  • [GC 44010K->41290K(44368K), 0.0071010 secs]

  • [Full GC 41290K->16677K(44368K), 0.9458450 secs]

  • [GC 22642K->20059K(44368K), 0.0078300 secs]

  • [GC 33637K->30787K(44368K), 0.0072790 secs]

  • [GC 43817K->41049K(44368K), 0.0063940 secs]

  • [GC 43865K->41657K(44496K), 0.0124660 secs]

  • [Full GC 41657K->17253K(44496K), 0.9132000 secs]

  • [GC 23044K->20879K(44368K), 0.0138540 secs]

  • [GC 23691K->21423K(44368K), 0.0141100 secs]

  • [GC 24234K->21868K(44368K), 0.0131710 secs]

  • [GC 24684K->23311K(44368K), 0.0262750 secs]

  • [GC 26126K->23893K(44368K), 0.0163500 secs]

  • [GC 26708K->24277K(44368K), 0.0124560 secs]

  • [GC 27081K->24749K(44368K), 0.0143000 secs]

  • [GC 27563K->25908K(44368K), 0.0191340 secs]

  • [GC 28708K->26359K(44368K), 0.0139570 secs]

  • [GC 29175K->27535K(44368K), 0.0185610 secs]

  • GC 30351K->27820K(44368K), 0.0125210 secs]

  • [GC 30635K->28441K(44368K), 0.0281460 secs]

  • [GC 31247K->28785K(44368K), 0.0128440 secs]

  • [GC 31600K->29805K(44368K), 0.0275490 secs]

  • [GC 32621K->31303K(44368K), 0.0394960 secs]

  • [GC 34118K->33026K(44368K), 0.0486900 secs]

  • [GC 35836K->34692K(44368K), 0.0554940 secs]

  • [GC 37508K->36243K(44368K), 0.0576330 secs]

  • [GC 39056K->37789K(44368K), 0.0641460 secs]

  • [GC 40605K->39362K(44368K), 0.0687840 secs]

  • [GC 42178K->41334K(44368K), 0.0804310 secs]

  • [GC 44149K->42679K(45520K), 0.0803990 secs]

  • --------------------------------------------- 여기부터 서버 느려서 페이지가 뜨지않음

  • [Full GC 42679K->31065K(45520K), 1.3358060 secs]

  • -------------------------------------------- 페이지 안뜬상태에서 조금있다가 로그 남음

  • [GC 34649K->33041K(55748K), 0.0914980 secs]

  • -------------------------------------------- 페이지 안뜬상태에서 조금있다가 로그 남음

  • [GC 36624K->35406K(55748K), 0.1062460 secs]

  • -------------------------------------------- 웹서비스를 위해 톰캣을 재시작처리함


5754463

* ps 내용 (평상시)USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.0 0.0 1400 468 ? S May15 0:19 initroot 2 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU0]root 3 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU1]root 4 0.0 0.0 0 0 ? SW May15 0:00 [keventd]root 5 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU0]root 6 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU1]root 7 0.0 0.0 0 0 ? SW May15 2:43 [kswapd]root 8 0.0 0.0 0 0 ? SW May15 0:05 [bdflush]root 9 0.0 0.0 0 0 ? SW May15 0:10 [kupdated]root 10 0.0 0.0 0 0 ? SW May15 0:00 [mdrecoveryd]root 16 0.0 0.0 0 0 ? SW May15 0:00 [scsi_eh_0]root 19 0.0 0.0 0 0 ? SW May15 1:08 [kjournald]root 75 0.0 0.0 0 0 ? SW May15 0:00 [khubd]root 167 0.0 0.0 0 0 ? SW May15 0:00 [kjournald]root 441 0.0 0.0 1472 524 ? S May15 0:02 syslogd -m 0root 445 0.0 0.0 1400 444 ? S May15 0:00 klogd -xroot 483 0.0 0.1 3356 1260 ? S May15 0:18 /usr/sbin/sshdroot 494 0.0 0.0 2036 608 ? S May15 0:00 xinetd -stayalive -reuse -pidfile /var/runroot 513 0.0 0.1 5604 1512 ? S May15 0:43 [sendmail]smmsp 522 0.0 0.1 4952 1372 ? S May15 0:00 [sendmail]root 532 0.0 0.0 1460 580 ? S May15 0:01 cronddaemon 550 0.0 0.0 1444 544 ? S May15 0:00 [atd]root 561 0.0 0.0 3432 544 ? S May15 0:00 rhnsd --interval 120root 567 0.0 0.0 1376 368 tty1 S May15 0:00 /sbin/mingetty tty1root 3974 0.0 0.2 6936 2156 ? S May24 0:29 /usr/sbin/sshdroot 4059 0.0 0.1 3228 1176 ? S May24 0:01 /usr/libexec/openssh/sftp-serverroot 4087 0.0 0.1 6868 2048 ? S May24 0:01 /usr/sbin/sshdroot 4089 0.0 0.1 3208 1108 ? S May24 0:00 /usr/libexec/openssh/sftp-serverroot 26908 8.2 57.8 728308 596340 ? S 08:58 10:17 /usr/java/j2se/bin/java -Xmx512m -Xms512m root 26925 0.0 0.2 4972 2344 ? S 08:58 0:00 /usr/local/httpd/bin/httpd -k startroot 27415 0.0 0.1 4400 1464 pts/0 S 11:02 0:00 -bashnobody 27502 0.0 0.2 5080 2556 ? S 11:08 0:00 [httpd] nobody 27511 0.0 0.2 5076 2524 ? S 11:09 0:00 [httpd]nobody 27527 0.0 0.2 5080 2552 ? S 11:11 0:00 [httpd]nobody 27528 0.0 0.2 5080 2552 ? S 11:11 0:00 [httpd]nobody 27530 0.0 0.2 5076 2520 ? S 11:12 0:00 [httpd]nobody 27531 0.0 0.2 5072 2544 ? S 11:12 0:00 [httpd]nobody 27533 0.0 0.2 5076 2516 ? S 11:12 0:00 [httpd]nobody 27535 0.0 0.2 5068 2512 ? S 11:14 0:00 [httpd]nobody 27540 0.0 0.2 5068 2508 ? S 11:17 0:00 [httpd]nobody 27543 0.0 0.2 5068 2504 ? S 11:17 0:00 [httpd]nobody 27544 0.0 0.2 5068 2504 ? S 11:17 0:00 [httpd]root 27545 0.0 0.0 2776 836 pts/0 R 11:17 0:02 ps -aux

3.서버 정보 (ps -aux )

* ps 내용 (평상시)

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1 0.0 0.0 1400 468 ? S May15 0:19 init

root 2 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU0]

root 3 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU1]

root 4 0.0 0.0 0 0 ? SW May15 0:00 [keventd]

root 5 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU0]

root 6 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU1]

root 7 0.0 0.0 0 0 ? SW May15 2:43 [kswapd]

root 8 0.0 0.0 0 0 ? SW May15 0:05 [bdflush]

root 9 0.0 0.0 0 0 ? SW May15 0:10 [kupdated]

root 10 0.0 0.0 0 0 ? SW May15 0:00 [mdrecoveryd]

root 16 0.0 0.0 0 0 ? SW May15 0:00 [scsi_eh_0]

root 19 0.0 0.0 0 0 ? SW May15 1:08 [kjournald]

root 75 0.0 0.0 0 0 ? SW May15 0:00 [khubd]

root 167 0.0 0.0 0 0 ? SW May15 0:00 [kjournald]

root 441 0.0 0.0 1472 524 ? S May15 0:02 syslogd -m 0

root 445 0.0 0.0 1400 444 ? S May15 0:00 klogd -x

root 483 0.0 0.1 3356 1260 ? S May15 0:18 /usr/sbin/sshd

root 494 0.0 0.0 2036 608 ? S May15 0:00 xinetd -stayalive -reuse -pidfile /var/run

root 513 0.0 0.1 5604 1512 ? S May15 0:44 [sendmail]

smmsp 522 0.0 0.1 4952 1372 ? S May15 0:00 [sendmail]

root 532 0.0 0.0 1460 580 ? S May15 0:01 crond

daemon 550 0.0 0.0 1444 544 ? S May15 0:00 [atd]

root 561 0.0 0.0 3432 544 ? S May15 0:00 rhnsd --interval 120

root 567 0.0 0.0 1376 368 tty1 S May15 0:00 /sbin/mingetty tty1

root 3974 0.0 0.2 6936 2156 ? S May24 0:29 /usr/sbin/sshd

root 4059 0.0 0.1 3228 1176 ? S May24 0:01 /usr/libexec/openssh/sftp-server

root 4087 0.0 0.1 6868 2048 ? S May24 0:01 /usr/sbin/sshd

root 4089 0.0 0.1 3208 1108 ? S May24 0:00 /usr/libexec/openssh/sftp-server

root 26908 5.8 58.5 738628 603804 ? S 08:58 9:33 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 26925 0.0 0.2 4972 2344 ? S 08:58 0:01 /usr/local/httpd/bin/httpd -k start

root 27066 0.3 58.5 738628 603804 ? S 10:07 0:16 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27067 0.3 58.5 738632 603816 ? S 10:07 0:19 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27072 0.4 58.5 738628 603816 ? S 10:07 0:22 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27073 0.4 58.5 738628 603816 ? S 10:07 0:23 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27074 0.2 58.5 738628 603816 ? S 10:07 0:13 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27075 0.2 58.5 738628 603816 ? S 10:07 0:13 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27079 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27080 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27081 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27082 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27087 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27088 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27089 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27090 0.3 58.5 738628 603816 ? S 10:07 0:15 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27415 0.0 0.1 4400 1464 pts/0 S 11:02 0:00 -bash

nobody 27620 0.0 0.2 5068 2512 ? S 11:27 0:00 [httpd]

nobody 27622 0.0 0.2 5076 2524 ? S 11:27 0:00 [httpd]

nobody 27624 0.0 0.2 5076 2516 ? S 11:28 0:00 [httpd]

nobody 27633 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd]

nobody 27637 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd]

nobody 27638 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd]

nobody 27642 0.0 0.2 5080 2548 ? S 11:30 0:00 [httpd]

nobody 27643 0.0 0.2 5068 2508 ? S 11:30 0:00 [httpd]

nobody 27645 0.0 0.2 5068 2496 ? S 11:31 0:00 [httpd]

nobody 27646 0.0 0.2 5068 2488 ? S 11:31 0:00 [httpd]

nobody 27647 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd]

nobody 27648 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd]

nobody 27649 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd]

nobody 27650 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd]

nobody 27651 0.0 0.2 5052 2488 ? S 11:31 0:00 [httpd]

nobody 27652 0.0 0.2 5052 2492 ? S 11:31 0:00 [httpd]

nobody 27653 0.0 0.2 5044 2472 ? S 11:32 0:00 [httpd]

nobody 27654 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27655 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27656 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27657 0.0 0.2 5052 2488 ? S 11:32 0:00 [httpd]

nobody 27658 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27659 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27660 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27661 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27662 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27663 0.0 0.2 5060 2496 ? S 11:32 0:00 [httpd]

nobody 27664 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd]

nobody 27665 0.0 0.2 5052 2492 ? S 11:32 0:00 [httpd]

nobody 27666 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27667 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd]

nobody 27668 0.0 0.2 5068 2500 ? S 11:32 0:00 [httpd]

nobody 27669 0.0 0.2 5068 2496 ? S 11:32 0:00 [httpd]

nobody 27670 0.0 0.2 5068 2488 ? S 11:32 0:00 [httpd]

nobody 27671 0.0 0.2 5068 2496 ? S 11:32 0:00 [httpd]

nobody 27672 0.0 0.2 5060 2492 ? S 11:32 0:00 [httpd]

nobody 27673 0.0 0.2 5068 2484 ? S 11:32 0:00 [httpd]

nobody 27674 0.0 0.2 5076 2500 ? S 11:32 0:00 [httpd]

nobody 27675 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd]

nobody 27676 0.0 0.2 5076 2496 ? S 11:32 0:00 [httpd]

nobody 27677 0.0 0.2 5052 2488 ? S 11:32 0:00 [httpd]

nobody 27678 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27679 0.0 0.2 5068 2504 ? S 11:32 0:00 [httpd]

nobody 27680 0.0 0.2 5060 2488 ? S 11:32 0:00 [httpd]

nobody 27681 0.0 0.2 5060 2488 ? S 11:33 0:00 [httpd]

nobody 27682 0.0 0.2 5060 2484 ? S 11:33 0:00 [httpd]

nobody 27683 0.0 0.2 5052 2488 ? S 11:33 0:00 [httpd]

nobody 27684 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd]

nobody 27685 0.0 0.2 5060 2484 ? S 11:33 0:00 [httpd]

nobody 27686 0.0 0.2 5052 2488 ? S 11:33 0:00 [httpd]

root 27687 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27688 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27689 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27690 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

nobody 27691 0.0 0.2 5052 2484 ? S 11:33 0:00 [httpd]

root 27692 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27693 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27694 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27695 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

nobody 27696 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd]

nobody 27697 0.0 0.2 5076 2500 ? S 11:33 0:00 [httpd]

nobody 27698 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd]

nobody 27699 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd]

nobody 27700 0.0 0.2 5076 2508 ? S 11:33 0:00 [httpd]

root 27701 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27702 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27703 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27704 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

nobody 27705 0.0 0.2 5068 2508 ? S 11:33 0:00 [httpd]

nobody 27706 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd]

nobody 27707 0.0 0.2 5044 2472 ? S 11:33 0:00 [httpd]

nobody 27708 0.0 0.2 5076 2508 ? S 11:33 0:00 [httpd]

nobody 27709 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd]

nobody 27710 0.0 0.2 5104 2540 ? S 11:33 0:00 [httpd]

nobody 27711 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd]

nobody 27712 0.0 0.2 5044 2476 ? S 11:33 0:00 [httpd]

root 27713 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27714 0.1 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27715 0.1 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27716 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27717 0.2 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27718 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27719 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27720 0.2 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java -Xmx512m -Xms512m

root 27721 0.0 0.0 2844 904 pts/0 R 11:33 0:02 ps -aux

nobody 27722 0.0 0.2 4996 2384 ? S 11:33 0:00 [httpd]


5754463

* 메모리 heap (평상시)Tue May 25 11:47:28 KST 2004F:443MB T:508 MB F:469MB T:508 MB Tue May 25 11:49:23 KST 2004F:466MB T:508 MB F:491MB T:508 MB Tue May 25 11:51:11 KST 2004F:466MB T:508 MB F:495MB T:508 MB Tue May 25 11:53:27 KST 2004F:434MB T:508 MB F:476MB T:508 MBTue May 25 12:08:09 KST 2004F:347MB T:508 MB F:471MB T:508 MB Tue May 25 12:08:25 KST 2004F:452MB T:508 MB F:468MB T:508 MB Tue May 25 12:10:32 KST 2004F:481MB T:508 MB F:487MB T:508 MBTue May 25 12:15:33 KST 2004F:431MB T:508 MB F:470MB T:508 MBTue May 25 13:47:02 KST 2004F:148MB T:508 MB F:462MB T:508 MB Tue May 25 14:38:21 KST 2004F:77MB T:508 MB F:461MB T:508 MB

3.서버 정보 (메모리 heap 변화 모니터링)

  • 메모리 heap (서버장애시)

  • JSP 페이지 자체가 뜨지 않아 파악하지 못함


5754463

4.N/W 정보 (10M 단독 라인)

[ 주간 ]

[ 월간 ]

[ 년간 ]


5754463

* Runtime.exec()를 통해 JVM 외부의 shell script나 c program 을 호출하는 부분이 있는지요?현재 다운되는 서버 내의 JVM에서는 shell이나 c를 호출하는 부분이 없습니다.* 프로그램 어디선가 Socket 혹은 HttpUrlConnection으로 "localhost:80" 으로 접속을 시도하는 부분이 있는지요?어떤 용도로 쓰였는지요? Socket –파일 서버 와 연동중 무료이미지 관련 부분이 사용되고 있습니다. (첨부파일참조) URLConnection –파일서버와 연동중 폴더내 파일리스트를 가져오는 부분이 있습니다. (첨부파일참조)* java process 개수가 80+1 에서 80 이 갖는 의미가 뭘까요? 어떤 파라메터가 80인가요? 저도 정확히는 모르겠습니다만 아파치 프로세스중에 좀비가 가끔씩 뜨지만 잠시 떴다가 금방 사라지며, 이부분에 대해 자바서비스넷 자료를 찾는중에 금방 사라지는 좀비프로세스는 괜찮다는 글을 읽은걸로 알고있습니다.* dump파일 생성 관련  ibm JDK 가 아닌 sun JDK사용으로 인하여 dump 파일이 생기지 않습니다.* server.xml의 내용<Host name="test.test.co.kr" debug="0" appBase=“appbase" liveDeploy="true" unpackWARs="true" autoDeploy="true"><Context cookies="true" docBase=“test" path="/test" reloadable="true" debug="0" crossContext="false"> <Logger className="org.apache.catalina.logger.FileLogger" prefix="localhost_test_log." suffix=".txt" timestamp="true"/> <Realm className="org.apache.catalina.realm.JDBCRealm" connectionName="sa" connectionPassword=“xxxxx" connectionURL="jdbc:microsoft:sqlserver://database-ip:1433;DatabaseName=test;SelectMethod=cursor;SendStringParametersAsUnicode=true" debug="0" driverName="com.microsoft.jdbc.sqlserver.SQLServerDriver" userTable="T_testmember" userNameCol=“userid" userCredCol=“pwd" userRoleTable="T_룰" roleNameCol="U_룰"/> <ResourceLink global="jdbc/TESTDB" name="jdbc/TESTDB"/> </Context>~~~~~~~~~~~~이하생략

4. 질문 사항


5754463

5. 예상 장애 요인

1 –이미지 다운로드 관련 로직 (첨부파일 참조 ) : 웹서버  파일서버 소켓연동

( 파일서버에서 socket은 체크후 해제 처리를 합니다. 현재 첨부파일소스는 웹서버의 client소스 입니다.)

2 –폴더내 이미지 리스트 호출 관련 로직 : 웹서버  파일서버 URLConnection 이용

아래 소스 참고

FileServer페이지

<[email protected] 선언부 %>

<[email protected] import=“~~”%>

<% String fpath="";

fpath=(String)(request.getParameter("fpath")); //---request받음

int filesize=0;

try {

File file = new File(fpath);

filesize=(int)file.length();

if(filesize<1){

String fileList[]=new String[filesize];

if(file.list()!=null){

fileList=file.list();

for(int k=0;k<fileList.length;k++){

File file_List = new File(fpath+fileList[k]);

out.println(file_List.getName());

}//-------end for for(int k=0;k<

}else{//---if(file.list()!=null

out.println("");

}

file_List=null;

}else{

out.println("");

}

} catch(Exception e) {

e.printStackTrace();

}finally{

System.out.println("fpath==>"+fpath);

}

%>

WebServer페이지

public static Vector mbJspFileList(String path){

Vector list = new Vector();

BufferedInputStream is = null;

try {

URL url = new URL("http://FileServer페이지주소?fpath="+path);

URLConnection conn = url.openConnection();

if(conn!=null){

is = new BufferedInputStream(conn.getInputStream());

int _value;

char value;

StringBuffer sb = new StringBuffer();

while((_value = is.read())!=-1){

value = (char)_value;

if(value != '\n') sb.append(value);

else {

list.addElement(sb.toString());

sb = new StringBuffer();

}

}

}else{

list=null;

}

}catch(Exception e){

e.printStackTrace();

}finally {

if(is!=null){try {is.close();}catch(Exception e){}}

}

return list;

}


  • Login