robots.txt파일로 막는 방법말고 httpd.conf에서 일괄적으로 
트레픽 증가로 로봇을 막을려고 여기저기서 끌고와 httpd.conf에
아래와 같이 
설정하였습니다.
 맞게 설정했는지 궁금하네요.

또 nokeepalive ,  nokeepalive downgrade-1.0 force-response-1.0,
force-response-1.0, redirect-carefully 이 설정이 무엇인지 알고
싶습니다.
그냥 전부 go_out로 설정해도 되는지요? 

그리고 더 추가해야할 로봇을 알고계시면 알려주십시요.

<Directory /home/*/public_html>
    AllowOverride FileInfo AuthConfig Limit
    Options ExecCGI MultiViews SymLinksIfOwnerMatch IncludesNoExec
    <Limit GET POST OPTIONS PROPFIND>
        Order allow,deny
        Allow from all
    </Limit>
    <LimitExcept GET POST OPTIONS PROPFIND>
        Order deny,allow
        Deny from env=go_out
    </LimitExcept>
</Directory>

BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0
BrowserMatch "WebZip" go_out
BrowserMatch "Teleport" go_out
BrowserMatch "GetRight" go_out
BrowserMatch "WebCopier" go_out
BrowserMatch "NamoWebEditor" go_out
BrowserMatch "WebTrack-HTTPP" go_out
BrowserMatch "WebSymmetrix" go_out
BrowserMatch "Microsoft Data Access Internet Publishing Provider" 
redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[012]" redirect-carefully
BrowserMatch "^gnome-vfs" redirect-carefully
BrowserMatch "HTTrack" go_out
BrowserMatch "Download Ninja 2.0" go_out
BrowserMatch "JBH Agent 2.0" go_out
BrowserMatch "EmailCollector/1.0" go_out
BrowserMatch "EmailSiphon" go_out
BrowserMatch "EmailWolf 1.00" go_out
BrowserMatch "ExtractorPro" go_out
BrowserMatch "Crescent Internet ToolPak HTTP OLE Control v.1.0" go_out
BrowserMatch "Mozilla/2.0 (compatible; NEWT ActiveX; Win32)" go_out
BrowserMatch "CherryPicker/1.0" go_out
BrowserMatch "CherryPickerSE/1.0" go_out
BrowserMatch "CherryPickerElite/1.0" go_out
BrowserMatch "NICErsPRO" go_out
BrowserMatch "WebBandit/2.1" go_out
BrowserMatch "WebBandit/3.50" go_out
BrowserMatch "webbandit/4.00.0" go_out
BrowserMatch "WebEMailExtractor/1.0B" go_out
BrowserMatch "autoemailspider" go_out
BrowserMatch "JetCar" go_out
BrowserMatch "Go!Zilla" go_out
BrowserMatch "FlashGet" go_out
BrowserMatch "Namo" go_out
BrowserMatch "MSFrontPage" go_out
BrowserMatch "WebTrack-HTTPP" go_out
BrowserMatch "WebSymmetrix" go_out