- 浏览: 740351 次
- 性别:
- 来自: 杭州
-
文章分类
最新评论
-
lgh1992314：
a offset: 26b offset: 24c offse ...
java jvm字节占用空间分析 -
ls0609：
语音实现在线听书http://blog.csdn.net/ls ...
Android 语音输入API使用 -
wangli61289：
http://viralpatel-net-tutorials ...
Android 语音输入API使用 -
zxjlwt：
学习了素人派http://surenpi.com
velocity宏加载顺序 -
tt5753：
谢啦........
Lucene的IndexWriter初始化时的LockObtainFailedException的解决方法
最近有个需求需要对于获取URL页面进行host绑定并且立即生效，在java里面实现可以用代理服务器来实现：因为在测试环境下可能需要通过绑定来访问测试环境的应用
实现代码如下：
public static String getResponseText(String queryUrl,String host,String ip) { //queryUrl，完整的url，host和ip需要绑定的host和ip
InputStream is = null;
BufferedReader br = null;
StringBuffer res = new StringBuffer();
try {
HttpURLConnection httpUrlConn = null;
URL url = new URL(queryUrl);
if(ip!=null){
String str[] = ip.split("\\.");
byte[] b =new byte[str.length];
for(int i=0,len=str.length;i<len;i++){
b[i] = (byte)(Integer.parseInt(str[i],10));
}
Proxy proxy = new Proxy(Proxy.Type.HTTP,
new InetSocketAddress(InetAddress.getByAddress(b), 80)); //b是绑定的ip，生成proxy代理对象，因为http底层是socket实现，
httpUrlConn = (HttpURLConnection) url
.openConnection(proxy);
}else{
httpUrlConn = (HttpURLConnection) url
.openConnection();
}
httpUrlConn.setRequestMethod("GET");
httpUrlConn.setDoOutput(true);
httpUrlConn.setConnectTimeout(2000);
httpUrlConn.setReadTimeout(2000);
httpUrlConn.setDefaultUseCaches(false);
httpUrlConn.setUseCaches(false);
is = httpUrlConn.getInputStream();
那么底层对于proxy对象到底是怎么处理，底层的socket实现到底怎么样，带着这个疑惑看了下jdk的rt.jar对于这块的处理
httpUrlConn = (HttpURLConnection) url.openConnection(proxy)
java.net.URL类里面的openConnection方法：
public URLConnection openConnection(Proxy proxy){
…
return handler.openConnection(this, proxy); Handler是sun.net.www.protocol.http.Handler.java类，继承java.net. URLStreamHandler.java类，用来处理http连接请求响应的。
}
Handler的方法：
protected java.net.URLConnection openConnection(URL u, Proxy p)
throws IOException {
return new HttpURLConnection(u, p, this);
}
只是简单的生成sun.net.www.protocl.http.HttpURLConnection对象，并进行初始化
protected HttpURLConnection(URL u, Proxy p, Handler handler) {
super(u);
requests = new MessageHeader(); 请求头信息生成类
responses = new MessageHeader(); 响应头信息解析类
this.handler = handler;
instProxy = p; 代理服务器对象
cookieHandler = (CookieHandler)java.security.AccessController.doPrivileged(
new java.security.PrivilegedAction() {
public Object run() {
return CookieHandler.getDefault();
}
});
cacheHandler = (ResponseCache)java.security.AccessController.doPrivileged(
new java.security.PrivilegedAction() {
public Object run() {
return ResponseCache.getDefault();
}
});
}
最终在httpUrlConn.getInputStream();才进行socket连接，发送http请求，解析http响应信息。具体过程如下：
sun.net.www.protocl.http.HttpURLConnection.java的getInputStream方法：
public synchronized InputStream getInputStream() throws IOException {
...socket连接
connect();
...
ps = (PrintStream)http.getOutputStream(); 获得输出流，打开连接之后已经生成。
if (!streaming()) {
writeRequests(); 输出http请求头信息
}
...
http.parseHTTP(responses, pi, this); 解析响应信息
if(logger.isLoggable(Level.FINEST)) {
logger.fine(responses.toString());
}
inputStream = http.getInputStream(); 获得输入流
}
其中connect()调用方法链：
plainConnect(){
...
Proxy p = null;
if (sel != null) {
URI uri = sun.net.www.ParseUtil.toURI(url);
Iterator<Proxy> it = sel.select(uri).iterator();
while (it.hasNext()) {
p = it.next();
try {
if (!failedOnce) {
http = getNewHttpClient(url, p, connectTimeout);
...
}
getNewHttpClient(){
...
return HttpClient.New(url, p, connectTimeout, useCache);
...
}
下面跟进去最终建立socket连接的代码：
sun.net.www.http.HttpClient.java的openServer()方法建立socket连接：
protected synchronized void openServer() throws IOException {
...
if ((proxy != null) && (proxy.type() == Proxy.Type.HTTP)) {
sun.net.www.URLConnection.setProxiedHost(host);
if (security != null) {
security.checkConnect(host, port);
}
privilegedOpenServer((InetSocketAddress) proxy.address());最终socket连接的是设置的代理服务器的地址，
...
}
private synchronized void privilegedOpenServer(final InetSocketAddress server)
throws IOException
{
try {
java.security.AccessController.doPrivileged(
new java.security.PrivilegedExceptionAction() {
public Object run() throws IOException {
openServer(server.getHostName(), server.getPort()); 注意openserver函数 这里的server的getHostName是设置的代理服务器，（ip或者hostname，如果是host绑定设置的代理服务器的ip，那么这里getHostName出来的就是ip地址，可以去查看InetSocketAddress类的getHostName方法）
return null;
}
});
} catch (java.security.PrivilegedActionException pae) {
throw (IOException) pae.getException();
}
}
public void openServer(String server, int port) throws IOException {
serverSocket = doConnect(server, port); 生成的Socket连接对象
try {
serverOutput = new PrintStream(
new BufferedOutputStream(serverSocket.getOutputStream()),
false, encoding); 生成输出流，
} catch (UnsupportedEncodingException e) {
throw new InternalError(encoding+" encoding not found");
}
serverSocket.setTcpNoDelay(true);
}
protected Socket doConnect (String server, int port)
throws IOException, UnknownHostException {
Socket s;
if (proxy != null) {
if (proxy.type() == Proxy.Type.SOCKS) {
s = (Socket) AccessController.doPrivileged(
new PrivilegedAction() {
public Object run() {
return new Socket(proxy);
}});
} else
s = new Socket(Proxy.NO_PROXY);
} else
s = new Socket();
// Instance specific timeouts do have priority, that means
// connectTimeout & readTimeout (-1 means not set)
// Then global default timeouts
// Then no timeout.
if (connectTimeout >= 0) {
s.connect(new InetSocketAddress(server, port), connectTimeout);
} else {
if (defaultConnectTimeout > 0) {
s.connect(new InetSocketAddress(server, port), defaultConnectTimeout);//连接到代理服务器，看下面Socket类的connect方法代码
} else {
s.connect(new InetSocketAddress(server, port));
}
}
if (readTimeout >= 0)
s.setSoTimeout(readTimeout);
else if (defaultSoTimeout > 0) {
s.setSoTimeout(defaultSoTimeout);
}
return s;
}
上面的new InetSocketAddress(server, port)这里会涉及到java DNS cache的处理，
public InetSocketAddress(String hostname, int port) {
if (port < 0 || port > 0xFFFF) {
throw new IllegalArgumentException("port out of range:" + port);
}
if (hostname == null) {
throw new IllegalArgumentException("hostname can't be null");
}
try {
addr = InetAddress.getByName(hostname); //这里会有java DNS缓存的处理，先从缓存取hostname绑定的ip地址，如果取不到再通过OS的DNS cache机制去取，取不到再从DNS服务器上取。
} catch(UnknownHostException e) {
this.hostname = hostname;
addr = null;
}
this.port = port;
}
当然最终的Socket.java的connect方法
java.net.socket
public void connect(SocketAddress endpoint, int timeout) throws IOException {
if (endpoint == null)
if (timeout < 0)
throw new IllegalArgumentException("connect: timeout can't be negative");
if (isClosed())
throw new SocketException("Socket is closed");
if (!oldImpl && isConnected())
throw new SocketException("already connected");
if (!(endpoint instanceof InetSocketAddress))
throw new IllegalArgumentException("Unsupported address type");
InetSocketAddress epoint = (InetSocketAddress) endpoint;
SecurityManager security = System.getSecurityManager();
if (security != null) {
if (epoint.isUnresolved())
security.checkConnect(epoint.getHostName(),
epoint.getPort());
else
security.checkConnect(epoint.getAddress().getHostAddress(),
epoint.getPort());
}
if (!created)
createImpl(true);
if (!oldImpl)
impl.connect(epoint, timeout);
else if (timeout == 0) {
if (epoint.isUnresolved()) //如果没有设置SocketAddress的ip地址，则用域名去访问
impl.connect(epoint.getAddress().getHostName(),
epoint.getPort());
else
impl.connect(epoint.getAddress(), epoint.getPort()); 最终socket连接的是设置的SocketAddress的ip地址，
} else
throw new UnsupportedOperationException("SocketImpl.connect(addr, timeout)");
connected = true;
/*
* If the socket was not bound before the connect, it is now because
* the kernel will have picked an ephemeral port & a local address
*/
bound = true;
}
我们再看下通过socket来发送HTTP请求的处理代码，也就是sun.net.www.protocl.http.HttpURLConnection.java的getInputStream方法中调用的writeRequests()方法：
private void writeRequests() throws IOException { 这段代码就是封装http请求的头请求信息，通过socket发送出去
/* print all message headers in the MessageHeader
* onto the wire - all the ones we've set and any
* others that have been set
*/
// send any pre-emptive authentication
if (http.usingProxy) {
setPreemptiveProxyAuthentication(requests);
}
if (!setRequests) {
/* We're very particular about the order in which we
* set the request headers here. The order should not
* matter, but some careless CGI programs have been
* written to expect a very particular order of the
* standard headers. To name names, the order in which
* Navigator3.0 sends them. In particular, we make *sure*
* to send Content-type: <> and Content-length:<> second
* to last and last, respectively, in the case of a POST
* request.
*/
if (!failedOnce)
requests.prepend(method + " " + http.getURLFile()+" " +
httpVersion, null);
if (!getUseCaches()) {
requests.setIfNotSet ("Cache-Control", "no-cache");
requests.setIfNotSet ("Pragma", "no-cache");
}
requests.setIfNotSet("User-Agent", userAgent);
int port = url.getPort();
String host = url.getHost();
if (port != -1 && port != url.getDefaultPort()) {
host += ":" + String.valueOf(port);
}
requests.setIfNotSet("Host", host);
requests.setIfNotSet("Accept", acceptString);
/*
* For HTTP/1.1 the default behavior is to keep connections alive.
* However, we may be talking to a 1.0 server so we should set
* keep-alive just in case, except if we have encountered an error
* or if keep alive is disabled via a system property
*/
// Try keep-alive only on first attempt
if (!failedOnce && http.getHttpKeepAliveSet()) {
if (http.usingProxy) {
requests.setIfNotSet("Proxy-Connection", "keep-alive");
} else {
requests.setIfNotSet("Connection", "keep-alive");
}
} else {
/*
* RFC 2616 HTTP/1.1 section 14.10 says:
* HTTP/1.1 applications that do not support persistent
* connections MUST include the "close" connection option
* in every message
*/
requests.setIfNotSet("Connection", "close");
}
// Set modified since if necessary
long modTime = getIfModifiedSince();
if (modTime != 0 ) {
Date date = new Date(modTime);
//use the preferred date format according to RFC 2068(HTTP1.1),
// RFC 822 and RFC 1123
SimpleDateFormat fo =
new SimpleDateFormat ("EEE, dd MMM yyyy HH:mm:ss 'GMT'", Locale.US);
fo.setTimeZone(TimeZone.getTimeZone("GMT"));
requests.setIfNotSet("If-Modified-Since", fo.format(date));
}
// check for preemptive authorization
AuthenticationInfo sauth = AuthenticationInfo.getServerAuth(url);
if (sauth != null && sauth.supportsPreemptiveAuthorization() ) {
// Sets "Authorization"
requests.setIfNotSet(sauth.getHeaderName(), sauth.getHeaderValue(url,method));
currentServerCredentials = sauth;
}
if (!method.equals("PUT") && (poster != null || streaming())) {
requests.setIfNotSet ("Content-type",
"application/x-www-form-urlencoded");
}
if (streaming()) {
if (chunkLength != -1) {
requests.set ("Transfer-Encoding", "chunked");
} else {
requests.set ("Content-Length", String.valueOf(fixedContentLength));
}
} else if (poster != null) {
/* add Content-Length & POST/PUT data */
synchronized (poster) {
/* close it, so no more data can be added */
poster.close();
requests.set("Content-Length",
String.valueOf(poster.size()));
}
}
// get applicable cookies based on the uri and request headers
// add them to the existing request headers
setCookieHeader();
…
}
再来看看把socket响应信息解析为http的响应信息的代码：
sun.net.www.http.HttpClient.java的parseHTTP方法：
private boolean parseHTTPHeader(MessageHeader responses, ProgressSource pi, HttpURLConnection httpuc)
throws IOException {
/* If "HTTP/*" is found in the beginning, return true. Let
* HttpURLConnection parse the mime header itself.
*
* If this isn't valid HTTP, then we don't try to parse a header
* out of the beginning of the response into the responses,
* and instead just queue up the output stream to it's very beginning.
* This seems most reasonable, and is what the NN browser does.
*/
keepAliveConnections = -1;
keepAliveTimeout = 0;
boolean ret = false;
byte[] b = new byte[8];
try {
int nread = 0;
serverInput.mark(10);
while (nread < 8) {
int r = serverInput.read(b, nread, 8 - nread);
if (r < 0) {
break;
}
nread += r;
}
String keep=null;
ret = b[0] == 'H' && b[1] == 'T'
&& b[2] == 'T' && b[3] == 'P' && b[4] == '/' &&
b[5] == '1' && b[6] == '.';
serverInput.reset();
if (ret) { // is valid HTTP - response started w/ "HTTP/1."
responses.parseHeader(serverInput);
// we've finished parsing http headers
// check if there are any applicable cookies to set (in cache)
if (cookieHandler != null) {
URI uri = ParseUtil.toURI(url);
// NOTE: That cast from Map shouldn't be necessary but
// a bug in javac is triggered under certain circumstances
// So we do put the cast in as a workaround until
// it is resolved.
if (uri != null)
cookieHandler.put(uri, (Map<java.lang.String,java.util.List<java.lang.String>>)responses.getHeaders());
}
/* decide if we're keeping alive:
* This is a bit tricky. There's a spec, but most current
* servers (10/1/96) that support this differ in dialects.
* If the server/client misunderstand each other, the
* protocol should fall back onto HTTP/1.0, no keep-alive.
*/
if (usingProxy) { // not likely a proxy will return this
keep = responses.findValue("Proxy-Connection");
}
if (keep == null) {
keep = responses.findValue("Connection");
}
if (keep != null && keep.toLowerCase().equals("keep-alive")) {
/* some servers, notably Apache1.1, send something like:
* "Keep-Alive: timeout=15, max=1" which we should respect.
*/
HeaderParser p = new HeaderParser(
responses.findValue("Keep-Alive"));
if (p != null) {
/* default should be larger in case of proxy */
keepAliveConnections = p.findInt("max", usingProxy?50:5);
keepAliveTimeout = p.findInt("timeout", usingProxy?60:5);
}
} else if (b[7] != '0') {
/*
* We're talking 1.1 or later. Keep persistent until
* the server says to close.
*/
if (keep != null) {
/*
* The only Connection token we understand is close.
* Paranoia: if there is any Connection header then
* treat as non-persistent.
*/
keepAliveConnections = 1;
} else {
keepAliveConnections = 5;
}
}
……
}
对于java.net包的http，ftp等各种协议的底层实现，可以参考rt.jar下面的几个包的代码：
sun.net.www.protocl下的几个包。
在http client中也可以设置代理：
HostConfiguration conf = new HostConfiguration();
conf.setHost(host);
conf.setProxy(ip, 80);
statusCode = httpclient.executeMethod(conf,getMethod);
httpclient自己也是基于socket封装的http处理的库。底层代理的实现是一样的。
另外一种不设置代理，通过反射修改InetAddress的cache也是ok的。但是这种方法非常不推荐，不要使用，因为对于proxy代理服务器概念了解不清楚，最开始还使用这种方法，
public static void jdkDnsNoCache(final String host, final String ip)
throws SecurityException, NoSuchFieldException,
IllegalArgumentException, IllegalAccessException {
if (StringUtils.isBlank(host)) {
return;
}
final Class clazz = java.net.InetAddress.class;
final Field cacheField = clazz.getDeclaredField("addressCache");
cacheField.setAccessible(true);
final Object o = cacheField.get(clazz);
Class clazz2 = o.getClass();
final Field cacheMapField = clazz2.getDeclaredField("cache");
cacheMapField.setAccessible(true);
final Map cacheMap = (Map) cacheMapField.get(o);
AccessController.doPrivileged(new PrivilegedAction() {
public Object run() {
try {
synchronized (o) {// 同步是必须的,因为o可能会有多个线程同时访问修改。
// cacheMap.clear();//这步比较关键，用于清除原来的缓存
// cacheMap.remove(host);
if (!StringUtils.isBlank(ip)) {
InetAddress inet = InetAddress.getByAddress(host,IPUtil.int2byte(ip));
InetAddress addressstart = InetAddress.getByName(host);
Object cacheEntry = cacheMap.get(host);
cacheMap.put(host,newCacheEntry(inet,cacheEntry));
// cacheMap.put(host,newCacheEntry(newInetAddress(host, ip)));
}else{
cacheMap.remove(host);
}
// System.out.println(getStaticProperty(
// "java.net.InetAddress", "addressCacheInit"));
// System.out.println(invokeStaticMethod("java.net.InetAddress","getCachedAddress",new
// Object[]{host}));
}
} catch (Throwable te) {
throw new RuntimeException(te);
}
return null;
}
});
final Map cacheMapafter = (Map) cacheMapField.get(o);
System.out.println(cacheMapafter);
}
关于java中对于DNS的缓存设置可以参考：
1.在${java_home}/jre/lib/secuiry/java.secuiry文件，修改下面为
networkaddress.cache.negative.ttl=0 DNS解析不成功的缓存时间
networkaddress.cache.ttl=0 DNS解析成功的缓存的时间
2.jvm启动时增加下面两个启动环境变量
-Dsun.net.inetaddr.ttl=0
-Dsun.net.inetaddr.negative.ttl=0
如果在java程序中使用，可以这么设置设置：
java.security.Security.setProperty("networkaddress.cache.ttl" , "0");
java.security.Security.setProperty("networkaddress.cache.negative.ttl" , "0");
还有几篇文档链接可以查看：
http://www.rgagnon.com/javadetails/java-0445.html
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6247501
linux下关于ＯＳ ＤＮＳ设置的几个文件是
/etc/resolve.conf
/etc/nscd.conf
/etc/nsswitch.conf
http://www.linuxfly.org/post/543/
http://linux.die.net/man/5/nscd.conf
http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch18_:_Configuring_DNS
http://linux.die.net/man/5/nscd.conf
发表评论
-
对字符串进行验证之前先进行规范化
2013-09-17 23:18 13884对字符串进行验证之前先进行规范化 应用系统中经常对字 ... -
使用telnet连接到基于spring的应用上执行容器中的bean的任意方法
2013-08-08 09:17 1432使用telnet连接到基于spring的应用上执行容器中 ... -
jdk7和8的一些新特性介绍
2013-07-06 16:07 10070更多ppt内容请查看：htt ... -
java对于接口和抽象类的代理实现，不需要有具体实现类
2013-06-12 09:50 2906原文链接：http://www.javaarch.net/j ... -
Java EE 7中对WebSocket 1.0的支持
2013-06-05 09:27 3799原文链接：http://www.javaarch.n ... -
Java Web使用swfobject调用flex图表
2013-05-28 19:05 1079Java Web使用swfobject调用 ... -
spring使用PropertyPlaceholderConfigurer扩展来满足不同环境的参数配置
2013-05-21 15:57 3288spring使用PropertyPlaceholderCon ... -
java国际化
2013-05-20 20:57 4438java国际化 本文来自：http://www.j ... -
RSS feeds with Java
2013-05-20 20:52 1184RSS feeds with Java 原文来自：htt ... -
使用ibatis将数据库从oracle迁移到mysql的几个修改点
2013-04-29 10:40 1634我们项目在公司的大战略下需要从oracle ... -
线上机器jvm dump分析脚本
2013-04-19 10:48 2863#!/bin/sh DUMP_PIDS=`p ... -
eclipse远程部署，静态文件实时同步插件
2013-04-06 20:18 5418eclipse 远程文件实时同步，eclipse远程 ... -
java价格处理的一个问题
2013-03-26 21:21 1792我们经常会处理一些价格，比如从运营上传的文件中将某 ... -
java 服务降级开关设计思路
2013-03-23 16:35 3719java 服务屏蔽开关系统，可以手工降级服务，关闭服 ... -
poi解析excel内存溢出
2013-03-20 22:21 6344真是悲剧啊，一个破内部使用系统20多个人使用的后 ... -
简单web安全框架
2013-03-16 11:56 1498web安全框架,主要用servlet filter方 ... -
基于servlet的简单的页面缓存框架
2013-03-11 19:27 1177基于servlet的页面级缓存框架的基本用法： 代码参考： ... -
Eclipse使用过程中出现java.lang.NoClassDefFoundError的解决方案
2013-02-01 17:22 1474如果jdk，classpath设置正确，突然在eclipse ... -
jetty对于包的加载顺序的处理
2013-01-28 22:58 40641.问题 今天在本地和测试环境用jet ... -
hsqldb源码分析系列6之事务处理
2013-01-20 15:20 1675在session的 public Result ...
相关推荐
主要介绍了JAVA通过HttpURLConnection 上传和下载文件的方法，非常具有实用价值，需要的朋友可以参考下
java实现多次HttpURLConnection共享session，发送两次请求共享同一个session，这样做爬虫的时候就可以爬网站登录后能看到的内容了
Java自带的HttpURLConnection访问接口文件上传， 含普通参数值传递和文件传递
java HttpURLConnection示例 HttpURLConnection工具
Java 实现 客户端 HttpURLConnection 用于进行web服务器测试的http客户端
这个是我自己集成框架的时候写的一个HTTP请求的辅助类，主要是通过HttpURLConnection 来实现JAVA的HTTP请求，有的人会问、为什么有些网站发请求没问题，有些网站发请求却返回数据，在给网站发送请求的时候，请看清楚...
CUrl类是以命令行工具CUrl为参考，使用标准Java的HttpURLConnection实现的Http工具类
HttpURLConnection实现三方接口HTTP调用调用，带有FILE文件参数
基于HttpURLConnection的网络访问 已封装好，一句话访问网络 可采用GET和POST两种方式访问
用于对基于HTTP的服务进行抓取 HttpClient 现在是HttpComponents 把问题搞复杂了 其实Java内置的HttpURLConnection很好的 内含文档 使用方法见http: blog csdn net jdgdf566 article details 16357677
java网络请求工具类 HttpURLConnection post请求工具类HttpURLConnection httpURLConnection = (HttpURLConnection) url .openConnection(); httpURLConnection.setRequestMethod("POST");// 提交模式
主要介绍了java后台调用HttpURLConnection类模拟浏览器请求实例,该实例可用于接口调用,具有一定的实用价值,需要的朋友可以参考下
使用HttpURLConnection实现断点续传
分别使用HttpURLConnection OKHttp实现请求
HttpURLConnection文件下载\httpURLConnection文件下载
本人原创测试了百度，搜狗，360搜索都可以获取重定向后的真实地址，直接运行就可以，也可以根据自己项目具体修改功能。java使用HttpURLConnection获取百度搜狗360搜索链接重定向真实目标链接地址url。
本示例使用HttpUrlConnection实现上传文件，可以将手机上的文件上传到服务器 必须与my_upload工程配合运行,先启动my_upload工程
实例Demo程序来示范使用HttpURLConnection实现多线程下载。 使用多线程下载文件可以更快完成文件的下载，因为客户端启动多条线程进行下载就意味着服务器也需要为该客户端提供响应的服务。假设服务器同时最多服务100...
主要介绍了Java HttpURLConnection超时和IO异常处理的相关资料,需要的朋友可以参考下
用于基于Java HttpURLConnection类测试https代理设置的Java代码。 该代码实际上可以连接到任何URL，但仅响应https.proxyUser和https.ProxyPassword系统属性。 用法： $ java <java> -jar testwebpage.jar sherif@...