Here is my confusion -carletonwong(自己修车) @佛州华人论坛:佛州枫下论坛 The Rolia Forum of Florida

工作学习 / 学科技术讨论 / [Q]how to download files using HttpWebRequest via ssl (https). Thanks -carletonwong(自己修车); 2008-6-6 (#4488421@0)

Try WebClient.DownloadFile Method. -readonly(ReadOnly); 2008-6-6 (#4488446@0)

Check you email please? -carletonwong(自己修车); 2008-6-6 (#4488505@0)

具体我没做过，主要没有SSL环境，按.NET文档，对.NET程序员是没有区别的，只要用https代替http就行了。 -readonly(ReadOnly); 2008-6-6 (#4488666@0)

I implemented samiliar request before. the difficult part is certificate. If the server doesn't require client certificate, that should be easy. -holdon(again); 2008-6-6 (#4488562@0)

[Q]do you mean that not all https require client certificate? -carletonwong(自己修车); 2008-6-6 {296} (#4488865@0)
Could you email me your sample code please? (https/ssl).
Do you know where to find the downloaded client certificate please? (I can
navigate the web sites using IE browser). I need to get the certificate and save
to a file before I can loaded it in my code.

Thanks

carletonwong@yahoo.com

most ssl server doesn't require client certificate. they only has a server certificate to prove they can be trusted. -holdon(again); 2008-6-6 {1068} (#4489039@0)
本文发表在 rolia.net 枫下论坛only some private ssl server require client certificate signed by themself, so without client certificate, you can't even browse their website. so if you don't have a client certificate, you probably don't need to worry about it.

about server certificate, there are two situation:

1. server has a valid certificate signed by trusted root, then you don't need to do any thing, just call httpwebrequest, or webrequest? , that should do it.

2. server only has a self signed "fake" certificate ( you need to pay money to buy real one ), then you need to do extra work to let your client trust the server, even the certificate is fake.

so now you need to find the answer, does your https server has real certificate? you can easily find it by access the https server by browser. if browser doesn't compalin, then it's a real one (unless your machine already trusted it before, or by some group policy.)

If it's fake, you can google "bypass ssl certificate .Net“, here is one link I found:

http://west-wind.com/weblog/posts/48909.aspx

I更多精彩文章及讨论，请光临枫下论坛 rolia.net

Here is my confusion -carletonwong(自己修车); 2008-6-8 {557} (#4490559@0)
Yes, I can just https the website using IE without any problem at all. Thus, according to your instruction, it does not require a client certificate. My confusion is that if without a client certificate,
how does client side send encryped information to the server (for example, userid, password). When the server side sends back confidential info (etrade trading information and portfolio), how
does client side deciper it?
Your further advice is appreciated.

P.S.
It is very appreciated if you can share sample code that https a public ssl website.

certificate is not for encryption, it's for identification. you only need a key to encrypt data. as long as only server and client know the key, nobody else can decrypt it. -holdon(again); 2008-6-8 {1188} (#4490678@0)
本文发表在 rolia.net 枫下论坛ssl use private/public key pair to pass a encryption key between them so the key is only known by the server and client.

server certificate is just a way to tell you this server is trusted by some wellknown institution, so you can trust it too.

to download a file from https, it can be as easy as

WebClient client = new WebClient ();
client.DownloadFile("https://www.paypal.com/en_US/i/logo/paypal_logo.gif", "1.gif");

try
{
client.DownloadFile("https://64.4.241.33/en_US/i/logo/paypal_logo.gif", "2.gif");
}
catch (Exception e)
{
string str = e.ToString();
}
see the second try failed, cause the server's certificate is not match.

But now I guess what you want might be more complicated: login to a https site with username and password then download a file.

Then you have to go back to httpwebrequest, send a request to ssl server with POST username and password, analyse the result and find out something like SESSIONKEY or cookie, then send back the server another request with the session key ( to let the server know you are a login user) to download the file.

you need to understand http protocol to do it.更多精彩文章及讨论，请光临枫下论坛 rolia.net

webrequst -carletonwong(自己修车); 2008-6-8 {327} (#4491466@0)
Yes, my situation is log to the website using https and navigate one or two pages before downloading the files (all https/ssl). What I understood is that information from my computer
to the website is encrypted and vice versa all infromation from the server to my machine are
encrypted as well.
{Q] where do I get the key?

short Answer: you don't need to care. -holdon(again); 2008-6-8 {629} (#4491583@0)
a little more explaination is ssl server has private/public key pair, any data encrypted by public key can be only decrypted by private key ( vice versa) ; public key is public to every body as part of the server certificate, private key is only known by the server.

your client access a server, get the public key and verify the certificate, generate a random key, encrypt it by public key, then send it to server; server decrypt it by private key to get the random key, then encrypt any data sent to you by the random key.

if you need to fully understand how ssl work ( I don't either ), you need to read ssl document.

How to send a client certificate by using the HttpWebRequest and HttpWebResponse classes in Microsoft Visual C# .NET -readonly(ReadOnly); 2008-6-9 (#4492346@0)

From my understating -deep_blue(BLUE); 2008-6-9 {694} (#4492199@0)
When building a client application to access a SSL web server, generally you don’t need worry about Certificate issue. However, you need Credentials info. In that case, you can easily use WebClient to download file from SSL web server:
WebClient client = new WebClient(); client.Credentials = new NetworkCredential(uid, password, domain); // or new NetworkCredential(uid, password); client.DownloadFile(URL, fileName);

If the SSL web server does require Certificate info, you can try System.Security.Cryptography.X509Certificates.X509Certificate to instantiate Certificate value and put it into HttpWebRequest object. Then from HttpWebResponse object to get data from server.

it's really depend on how the web server secure the data file OP needed. Credential only works when the data file is actually existing on the website, just protected by user permission ( no anonymouse user access) -holdon(again); 2008-6-9 {524} (#4492696@0)
if the data file is not existing on website ( could be in database ) and you have to go though a login page, then download the file by some server page, then Credential won't work.

About client certificate, yes you need to create a X509 certificate, but before that , you have to load the private key into that machine. a X509 certificate is just a signed public key , it won't work without the related private key. One thing I didn't find in .Net library is a function to load a private key from a private key file.

use "wget" if you didn't mean to write your own program -lachom(注册啦 (免费的)！); 2008-6-9 (#4492204@0)

yes, if no login is required, wget could be a easy solution. it has all the functions for https:--no-check-certificate, --certificate=FILE,--private-key=FILE,--header=STRING -holdon(again); 2008-6-9 (#4492668@0)

[Q]Thanks for all previous discussion. Reorganize my question -carletonwong(自己修车); 2008-6-9 {1016} (#4493024@0)
1. I need to login in to website and navigate a couple of pages before download files. These are
all https/ssl. I do not have any problem at all using IE browser with userid and password. But I do not know how to do the same things using webrequest/C#.

2. I have tried all methods that all friends mentioned here. No luck. (I have tried all of them before
I posted question).

3. I initially thought using X509Certificate. After disucssed with holdon, I agree we do not need it
since I did not use any certificaate while using IE browser.

4. holdon also mentioned public/private keys. I guess we need to figure out how IE Browser does
the job before using WebRequest API(s).

5. I guess the problem is
a. how to encrypt my request messge before sending to the server. (login requirement)
b. how to keep the same credential for the rest of navigation after login.

I do not think we need a private key since public key with credential can make the client
unique. However, where is the public key?

You can start with a non ssl site, sent it a sequence of HTTP POST, parse the response result. -holdon(again); 2008-6-9 {537} (#4493252@0)
you have to read http protocol to understand what you need to send to the server.

Say the request is automatically login to rolia and reply a simple "hello" to this thread, if you can do this, you will have no problem to login to a ssl web site.

5. I guess the problem is
a. how to encrypt my request messge before sending to the server. (login requirement)
b. how to keep the same credential for the rest of navigation after login.

You don't need to worry about it. HttpWebRequest handle all the ssl detail for you!

回你的贴兼回楼主吧 -readonly(ReadOnly); 2008-6-10 {355} (#4493560@0)
a: encrpt/decrpt由SSL实现。
b:如果server端authentication 是Form, 用HttpWebRequest.CookieContainer Property；
如果server端authentication 是Windows, 用HttpWebRequest.Credentials Property ；

关于如何用HttpWebRequest发送POST的请求，去GOOGLE吧。

说点题外话，用SSL，说明安全要求比较高，要我做就会只允许browser访问，所以有可能是不让HttpWebRequest访问的。

how can you "only allow browser" access? web server only identify client by header and you can put any header in your request. a Opera brower can claim it's IE, so does HttpWebRequest. -holdon(again); 2008-6-10 (#4493579@0)

从安全性的角度，HttpRequest.UserAgent Property 应该是不好修改的，有人能改就是黑客技术了。 -readonly(ReadOnly); 2008-6-10 (#4493604@0)

it doesn't need any hacker skill, even wget can do it. there is no way web server can really tell who is on the other side. it could be even telnet client for http. -holdon(again); 2008-6-10 {175} (#4493644@0)
for https, not 100% sure about httpwebrequest ( but I think it shouldn't be a problem, it's just a header above all) , but at least sslClientStream can send anything I want.

关于黑客，一是指黑客技术，二是指用黑客软件。我回答的是关于如何名门正派地设计软件，对于你关于如何冒名是另外一个话题。看看你回答的问题都是啥，满嘴跑火车。 -readonly(ReadOnly); 2008-6-10 (#4493678@0)

couln't understand since when changing header become hacker technology. -holdon(again); 2008-6-10 (#4493683@0)
and wget is a hacker software i suppose? -holdon(again); 2008-6-10 (#4493687@0)

HttpWebRequest.UserAgent Property public string UserAgent {get; set;}[C#] Gets or sets the value of the User-agent HTTP header. how difficult is that? Hacker tech??? -holdon(again); 2008-6-10 (#4493704@0)

哈哈，查了查，是可以的修改HttpWebRequest.UserAgent Property ，我指出你那么多错，估计你也心成不服，让你扳回一局，应该是心里平衡一些。 -readonly(ReadOnly); 2008-6-10 (#4493725@0)

yeah, right. -holdon(again); 2008-6-10 (#4493728@0)

想了想，还是可以做到的，在login网页里藏动态生成安全认证字符串（或藏在cookie里），并且只认可postback的login，这样只能从browser里登陆。 -readonly(ReadOnly); 2008-6-10 (#4493830@0)

yes, you are partly right for the dymamic string, but it must be image. any string in cookie ,or require POST BACK login should be peice of cake for "senior hacker" or intermidiate web developer. -holdon(again); 2008-6-10 (#4493863@0)

实现是肯定没问题，初级就可以，展示一下你如何用HttpWebRequest的风采吧。 -readonly(ReadOnly); 2008-6-10 (#4493922@0)

I agree using CookieContainer though the server is not IIS. Credentials is only for company domains. This server is public commercial webserver (Internet) and has nothing to do with client application. -carletonwong(自己修车); 2008-6-10 {113} (#4493950@0)
I wonder if anyone really try his/her proposal using an Internet https/ssl server.

More advice is appreciated.

Simple web File download in VB.NET -readonly(ReadOnly); 2008-6-10 (#4494197@0)
POST WebRequest -readonly(ReadOnly); 2008-6-10 (#4494199@0)
跟我前面说的，可能HttpWebRequest行不通（server端安全性的机制）。要是能行但你还是做不出来，我觉得你吃这碗饭很累。 -readonly(ReadOnly); 2008-6-10 (#4494208@0)
It may be some other reasons rather than SSL. -deep_blue(BLUE); 2008-6-11 {744} (#4495658@0)
When downloading file from static url, it’s simple to use WebClient.

But in order to secure data file, data file cannot locate in web site. Instead, server code is used to retrieve data file from somewhere. In that case, client request should contain proper data for triggering server side event. You cannot put data into WebClient on client side. Even with HttpWebRequest, it’s hard to put proper data for triggering server side event. If server side uses .NET, it’s even more complicated, due to postback mechanism and viewstate are used.

BTW, Credentials can be used for both Windows authentication (what you called company domains) and Forms authentication (it’s used by most internet application.)

诚心请教，能否贴点代码关于 Credentials can be used for Forms authentication (it’s used by most internet application.) -readonly(ReadOnly); 2008-6-11 (#4495715@0)
Really? That's something new to know. Thanks, I will check how it work later. -holdon(again); 2008-6-11 {1330} (#4495733@0)
本文发表在 rolia.net 枫下论坛> Credentials can be used for both Windows authentication (what you called company domains) and Forms authentication

From my understanding, there are mainly two ways to secure web content:

HTTP level and application level.

HTTP level security is impelemented directly in http protocol, you can get permission from any entry of a web site. standard HTTP only support two authentication: BASIC and another one ( digest? never saw it used ). Microsoft added its own NTLM authentication. .Net Windows authentication is using NTLM authentication. since NTLM is not standard HTTP and not all the browser support it, few people will use it in internet application. from my point of view, it's same as BASIC , you see a popup window asking username and password.

Application level security will have a single login enery for whole site, once login server remember client's login information by cookie or url parameter. .Net Form authentication is using application level security. Most other language web server ( php, perl, java? ) use application level security.

I thought .Net Credentials is only for Http level application ( I believe it also work for BASIC authentication), but from what you say, it also work for .Net Form authentication ? But one thing for sure, it won't work for other language web server.更多精彩文章及讨论，请光临枫下论坛 rolia.net
HttpWebRequest.Credentials Property: Supported authentication schemes include Digest, Negotiate, Kerberos, NTLM, and Basic. -readonly(ReadOnly); 2008-6-11 (#4495745@0)

The solution in C++. Although the sample code is about soap, it is easy to change it to handle https. If cookie was used by website, just looking for them in raw http headers and add them when sending request later. -liquid(豆泡松果之松果豆泡); 2008-6-11 (#4496622@0)

@Florida

Here is my confusion

Replies, comments and Discussions:

More Topics