Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
HTTP persistent connection
#1
When I tried

Macro
Code:
Select All      Help
str s
rep 10
,Http h.Connect("www.xxx.yyy" "user" "password")
,h.Post("test/test.php" "name1=value1&name2=value2" s)
,out s

the time used is high.

I tried also to use several functions in parallel.

It can be done faster?

note: in wikipedia (Hypertext Transfer Protocol):

In HTTP/0.9 and 1.0, the connection is closed after a single request/response pair. In HTTP/1.1 a keep-alive-mechanism was introduced, where a connection could be reused for more than one request.

Such persistent connections reduce lag perceptibly, because the client does not need to re-negotiate the TCP connection after the first request has been sent.

Version 1.1 of the protocol made bandwidth optimization improvements to HTTP/1.0. For example, HTTP/1.1 introduced chunked transfer encoding to allow content on persistent connections to be streamed, rather than buffered. HTTP pipelining further reduces lag time, allowing clients to send multiple requests before a previous response has been received to the first one. Another improvement to the protocol was byte serving, which is when a server transmits just the portion of a resource explicitly requested by a client.
Reply
#2
I don't know much about HTTP authentication, speed, etc.

QM uses wininet functions and HTTP 1.1. I think they don't close connection after each request but not sure. Try to add INTERNET_FLAG_KEEP_CONNECTION as 4-th argument of Post.

You can read about wininet functions in MSDN library.
Reply
#3
With INTERNET_FLAG_KEEP_CONNECTION works better but still slow.

I found this:

http://www.codeproject.com/KB/IP/asyncwininet.aspx
http://www.codeproject.com/KB/cpp/WinInetRunScript.aspx

Can it be used in QM?
Reply
#4
It is not related to internet connection speed. It means that the thread can do other things while wininet functions are connecting and transferring data.
Reply
#5
I also found an activex dll: http://www.coalesys.com/products/httpcl ... efault.asp

Do you know if uses different functions that QM?

In affirmative case, could you add an example of several Post in parallel to test if is faster?
Reply
#6
Slower. Every time about 400 ms. QM - first time 400 ms then 190.

GET example
Macro
Code:
Select All      Help
out
typelib CSHttpClientLib {12CB8C40-6A6B-11D0-A74C-444553540000} 1.0
CSHttpClientLib.CSHttpClient c._create
c.RequestURL="http://www.google.com"
c.Execute("GET")
out c.ResponseBody(0)

POST example
Macro
Code:
Select All      Help
out
typelib CSHttpClientLib {12CB8C40-6A6B-11D0-A74C-444553540000} 1.0
CSHttpClientLib.CSHttpClient c._create
c.RequestURL="http://www.quickmacros.com/form2.php"
str s="a=b&c=d"
VARIANT v=s
c.RequestBody=v
c.RequestHeaders=_s.format("Content-Type: application/x-www-form-urlencoded[]Content-Length: %i[]Cache-Control: no-cache[]" s.len)
rep 5
,int t1=timeGetTime
,c.Execute("POST")
,out timeGetTime-t1
,out c.ResponseBody(0)

Don't know how to set password. Maybe in URL, but did not test.
Reply
#7
How can I measure the time of the Function_test?

Function Function_test
Code:
Select All      Help
rep 10
,mac("test_post" "" _s)
Reply
#8
Macro
Code:
Select All      Help
ARRAY(int) a
int t1=timeGetTime
rep 10 ;;max 64
,a[]=mac("test_post" "" _s)
;wait until all threads ended
if(WaitForMultipleObjects(a.len &a[0] 1 INFINITE)=WAIT_FAILED) end _s.dllerror
out timeGetTime-t1
Reply
#9
Thank you very much.

Is not possible for >64 threads?
Reply
#10
WaitForMultipleObjects supports max 64 handles.
You should not use many threads because each thread gets 1 MB of memory for stack. For example, on 256 MB RAM PC you cannot have more than about 200 threads.
Reply
#11
Almost without differences...


64 Post threads

Without INTERNET_FLAG_KEEP_CONNECTION

4943 ms
5712 ms
5218 ms
5567 ms

With INTERNET_FLAG_KEEP_CONNECTION

5053 ms
5198 ms
4903 ms
5251 ms
Reply
#12
It seems that the solution is here: http://forums.techarena.in/tips-tweaks/990999.htm

with 100 Post (time of each individual post 5-15 seconds):

before edit registry ----> 12 minutes
after edit registry -----> 4 minutes


Questions:

1) Can I do it without edit registry?
2) Can I Know the number of simultaneous connections of server for optimal performance?
3) How can I add a timeout of 1 minute in Post.

Note: sometimes any post time is very high (4-6 minutes) and if make the same post in that moment with IE the time is 5-15 seconds. Do you know why?
Reply
#13
can answer only question 3.

Member function Http.PostWithTimeout
Code:
Select All      Help
function# timeout_s $action $data [str&responsepage] [$headers] [inetflags] [str&responseheaders] [&unused]

;Posts web form data. Returns 1 on success, 0 on failure. Error on timeout.
;This function cannot post files. To post files use PostFormData.

;timeout_s - timeout in seconds.
;data - urlencoded string, eg "name=John+Smith&email=j.smith@xxx.com".
;action, responsepage, headers, inetflags, responseheaders - the same as with PostFormData.
;unused - used internally.

;See also: <Http.PostFormData>.


if(!&unused)
,if(!m_hi) end ES_INIT
,__HInternet _hi; int th _ret
,th=mac("__Http_PostWithTimeout" "" &this action data &responsepage headers inetflags &responseheaders &_hi &_ret)
,wait timeout_s H th
,err
,,InternetCloseHandle _hi; _hi.handle=0 ;;ends thread
,,wait 0 H th ;;because the thread uses this. ~0.5ms
,,end _error
,ret _ret

lpstr sh="Content-Type: application/x-www-form-urlencoded"
if(empty(headers)) headers=sh
else if(findrx(headers "(?i)^Content-Type *:" 8)<0) headers=_s.from(sh "[]" headers)

unused=HttpOpenRequest(m_hi "POST" action 0 0 0 INTERNET_FLAG_RELOAD|inetflags 0); if(!unused) ret Error
if(!HttpSendRequest(unused headers -1 data len(data))) ret Error

if(&responseheaders and !GetResponseHeaders(unused responseheaders)) ret Error

if(&responsepage) ret Read(unused responsepage)
ret 1

Function __Http_PostWithTimeout
Code:
Select All      Help
;\
function Http&h $action $data str&responsepage $headers inetflags str&responseheaders &handle &_ret

_ret=h.PostWithTimeout(0 action data responsepage headers inetflags responseheaders handle)
Reply
#14
Can Winhttp be better than Wininet (http://msdn.microsoft.com/en-us/magazine/cc716528.aspx) to do this?

Can you add an example of

rep 10
,mac("test_post" "" _s)

using winhttp (this API has Timeout, Asynchronous...) to test?

Thanks in advance.
Reply
#15
WinHTTP is available on XP SP1 and later. Is it OK?
Reply
#16
http://msdn.microsoft.com/en-us/library/aa384276(VS.85).aspx

download

http://www.dlldll.com/winhttp.dll_download.html
Reply
#17
Much to learn. I don't think it will be better than you already have in QM.
Reply
#18
http://www.autoitscript.com/forum/index ... opic=84133
Reply
#19
Need declarations?
More Windows API declarations for QM
example
Macro
Code:
Select All      Help
WINAPI2.WinHttpSendRequest
Reply
#20
No, I wonder if Winhttp functions are quicker than Wininet

Http.PostWithTimeout uses 2 threads and I need run more than 100 Post...
Reply
#21
I'm trying understand the function InternetSetOption

How can I create a Http member function to set:

INTERNET_OPTION_MAX_CONNS_PER_SERVER
INTERNET_OPTION_MAX_CONNS_PER_1_0_SERVER
INTERNET_OPTION_CONNECT_TIMEOUT
INTERNET_OPTION_RECEIVE_TIMEOUT
INTERNET_OPTION_FROM_CACHE_TIMEOUT

another question:

if set INTERNET_OPTION_RECEIVE_TIMEOUT is necessary to use Http.PostWithTimeout?
Reply
#22
The timeout options don't work, that is why I used another thread.

The options probably should be set on the hi variable that is in Http.Post. It is not a member variable and you cannot set options on it with a member function. Copy Http.Post and edit.

After

__HInternet hi=HttpOpenRequest(m_hi "POST" action 0 0 0 INTERNET_FLAG_RELOAD|inetflags 0); if(!hi) ret Error

add

Member function Http.Post2
Code:
Select All      Help
int option(10) optionlen(4); if(!InternetSetOption(hi INTERNET_OPTION_MAX_CONNS_PER_SERVER &option &optionlen)) ret Error

Not tested.
Reply
#23
Can I add it in Http.PostWithTimeout?

other question:

Which you believe that it is the better option to send several Posts and receive the answer?
Reply
#24
Yes, instead of hi there is unused. But maybe the option should be set on m_hi or m_hitop. Try all.

I don't know what is the best. test all options and you will see.
Reply
#25
Ok.

I will test all options.
Reply
#26
I tried it but:

unused, m_hi ------> Error: Failed to create thread __Http_PostWithTimeout. Max 1000 threads allowed.
m_hitop -----------> Error (RT) in Http.PostWithTimeout: Exception 0xC0000005. Access violation. Cannot read memory at 0x0. In at 0x0 (0x0+0x0).

Can you modify Http.PostWithTimeout?
Reply
#27
QM does not allow more than 1000 threads.
Don't know where is the exception. Maybe because the thread was not created.
Reply
#28
Http.PostWithTimeout without line

int option(5) optionlen(4); if(!InternetSetOption(m_hi INTERNET_OPTION_MAX_CONNS_PER_SERVER &option &optionlen)) ret Error

works

note: I only run 5 Http.PostWithTimeout.
Reply
#29
I tested, works like without the line. But maybe my testing code is different.
Reply
#30
My Http.PostWithTimeout

Member function Http.PostWithTimeout
Code:
Select All      Help
function# timeout_s $action $data [str&responsepage] [$headers] [inetflags] [str&responseheaders] [&unused]

;Posts web form data. Returns 1 on success, 0 on failure. Error on timeout.
;This function cannot post files. To post files use PostFormData.

;timeout_s - timeout in seconds.
;data - urlencoded string, eg "name=John+Smith&email=j.smith@xxx.com".
;action, responsepage, headers, inetflags, responseheaders - the same as with PostFormData.
;unused - used internally.

;See also: <Http.PostFormData>.


if(!&unused)
,if(!m_hi) end ES_INIT
,__HInternet _hi; int th _ret
,th=mac("__Http_PostWithTimeout" "" &this action data &responsepage headers inetflags &responseheaders &_hi &_ret)
,wait timeout_s H th
,err
,,InternetCloseHandle _hi; _hi.handle=0 ;;ends thread
,,wait 0 H th ;;because the thread uses this. ~0.5ms
,,end _error
,ret _ret

lpstr sh="Content-Type: application/x-www-form-urlencoded"
if(empty(headers)) headers=sh
else if(findrx(headers "(?i)^Content-Type *:" 8)<0) headers=_s.from(sh "[]" headers)

unused=HttpOpenRequest(m_hi "POST" action 0 0 0 INTERNET_FLAG_RELOAD|inetflags 0); if(!unused) ret Error
int option(5) optionlen(4); if(!InternetSetOption(m_hi INTERNET_OPTION_MAX_CONNS_PER_SERVER &option &optionlen)) ret Error

if(!HttpSendRequest(unused headers -1 data len(data))) ret Error

if(&responseheaders and !GetResponseHeaders(unused responseheaders)) ret Error

if(&responsepage) ret Read(unused responsepage)
ret 1
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)