回顾一下，星期一又把system搞down了。and True-or-False questions to gurus

647i(流浪的步行万里)

since i need fresh copy of a big file between two unix boxes for two days(last weekends). the file is about 100+GB. I added a rcp line to cron job at 8am. after job done, i forgot remove the job from cron job. the source unix has production database running on it.

at 8am Monday, 60+% terminal based application loss connection, because the rcp command took 90% of the ethernet bandwidth.

I was called by another early shift admin, after 30 minutes, I killed the rcp cmd, and everything became fine.

I don't know how come heavy traffic will cause application time-out so heavyly.

True or False:
in multiple router/switch unix/windows environment
o if there is havey traffic on ethernet, all tcp package should be queued and send,
o if there is havey traffic on ethernet, some of tcp packages will be droped by switch or Unix tcp module

(#3030171@0)
Last Updated: 2006-6-15
This post has been archived. It cannot be replied.

阿德迈尔ing -buma(buma); 2006-6-15 {876} (#3030374@0)

True or False:
in multiple router/switch unix/windows environment
o if there is havey traffic on ethernet, all tcp package should be queued and send,
it will be queued in router, but in ur case, the network traffic should be ok, the problem is the server, I don't think it is configured as a network server for file transfering, thus, the TCP packets were hammering at ur server end. and I don't think your server would have a big buffer for this.
TCP protocol it self does not queue traffic, it has a buffer window though, servers that enable QoS does
o if there is havey traffic on ethernet, some of tcp packages will be droped by switch or Unix tcp module
it could be dropped by both type of devices, in ur case, it is more likey the unix server is dropping connections not ur network switches.
review the TCP/IP protocol and you will understand.

my 2 cents

.. -647i(流浪的步行万里); 2006-6-15 {1117} (#3030531@0)
True or False:
in multiple router/switch unix/windows environment
o if there is havey traffic on ethernet, all tcp package should be queued and send, it will be queued in router, but in ur case, the network traffic should be ok, the problem is the server, I don't think it is configured as a network server for file transfering, thus, the TCP packets were hammering at ur server end. and I don't think your server would have a big buffer for this. what buffer? Rwin for NIC or file system buffer? TCP protocol it self does not queue traffic, it has a buffer window though, servers that enable QoS does
o if there is havey traffic on ethernet, some of tcp packages will be droped by switch or Unix tcp module
it could be dropped by both type of devices, in ur case, it is more likey the unix server is dropping connections not ur network switches.
review the TCP/IP protocol and you will understand.
I know tcp little bit, the reliability of TCP depend on ACK and resend. if ACK and resend package is dropped, then application or TCPIP client side will show time-out

TCP protocol it self does not queue traffic <--- wrong. -iwantcar(流浪的EnjoyStudying); 2006-6-15 (#3030587@0)

Queue has special meaning in network layer. Queued packets are stored as individuals so it can be reordered. TCP buffers incoming traffic only. Bufferred traffic is a single flow of bytes so it cannot be reordered. -diresu(手气不错); 2006-6-15 (#3030657@0)

TCP buffer outgoing data too. not for the purpose of reordering. but for the purpose of device driving and resent....etc. -iwantcar(流浪的EnjoyStudying); 2006-6-15 (#3030706@0)

Outbound traffic sent by write() is not buffered for sending. -diresu(手气不错); 2006-6-15 {293} (#3030911@0)
It does put into TCP window for retransmit purpose. But write() will wait for TCP pass the data to lower SAP then return. If the lower SAP rejects the packets, write() will return error immediately. On otherhand, inbound traffic will not sent to application unless PUSH, read() or window full.

device driving -iwantcar(流浪的EnjoyStudying); 2006-6-15 (#3030929@0)

between IP package and NIC, there is queue, buffer is at connection level. -647i(流浪的步行万里); 2006-6-15 {699} (#3030705@0)
for queue on linux <p>
<pre>
# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:40:F4:91:8D:95
inet addr:192.168.2.4 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::240:f4ff:fe91:8d95/64 Scope:Link
UP BROADCAST NOTRAILERS RUNNING MULTICAST MTU:1500 Metric:1
RX packets:685486 errors:0 dropped:0 overruns:0 frame:0
TX packets:835586 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:157447765 (150.1 Mb) TX bytes:750444316 (715.6 Mb)
Interrupt:10 Base address:0xb000
</b><p>

for buffer, the 2nd and 3rd column of netstat output show the two-way buffer usage
by my understanding, the data in received queue for application is safe, it be read when appliction call read(). -647i(流浪的步行万里); 2006-6-15 {242} (#3030718@0)
what I am thinging is if the queue is full and the traffic control package can't be send from the localhost (or, can't be received by remote sender), the rest incoming package received by NIC will lost? or not

Or it will be just discard?
I don't think we are talking about the queue in the same term -buma(buma); 2006-6-15 {214} (#3030732@0)
I am talking about queueing on the network switch/router, it is not done by TCP protocol. and I think lz's production network environment should be able handle rcp file transimission, even though it is a big file.

h -buma(buma); 2006-6-15 {63} (#3030728@0)
1. the TCP stack buffer for transmitting data.
2. good enough.

when the buffer is empty, the appliction should keep waiting for the block I/O on read() call to socket. it shouldn't have error. the tcp connection is as "keep-alive" option. -647i(流浪的步行万里); 2006-6-15 {60} (#3030735@0)
Need to find out how long is the timeout for "keep-alive".

That is the darkest part of TCP/IP protocol. timeout can never be a reasonable value because lack of QOS. -iwantcar(流浪的EnjoyStudying); 2006-6-15 (#3030747@0)

what kind of Qos? let Ack, and small package to through the NIC in higher priority? the fact is the rcp runs very well(that means Send, recieve, ACK) in 10MB/s on 100Mbps NIC. -647i(流浪的步行万里); 2006-6-15 (#3030770@0)

Forget it. That topic is out of your topic. In fact, I didn't read your post carefully. I just pick out some sentence that catched my eye. timeout is possible in TCP/IP. And it is quite application specific. -iwantcar(流浪的EnjoyStudying); 2006-6-15 (#3030786@0)
Next time please write in Chinese. Too much English will kill me. -iwantcar(流浪的EnjoyStudying); 2006-6-15 (#3030795@0)

fyi -buma(buma); 2006-6-15 {348} (#3030799@0)
# AIX TCP_KEEPIDLE
tcp_keepidle=600
* Default value:14400 milliseconds.
check ur server's setup.
increase/decrease both would have dramatic impact to the server. it really depends on what ur server does.

in ur scenario, stop using rcp, it has no flow control at all except for basic TCP protocol. strong recommending to use sftp or scp

will look into it, thanks. A question, both ftp, scp are protocal/application based on TCP, the flow control is build in TCP level. unless rcp is not based on TCP(IP, UDP). -647i(流浪的步行万里); 2006-6-15 (#3030846@0)

ssh has its own flow control -buma(buma); 2006-6-15 (#3031078@0)

TCP doesn't do keep-alive. -diresu(手气不错); 2006-6-15 {396} (#3031021@0)
It does use a derived value RTO to control retransmitting. The Idle-timeout is application timmer to savage TCP resource after unexpected termination without FIN. It should be set to long enough to cover any possible silence period of the application. The timmer of 14400 is in seconds, or 4 hours make sense. (You definately don't want to lose your connection of telnet for 15 seconds of idle).

Any packets can be queued and dropped -diresu(手气不错); 2006-6-15 {940} (#3030391@0)

Telnet is an interactive program and requires normally less than 64Kbps (this is less than .1% in standard LAN). If you knew your rcp only took 90% of bandwidth, it would havd not been the problem.

Packet is queued when some network resource is 100% used. Unless the queue is emptied, such resource keeps 100% usage (except for some QoS reservations).

Packet is dropped when queue lengh increase to some level. TCP is designed to respond packet loss by decreasing the sending speed.

Back to your question:
1 if there is havey traffic on ethernet, all tcp package should be queued and send.
Depends on how heavy the traffic is. Only if some resource is used up (say bandwidth is 100% used), packets (include TCP) are queued.

2 if there is havey traffic on ethernet, some of tcp packages will be droped by switch or Unix tcp module.
Packet can be droped anywhere when lack of some resource even there is no traffic on ethernet.

That's the AIX in our environment, Before file system parameter tunned, rcp will cause swap disk 100% busy. Now, it can cause tcp time-out.以后干活都要在晚上偷偷摸摸地了 -647i(流浪的步行万里); 2006-6-15 (#3030517@0)

thanks alot to all reply. Talked to unix admin, he said he will check system parameter. Will post feedback from him later. -647i(流浪的步行万里); 2006-6-15 (#3031285@0)

@Florida

回顾一下，星期一又把system搞down了。and True-or-False questions to gurus

Replies, comments and Discussions:

More Topics