Tip of the Week: A TCP flags and handshakes walk-through

Several weeks back in my initial Tip of the Week, I discussed why running a simple, constant ping to troubleshoot and/or validate TCP applications was not enough because of its reliability factor.
I suggested that, in order to get to the crux of a TCP application issue, a trace must be taken to substantiate the data transmissions/call flow and, ultimately, render a decision for root-cause analysis. This week, I'll expand on that bit of advice by focusing on TCP flags and handshakes and why they're important with regard to TCP applications.  
TCP segments carry data while others are simply acknowledging previously received data. The popular three-way handshake utilizes the SYNs and ACKs available in the TCP stack to help complete the connection before data is transferred. Each TCP segment has a purpose, and this is determined with the help of the TCP flag options that allow the sender or receiver to specify which flags should be used. This ensures the segment is handled correctly by the other end of the transmission. Take a look at the TCP flags…
 describe the image
Please note the two flags that are used in a three-way handshake (SYN, ACK) and data transfers. With regard to flags, a value of "1" means that a particular flag is set or on. Each flag is one bit long, hence there are six flags, making the flags section of the TCP/IP header six bits.  The most popular flags are the SYN, ACK and FIN. These flags are used to establish connections, acknowledge successful segment transfers, and terminate connections. The rest of the flags are not well-known, but they are equally as important.  At a high level, one can see that TCP flags would be an excellent place (amongst many) to check when analyzing a trace to figure out a TCP application transmission issue. Please allow me to bring some savvy to this matter so that analyzing traces in the future will not be so painful (if it is currently :->).

Technical Flag and Handshake Analysis

Urgent Pointer Flag

The urgent pointer flag is used to identify incoming data as urgent. The incoming segments do not have to wait until the previous segments are consumed by the receiving end but are sent directly and processed immediately. An urgent pointer could be used during a stream of data transfer and stop the data processing on the other end. The abort signal will be sent and queued at the remote machine until all previously sent data is processed. By setting the abort signal’s segment Urgent Pointer flag to "1," the remote machine will not wait until queued data is processed to execute the abort. Instead, it will give that specific segment priority, processing it immediately and stopping all further data processing.

ACKnowledgement Flag

The acknowledgement flag is used to acknowledge the successful receipt of packets.  If you run a packet sniffer while transferring data using TCP, you will notice every packet you send or receive is followed by an Acknowledgement. If you received a packet from a remote host, then your workstation will most probably send on back with the ACK field set to "1." Hmm, if a customer has identified he/she might be losing packets, I wonder what flag one should check when following the TCP stream?  RIGHT, now you are in the know.

Push Flag

The push flag (like the urgent flag) exists to ensure that the data is given the priority it deserves and is processed at the sending or receiving end. This flag is used quite frequently at the beginning and end of a data transfer, affecting the way it is handled at both ends.  A critical point to mention about the push flag is that it is usually set on the last segment of a file to prevent buffer deadlocks. It is also seen when used to send HTTP or other types of requests through a proxy ensuring the request is handled appropriately.

Reset Flag

The reset flag is used when a segment arrives that is not intended for the current connection. If you were to send a packet to a host in order to establish a connection, and there was no such service waiting to answer at the remote host, the host would automatically reject the request and then send you a reply with the RST flag set. This indicates that the remote host has reset the connection. Another point about the reset flag is that most hackers use this feature to scan hosts for open ports. All modern (Angry IP :->) port scanners are able to detect open or listening ports thanks to the reset function.

SYNchronization Flag

The synchronization flag is perhaps the best-known flag in TCP communications. The SYN flag is initially sent when establishing the classic three-way handshake between two hosts:
 describe the image
In the example above, host A needs to download data from host B using TCP as its transport protocol. The protocol requires the three-way handshake to take place so a virtual connection can be established by both ends in order to exchange data. Within a three-way handshake, we are able to count a total of two SYN flags transmitted, one by each host.  As files are exchanged and new connections created, more SYN flags will be sent and received. Hopefully, it is starting to sink in where to check traces when analyzing TCP transmissions. This stuff is the bedrock of all TCP connectivity (not by watching echo replies and/or request time outs  ;->).

FIN Flag

The FIN flag is used to tear down the virtual connections created using the previous SYN flag.  The FIN flag always appears when the last packets are exchanged between hosts or connections. It is important to note that when a host sends a FIN flag to close a connection that it may continue to receive data until the remote host has also closed the connection. Once the connection is torn down by both sides, the buffers set aside on each end for the connection are released. This is what a normal teardown procedure should look like in a trace:
 Tearing Down a Connection resized 600
In the example above, there is an existing connection transmitting between Host A and B.  Once the data transfer is complete, Host A sends a packet with the FIN, ACK flags set.  Host A is ACKknowledging the previous stream while at the same time initiating the TCP close procedure to kill the connection. At this point, Host A’s application will stop receiving an data and will close the connection from this side. In response to Host A’s request to close the connection, Host B will send an ACKnowledgement back, and also notify its application that the connection is no longer available.  Once this is complete, host B will send its own FIN, ACK flags to close their part of the connection.
I have tried to lay out with as little detail as possible (making for an easier read) an overview on how Flag and Handshake functionality might make troubleshooting a TCP application issue a bit more manageable. If one can master TCP Flags and Handshaking as well as TCP windowing (possible explanation in next week’s tip  :->), then identifying root cause analysis for TCP transmission issues by taking a sniffer trace becomes really easy.  I use the word easy because it just takes practice, and the more you’re in the know of how things work, practicing only makes the overall process more efficient (as with just about everything in life).
If you have any questions, feel free to post in the comments section below.