I know this is a little late but here you go
"The basics and must-know of networking + writing your first Socket program"
* Who is recommended to read this?
- This is for any beginner who wants to get their hands on networking.
- For any programmer who wants to refresh their networking skills.
* Prerequisite:
- Basics of C programming language.
- Sure you can read this without any programming knowledge but you will have trouble following along at some part. ( Tell me why you even read without any programming knowledge )
* IMPORTANT NOTE:
- If you have any doubt or confusion reading along, don't worry just keep going as some concepts will get clear later in the ending of the article.
---xxxxxx---
INTRODUCTION:
A common language and sign allow us to communicate, share experiences and transfer information or knowledge.
Our human race wouldn't be so progressive and connected without communication. Similarly, communication as networking plays a great role in the computer world.
Networking allows programs to communicate with other programs locally or remotely. A web browser wouldn't be so useful without its ability to communicate with web servers.
Networking has become a part of our day-to-day life. Services like email, the web, and instant messaging rely on networking.
The above applications or services rely on a particular network protocol but each protocol uses the same general network transport method (OSI model)
It's like we may speak different languages but the way/method the sound travels from our mouth to someone's ear remains the same.
Let us first understand this general network transport method or simply the OSI model.
---xxxxxx---
* IMPORTANT NOTE:
- If you have any doubt or confusion reading along, don't worry just keep going as some concepts will get clear later in the ending of the article.
The OSI Model:
The OSI (Open Systems Interconnection) model provides standards that allow hardware, such as routers and firewalls, to focus on one particular aspect of communication that applies to them and ignore others.
The OSI model is broken down into conceptual layers of communication. This way, routing and firewall hardware can focus on passing data at the lower layers, ignoring the higher layers of data encapsulation used by running applications.
The seven OSI layers are as follows:
Physical layer:
This layer deals with the physical connection between two points. This is the lowest layer, whose primary role is communication raw bit stream.
This layer includes your network interface cards, network cables, or hubs.
Data-link layer: In contrast with the physical layer, which takes care of sending the raw bits, this layer provides high-level functions, such as error correction and flow control.
Your MAC (Media Access Control) address and Switches get introduced in this layer.
Network layer: This layer works as a middle ground; its primary role is to pass information between the lower and the higher layers.
Your IP address gets introduced in this layer.
Transport layer: This layer provides reliable data communication allowing the higher layers to never worry about the reliability or cost-effectiveness of data transmission.
TCP, UDP protocols, and Port numbers get introduced here.
Session layer: This layer is responsible for establishing and maintaining connections between network applications.
Presentation layer: This layer is responsible for presenting the data to applications in syntax or language they understand.
This allows for things like encryption and data compression.
Application layer: This layer is concerned with keeping track of the requirement of the application.
This layer is where your HTTP (HyperText Transfer Protocol), SMTP (Simple Mail Transfer Protocol), FTP (File Transfer Protocol), etc gets introduced.
Just think of the above protocol as different languages just like we humans have different languages. The WEB uses HTTP, SMTP is used for receiving and sending emails, FTP for file transfer.
The above image I stole from the internet (hacker ;) well defines the OSI model.
---xxxxxx---
When data is transferred from one computer to another, it is sent in small pieces called packets.
Each packet contains implementations of these protocol layers. Starting from the application layer, the packet wraps the presentation layer around that data, which wraps the session layer, which wraps the transport layer, and so forth.
This process is called encapsulation.
Each wrapped layer contains a header and a body. The header contains the protocol information needed for that layer, while the body contains the data for that layer. The body of one layer contains the entire package of previously encapsulated layers, like the skin of an onion.
Confused? It's ok. An example will clear most of your doubts.
---xxxxxx---
EXAMPLE SCENARIO:
Whenever you browse the Web, the Ethernet cable and your network card make up the physical layer, taking care of the transmission of raw bits from one end of the cable to the other.
The next layer, the data-link layer adds support for MAC address, or in other words the data-link provides the low-level communications between devices on LAN.
MAC address is used as an identification for a device on LAN (Local Area Network).
The concept of IP address doesn't exist until the next layer, the network layer. These three lower layers together are able to send packets of data from one IP address to another.
Other addressing schemes exist at this layer; however, your web traffic probably uses IP version 4 (IPv4). IPv4 addresses follow a familiar form of XX.XX.XX.XX. IP version 6 (IPv6) also exists on this layer, with a totally different scheme. Since IPv4 is most common, IP will always refer to IPv4 in this article.
I am quite sure you must have come across the term TCP/IP. It actually describes the use of TCP on the transport layer and IP on the network layer.
As you have already learned, UDP and TCP get introduced in the transport layer. Wait! What the hell are these TCP and UDP?
Ok...TCP (Transmission Control Protocol) provides a seamless bidirectional socket connection or simply a type of connection that is reliable and ensures no packet loss on the destination.
While UDP (User Datagram Protocol) is used for establishing low-latency and loss-tolerating connections between applications on the internet.
In our example of web browsing, we are using TCP as we don't want any data losses.
But who wants data loss anyway? Why do we need UDP then?
UDP is used in services like media-streaming where losing frames are ok.
There are many advantages to using UDP over TCP and vise-versa.
Another images I stole from the internet will help you understand the differences
Going back to our example:
The web traffic itself uses HTTP (Hypertext Transfer Protocol) to communicate, which is in the top layer of the OSI model.
When you browse the Web, the web browser on your network is communicating across the Internet with the web-server located on a different private network.
When this happens, the data packets are encapsulated down to the physical layer where they are passed to a router. Since the router isn't concerned with what's actually in the packets, it only needs to implement protocols up the network layer. The router sends the packets out to the Internet, where they reach the other network's router. This router then encapsulates this packet with the lower-layer protocol headers needed for the packet to reach its final destination.
This process is shown in the following illustration.
All of this packet encapsulation makes up a complex language that hosts on the Internet (and other types of networks) use to communicate with each other.
These protocols are programmed onto routers, firewalls, and your computer's operating system so they can communicate.
Programs that use networking, such as web browsers and email clients, need to interface with the operating system which handles the network communications.
Since the operating system takes care of the details of network encapsulation, writing network programs is just a matter of using the network interface of the OS.
---xxxxxx---
Now let's actually learn to network an application.
* SOCKETS:
A socket allows a programmer to control the details of the OSI model described above. The lower layers of the OSI model are handled by the OS so we just need to handle the details from the transport layer and above.
Simplifying everything, to a programmer a socket can be used to send and receive data over a network.
There are several different types of sockets that determine the structure of the transport layer. The most common types are Stream sockets and Datagram sockets.
Stream socket just uses TCP on the transport layer while datagram uses UDP on the transport layer.
* Over-explaining TCP/UDP with stream socket and datagram sockets. You can skip this part if you are booooored (you won't lose much).
Stream sockets provide reliable two-way communication similar to when you call someone on the phone. One side initiates the connection to the other, and after the connection is established, either side can communicate to the other. In addition, there is immediate confirmation that what you said actually reached its destination. Stream sockets use a standard communication protocol called Transmission Control Protocol (TCP).
TCP is designed so that the packets of data will arrive without errors and in sequence, like words arriving at the other end in the order they were spoken when you are talking on the telephone.
Another common type of socket is a datagram socket. Communicating with a datagram socket is more like mailing a letter than making a phone call. The connection is one-way only and unreliable. If you mail several letters, you can't be sure that they arrived in the same order, or even that they reached their destination at all.
---xxxxxx---
For those who don't learn C yet, see you after some programming basics.
* SOCKET FUNCTIONS:
As we have learned that sockets allow a programmer to communicate on a network, let's dive into actually networking an application.
Note: we will be focusing strictly on C-based language. Just a reminder, you can be a decent programmer only with a strong C knowledge.
Now going back to Socket Functions
* IMPORTANT NOTE (hell once again!):
- If you have any doubt or confusion reading along, don't worry just keep going as some concepts will get clear later in the ending of the article.
WHY do I keep saying this? It's because I tend to get lazy and quit easily when I try to learn something I am not familiar with. And I don't want you to be like me and quit in the middle...
In C, sockets behave a lot like files since they use file descriptors to identify themselves. You don't know file descriptors? Seriously?
Just kidding :) It's ok...
For those who don't know
File Descriptor: A unique identifier for a file or other input/output resource, such as a pipe or network socket. File descriptors typically have non-negative integer values, with negative values being reserved to indicate "no value" or error conditions.
If you are still stuck with File Descriptor, open a new browser tab
Sockets behave so much like files that you can actually use the read() and write() functions to receive and send data using socket file descriptors.
However, there are several functions specifically designed for dealing with sockets. These functions have their prototypes defined /usr/include/sys/sockets.h
It's totally fine if you can't understand the functions below, as we go along using them you will learn them.
Explanation of the function arguments below after this section.
socket(int domain, int type, int protocol)
Used to create a new socket, returns a file descriptor for the socket or -1 on error.
connect(int fd, struct sockaddr *remote_host, socklen_t addr_length)
Connects a socket (described by file descriptor fd) to a remote host. Returns 0 on success and -1 on error.
bind(int fd, struct sockaddr *local_addr, socklen_t addr_length)
Binds a socket to a local address so it can listen for incoming connections. Returns 0 on success and -1 on error.
listen(int fd, int backlog_queue_size)
Listens for incoming connections and queues connection requests up to backlog_queue_size. Returns 0 on success and -1 on error.
accept(int fd, sockaddr *remote_host, socklen_t *addr_length)
Accepts an incoming on a bound socket. The address information from the remote host is written into the remote_host structure and the actual size of the address structure is written into *addr_length. This function returns a new socket file descriptor to identify the connected socket or -1 on error.
send(int fd, void *buffer, size_t n, int flags)
Sends n bytes from *buffer to socket fd; returns the number of bytes sent or -1 on error.
recv(int fd, void *buffer, size_t n, int flags)
Receives n bytes from socket fd into *buffer; returns the number of bytes received or -1 on error.
* Socket function(explanation of above function arguments):
When a socket is created with the socket() function, the domain, type, and protocol of the socket must be specified. The domain refers to the protocol family of the socket.
A socket can be used to communicate using a variety of protocols, from the standard Internet Protocol used when you browse the Web to amateur radio protocols such as AX.25. These protocol families are defined in bits/sockets.h, which are automatically included from sys/sockets.h
As mentioned before, there are several types of sockets, although stream sockets and datagram sockets are the most commonly used. The type argument above in the socket function just refers to this. The types of sockets are also defined in bit/socket.h
The final argument for the socket() function is the protocol, which should almost always be 0. The specification allows for multiple protocols within a protocol family, so this argument is used to select one of the protocols from the family. In practice, however, most protocol families only have one protocol, which means this should usually be set for 0; the first and only protocol in the enumeration of the family. This is the case for everything we will do with sockets in this article, so this argument will always be 0 in our examples.
* Socket Addresses:
Many of the socket functions reference a sockaddr structure as their argument, to pass address information that defines a host. This structure is also defined in bits/socket.h, as shown below.
You may find this socket address a little bit confusing but this should not be a problem after an example program (just read along till the end).
The macro for __SOCKADDR_COMMON is defined in the included bits/sockaddr.h file, which basically translates to an unsigned short int. This value defines the address family of the address (similar to the protocol family), and the sa_data buffer is saved for storing the actual address data (which in our example will be IP address and PORT).
* Why address family?
Since sockets can communicate using a variety of protocol families, each with its own way of defining endpoint addresses, the definition of an address must also be variable, depending on the address family. The possible address families are also defined in bits/socket.h; they usually translate directly to the corresponding protocol families.
Since an address can contain different types of information depending on the address family, there are several other socket address structure like the above struct sockaddr that is made especially for a particular protocol. For example, an IPv4's socket address structure will contain member for storing IP address and PORT number instead of just char sa_data[14].
These structures are also the same size, so they can be typecast to and from each other. This means that a socket() function will simply accept a pointer to a sockaddr structure, which in fact point to an address structure for IPv4, IPv6, or X.25. This allows the socket functions to operate on a variety of protocols.
In this article we are going to deal with Internet Protocol version 4, which is the protocol family PF_INET, using the address family AF_INET. The parallel socket address structure for AF_INET is defined in the netinet/in.h file (shown below).
The __SOCKADDR_COMMON part at the top of the structure is simply the unsigned short int mentioned above, which is used to define the address family.
Since an IPv4's socket endpoint address consists of an Internet address and a port number, these are the next two values in the structure. The port number is a 16-bit short, while the in_addr structure used for the Internet Address contains a 32-bit number.
The rest of the structure is just 8-byte of padding to fill out the rest of the sockaddr structure. This space isn't used for anything but must be saved so the structures can be interchangeably typecast.
In the end, the socket address structure ends up looking like this:
---xxxxxx---
* Network Byte Order
Do you know the endianness concept of computing?
First try to grab some idea on endianness (link to endianness on your favorite sites: Wikipedia, geeksforgeeks, google-it). You don't need a master's degree in endianness but at least learn the basic concept.
The port number and IP address used in the AF_INET socket address structure are expected to follow the network byte ordering, which is big-endian. This is the opposite of x86's little-endian byte ordering, so these values must be converted.
There are several functions specifically for these conversions, whose prototypes are defined in the netinet/in.h and arpa/inet.h include files. Here is a summary of these common byte order conversion functions:
htonl(long value) Host-to-Network Long
Converts a 32-bit integer from the host's byte order to network byte order.
htons(short value) Host-to-Network Short
Converts a 16-bit integer from the host's byte order to network byte order.
ntohl(long value) Network-to-Host Long
Converts a 32-bit integer from network byte order to host's byte order.
ntohs(long value) Network-to-Host Short
Converts a 16-bit integer from network byte order to the host's byte order.
For compatibility with all architecture, these conversion functions should still be used even if the host is using a processor with big-endian byte ordering.
---xxxxxx---
* Internet Address Conversion
When you see 12.110.110.203, you probably recognize this as an Internet address(IP version 4). This familiar dotted-number notation is a common way to specify Internet addresses, and there are functions to convert this notation(which is a string) to and from a 32-bit integer in network byte order. These functions are defined in the arpa/inet.h include file, and the two most useful conversion functions are:
inet_aton(char *ascii_addr, struct in_addr *network_addr)
ASCII to Network
This function converts an ASCII string containing an IP address in dotted-number format into an in_addr structure, which, as you remember, only contains a 32-bit integer representing the IP address in network byte order.
inet_ntoa(struct in_addr *network_addr)
Network to ASCII
This function converts the other way. It is passed a pointer to an in_addr structure containing an IP address, and the function returns a character pointer to an ASCII string containing the IP address in dotted-number format. This string is held in a statically allocated memory buffer in the function, so it can be accessed until the next call to inet_ntoa(), when the string will be overwritten.
---xxxxxx---
Enough of basics now let's dive directly into actually networking an application. As I already mentioned we will be using C programming language for our program.
* A Simple Server Example
The best way to show how and when the above functions are used is by an example. The following server code listens for TCP connections on port 7890. When a client connects, it sends the message "Hello, world!" and then receives data from the client until the connection is closed.
This is done using socket functions and structures the include files mentioned earlier, so these files are included at the beginning of the program.
The server program will be explained as you read the source code.
So far, the program sets up a socket using the socket() function. We want a TCP/IP socket, so the protocol family is PF_INET for IPv4 and the socket type is SOCK_STREAM for a stream socket. The final protocol argument is 0 since there is only one protocol in the PF_INET protocol family. This function returns a socket file descriptor which is stored in sockfd.
The setsockopt() function is simply used to set socket options. This function call sets the SO_REUSEADDR socket option to true, which will allow it to reuse a given address for binding.
Without this option set, when the program tries to bind to a given port, it will fail if that port is already in use. If a socket isn't closed properly, it may appear to be in use, so this option lets a socket bind to a port (and take over control of it), even if it seems to be in use.
The first argument to this function is the socket (referenced by a file descriptor), the second specifies the level of the option, and the third specifies the option itself. Since SO_REUSEADDR is a socket-level option, the level is set to SOL_SOCKET. (There are many socket options defined in /usr/include/asm/socket.h). The final two arguments are a pointer to the data that option should be set to and the length of that data.
A pointer to data and the length of that data are two arguments that are often used with socket functions. This allows the functions to handle all sorts of data, from single bytes to large data structures. The SO_REUSEADDR option uses a 32-bit integer for its value, so to set this option to true, the final two arguments must be a pointer to the integer value of 1 and the size of an integer (which is 4 bytes).
These next few lines set up the host_addr structure for use in the bind call. The address family is AF_INET, since we are using IPv4 and the sockaddr_instructure. The port is set to PORT, which is defined as 7890. This short integer value must be converted into network byte order, so the htons() function is used. The address is set to 0, which means it will automatically be filled with the host’s current IP address. Since the value 0 is the same regardless of byte order, no conversion is necessary.
The bind() call passes the socket file descriptor, the address structure, and the length of the address structure. This call will bind the socket to the current IP address on port 7890.
The listen() call tells the socket to listen for incoming connections, and a subsequent accept() call actually accepts an incoming connection. The listen() function places all incoming connections into a backlog queue until an accept() call accepts the connections. The last argument to the listen() call sets the maximum size for the backlog queue.
After getting a connection, the program prints out a connection message, using inet_ntoa() to convert the sin_addr address structure to a dotted-number IP string and ntohs() to convert the byte order of the sin_port number.
The send() function sends the 13 bytes of the string Hello, world!\n to the new socket that describes the new connection. The final argument for the send() and recv() functions are flags, that for our purposes, will always be 0.
Next is a loop that receives data from the connection and prints it out. The recv() function is given a pointer to a buffer and a maximum length to read from the socket. The function writes the data into the buffer passed to it and returns the number of bytes it actually wrote. The loop will continue as long as the recv() call continues to receive data.
Ok now that's it, let's compile and run our server program. Our program will bind to port 7890 of the host and waits for incoming connections.
A telnet client basically works like a generic TCP connection client, so it can be used to connect to the simple server by specifying the target IP address and port.
Upon connection, the server sends the string Hello, world!, and the rest is the local character echo of me typing this is a test and a line of keyboard mashing. Since telnet is line-buffered, each of these two lines is sent back to the server when ENTER is pressed.
Don't worry the end is near.
Even with all the above information, it's quite hard to relate a practical application to our server. The example below will help you relate those two.
Let's go back to our web browsing example once again:
HTTP exists in the application layer—the top layer—of the OSI model. At this layer, all of the networking details have already been taken care of by the lower layers, so HTTP uses plaintext for its structure. Many other application layer protocols also use plaintext, such as POP3, SMTP, IMAP, and FTP’s control channel. Since these are standard protocols, they are all well documented and easily researched. Once you know the syntax of these various protocols, you can manually talk to other programs that speak the same language. There’s no need to be fluent, but knowing a few important phrases will help you when traveling to foreign servers.
Going back to our example: Let's say you want to go to www.meandmyqwerty.com.
Note: the information below is simplified for your understanding.
- Your browser will first establish a TCP connection with www.meandmyqwerty.com
- Then your browser will send a plain-text request will look something like this "GET / HTTP/1.0"
- The above request will get the root document from the webserver using HTTP version 1.0. The request is actually for the root directory of /, but most webservers will automatically search for a default HTML document in that directory of index.html
- If the server finds the resource, it will respond using HTTP by sending several headers before sending the content.
If the command HEAD is used instead of GET, it will only return the HTTP headers without the content. These headers are plaintext and can usually provide information about the server. These headers can be retrieved manually using telnet by connecting to port 80 of a known website, then typing HEAD / HTTP/1.0 and pressing ENTER twice. In the output below, telnet is used to open a TCP-IP connection to the webserver at http://www.internic.net. Then the HTTP application layer is manually spoken to request the headers for the main index page.
---xxxxxx---
Congratulation! You made it till the end.
Hey...hey... calm down don't apply for networking jobs yet. You need some more of my articles :)
Check out more informational articles or some programming memes to get at least a moment of peace from the errors and bugs of life. CLICK_HERE
The goal of this type of article is to share useful information that helped me. So, the above article is taken out from books and other sources that helped me understood the concept.
Why don't you grab some good books on networking?
Reading books will really help you understand the topic more clearly.
As I am a human I make mistakes so please let me know if any.