[ACCEPTED]-How to write a scalable TCP/IP based server-scalability

I've written something similar to this in 97 the past. From my research years ago showed 96 that writing your own socket implementation 95 was the best bet, using the asynchronous sockets. This 94 meant that clients not really doing anything 93 actually required relatively few resources. Anything 92 that does occur is handled by the .NET thread 91 pool.

I wrote it as a class that manages 90 all connections for the servers.

I simply 89 used a list to hold all the client connections, but 88 if you need faster lookups for larger lists, you 87 can write it however you want.

private List<xConnection> _sockets;

Also you need 86 the socket actually listening for incoming 85 connections.

private System.Net.Sockets.Socket _serverSocket;

The start method actually starts 84 the server socket and begins listening for 83 any incoming connections.

public bool Start()
{
  System.Net.IPHostEntry localhost = System.Net.Dns.GetHostEntry(System.Net.Dns.GetHostName());
  System.Net.IPEndPoint serverEndPoint;
  try
  {
     serverEndPoint = new System.Net.IPEndPoint(localhost.AddressList[0], _port);
  }
  catch (System.ArgumentOutOfRangeException e)
  {
    throw new ArgumentOutOfRangeException("Port number entered would seem to be invalid, should be between 1024 and 65000", e);
  }
  try
  {
    _serverSocket = new System.Net.Sockets.Socket(serverEndPoint.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
   }
   catch (System.Net.Sockets.SocketException e)
   {
      throw new ApplicationException("Could not create socket, check to make sure not duplicating port", e);
    }
    try
    {
      _serverSocket.Bind(serverEndPoint);
      _serverSocket.Listen(_backlog);
    }
    catch (Exception e)
    {
       throw new ApplicationException("An error occurred while binding socket. Check inner exception", e);
    }
    try
    {
       //warning, only call this once, this is a bug in .net 2.0 that breaks if
       // you're running multiple asynch accepts, this bug may be fixed, but
       // it was a major pain in the rear previously, so make sure there is only one
       //BeginAccept running
       _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
    }
    catch (Exception e)
    {
       throw new ApplicationException("An error occurred starting listeners. Check inner exception", e);
    }
    return true;
 }

I'd just like to 82 note the exception handling code looks bad, but 81 the reason for it is I had exception suppression 80 code in there so that any exceptions would 79 be suppressed and return false if a configuration 78 option was set, but I wanted to remove it 77 for brevity sake.

The _serverSocket.BeginAccept(new 76 AsyncCallback(acceptCallback)), _serverSocket) above 75 essentially sets our server socket to call 74 the acceptCallback method whenever a user 73 connects. This method runs from the .NET 72 threadpool, which automatically handles 71 creating additional worker threads if you 70 have many blocking operations. This should 69 optimally handle any load on the server.

    private void acceptCallback(IAsyncResult result)
    {
       xConnection conn = new xConnection();
       try
       {
         //Finish accepting the connection
         System.Net.Sockets.Socket s = (System.Net.Sockets.Socket)result.AsyncState;
         conn = new xConnection();
         conn.socket = s.EndAccept(result);
         conn.buffer = new byte[_bufferSize];
         lock (_sockets)
         {
           _sockets.Add(conn);
         }
         //Queue receiving of data from the connection
         conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
         //Queue the accept of the next incoming connection
         _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
       }
       catch (SocketException e)
       {
         if (conn.socket != null)
         {
           conn.socket.Close();
           lock (_sockets)
           {
             _sockets.Remove(conn);
           }
         }
         //Queue the next accept, think this should be here, stop attacks based on killing the waiting listeners
         _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
       }
       catch (Exception e)
       {
         if (conn.socket != null)
         {
           conn.socket.Close();
           lock (_sockets)
           {
             _sockets.Remove(conn);
           }
         }
         //Queue the next accept, think this should be here, stop attacks based on killing the waiting listeners
         _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
       }
     }

The 68 above code essentially just finished accepting 67 the connection that comes in, queues BeginReceive which 66 is a callback that will run when the client 65 sends data, and then queues the next acceptCallback which 64 will accept the next client connection that 63 comes in.

The BeginReceive method call is what tells 62 the socket what to do when it receives data 61 from the client. For BeginReceive, you need to give 60 it a byte array, which is where it will 59 copy the data when the client sends data. The 58 ReceiveCallback method will get called, which is how we 57 handle receiving data.

private void ReceiveCallback(IAsyncResult result)
{
  //get our connection from the callback
  xConnection conn = (xConnection)result.AsyncState;
  //catch any errors, we'd better not have any
  try
  {
    //Grab our buffer and count the number of bytes receives
    int bytesRead = conn.socket.EndReceive(result);
    //make sure we've read something, if we haven't it supposadly means that the client disconnected
    if (bytesRead > 0)
    {
      //put whatever you want to do when you receive data here

      //Queue the next receive
      conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
     }
     else
     {
       //Callback run but no data, close the connection
       //supposadly means a disconnect
       //and we still have to close the socket, even though we throw the event later
       conn.socket.Close();
       lock (_sockets)
       {
         _sockets.Remove(conn);
       }
     }
   }
   catch (SocketException e)
   {
     //Something went terribly wrong
     //which shouldn't have happened
     if (conn.socket != null)
     {
       conn.socket.Close();
       lock (_sockets)
       {
         _sockets.Remove(conn);
       }
     }
   }
 }

EDIT: In this pattern 56 I forgot to mention that in this area of 55 code:

//put whatever you want to do when you receive data here

//Queue the next receive
conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);

Generally, in the whatever you want 54 code, I would do reassembly of packets into 53 messages, and then create them as jobs on 52 the thread pool. This way the BeginReceive 51 of the next block from the client isn't 50 delayed while whatever message processing 49 code is running.

The accept callback finishes 48 reading the data socket by calling end receive. This 47 fills the buffer provided in the begin receive 46 function. Once you do whatever you want 45 where I left the comment, we call the next 44 BeginReceive method which will run the callback again 43 if the client sends any more data.

Now here's 42 the really tricky part: When the client 41 sends data, your receive callback might 40 only be called with part of the message. Reassembly 39 can become very very complicated. I used 38 my own method and created a sort of proprietary 37 protocol to do this. I left it out, but 36 if you request, I can add it in. This handler 35 was actually the most complicated piece 34 of code I had ever written.

public bool Send(byte[] message, xConnection conn)
{
  if (conn != null && conn.socket.Connected)
  {
    lock (conn.socket)
    {
    //we use a blocking mode send, no async on the outgoing
    //since this is primarily a multithreaded application, shouldn't cause problems to send in blocking mode
       conn.socket.Send(bytes, bytes.Length, SocketFlags.None);
     }
   }
   else
     return false;
   return true;
 }

The above send 33 method actually uses a synchronous Send call. For 32 me that was fine due to the message sizes 31 and the multithreaded nature of my application. If 30 you want to send to every client, you simply 29 need to loop through the _sockets List.

The 28 xConnection class you see referenced above 27 is basically a simple wrapper for a socket 26 to include the byte buffer, and in my implementation 25 some extras.

public class xConnection : xBase
{
  public byte[] buffer;
  public System.Net.Sockets.Socket socket;
}

Also for reference here are 24 the usings I include since I always get annoyed 23 when they aren't included.

using System.Net.Sockets;

I hope that's 22 helpful. It may not be the cleanest code, but 21 it works. There are also some nuances to 20 the code which you should be weary about 19 changing. For one, only have a single BeginAccept called 18 at any one time. There used to be a very 17 annoying .NET bug around this, which was 16 years ago so I don't recall the details.

Also, in 15 the ReceiveCallback code, we process anything received 14 from the socket before we queue the next 13 receive. This means that for a single socket, we're 12 only actually ever in ReceiveCallback once at any point 11 in time, and we don't need to use thread 10 synchronization. However, if you reorder 9 this to call the next receive immediately 8 after pulling the data, which might be a 7 little faster, you will need to make sure 6 you properly synchronize the threads.

Also, I 5 hacked out a lot of my code, but left the 4 essence of what's happening in place. This 3 should be a good start for you're design. Leave 2 a comment if you have any more questions 1 around this.

There are many ways of doing network operations 42 in C#. All of them use different mechanisms 41 under the hood, and thus suffer major performance 40 issues with a high concurrency. Begin* operations 39 are one of these that many people often 38 mistake for being the faster/fastest way 37 of doing networking.

To solve these issues, they 36 introduced the Async set of methods: From MSDN, SocketAsyncEventArgs Class -

The SocketAsyncEventArgs 35 class is part of a set of enhancements to 34 the System.Net.Sockets..::.Socket class 33 that provide an alternative asynchronous 32 pattern that can be used by specialized 31 high-performance socket applications. This 30 class was specifically designed for network 29 server applications that require high performance. An 28 application can use the enhanced asynchronous 27 pattern exclusively or only in targeted 26 hot areas (for example, when receiving large 25 amounts of data).

The main feature of these 24 enhancements is the avoidance of the repeated 23 allocation and synchronization of objects 22 during high-volume asynchronous socket I/O. The 21 Begin/End design pattern currently implemented 20 by the System.Net.Sockets..::.Socket class 19 requires a System..::.IAsyncResult object 18 be allocated for each asynchronous socket 17 operation.

Under the covers, the *Async API 16 uses I/O completion ports which is the fastest 15 way of performing networking operations, see 14 Windows Sockets 2.0: Write Scalable Winsock Apps Using Completion Ports

And just to help you out, I am including 13 the source code for a telnet server I wrote 12 using the *Async API. I am only including 11 the relevant portions. Also to note, instead 10 of processing the data inline, I instead 9 opt to push it onto a lock free (wait free) queue 8 that is processed on a separate thread. Note 7 that I am not including the corresponding 6 Pool class which is just a simple pool which 5 will create a new object if it is empty, and 4 the Buffer class which is just a self-expanding 3 buffer which is not really needed unless 2 you are receiving an indeterministic amount 1 of data.

public class Telnet
{
    private readonly Pool<SocketAsyncEventArgs> m_EventArgsPool;
    private Socket m_ListenSocket;

    /// <summary>
    /// This event fires when a connection has been established.
    /// </summary>
    public event EventHandler<SocketAsyncEventArgs> Connected;

    /// <summary>
    /// This event fires when a connection has been shutdown.
    /// </summary>
    public event EventHandler<SocketAsyncEventArgs> Disconnected;

    /// <summary>
    /// This event fires when data is received on the socket.
    /// </summary>
    public event EventHandler<SocketAsyncEventArgs> DataReceived;

    /// <summary>
    /// This event fires when data is finished sending on the socket.
    /// </summary>
    public event EventHandler<SocketAsyncEventArgs> DataSent;

    /// <summary>
    /// This event fires when a line has been received.
    /// </summary>
    public event EventHandler<LineReceivedEventArgs> LineReceived;

    /// <summary>
    /// Specifies the port to listen on.
    /// </summary>
    [DefaultValue(23)]
    public int ListenPort { get; set; }

    /// <summary>
    /// Constructor for Telnet class.
    /// </summary>
    public Telnet()
    {
        m_EventArgsPool = new Pool<SocketAsyncEventArgs>();
        ListenPort = 23;
    }

    /// <summary>
    /// Starts the telnet server listening and accepting data.
    /// </summary>
    public void Start()
    {
        IPEndPoint endpoint = new IPEndPoint(0, ListenPort);
        m_ListenSocket = new Socket(endpoint.AddressFamily, SocketType.Stream, ProtocolType.Tcp);

        m_ListenSocket.Bind(endpoint);
        m_ListenSocket.Listen(100);

        //
        // Post Accept
        //
        StartAccept(null);
    }

    /// <summary>
    /// Not Yet Implemented. Should shutdown all connections gracefully.
    /// </summary>
    public void Stop()
    {
        //throw (new NotImplementedException());
    }

    //
    // ACCEPT
    //

    /// <summary>
    /// Posts a requests for Accepting a connection. If it is being called from the completion of
    /// an AcceptAsync call, then the AcceptSocket is cleared since it will create a new one for
    /// the new user.
    /// </summary>
    /// <param name="e">null if posted from startup, otherwise a <b>SocketAsyncEventArgs</b> for reuse.</param>
    private void StartAccept(SocketAsyncEventArgs e)
    {
        if (e == null)
        {
            e = m_EventArgsPool.Pop();
            e.Completed += Accept_Completed;
        }
        else
        {
            e.AcceptSocket = null;
        }

        if (m_ListenSocket.AcceptAsync(e) == false)
        {
            Accept_Completed(this, e);
        }
    }

    /// <summary>
    /// Completion callback routine for the AcceptAsync post. This will verify that the Accept occured
    /// and then setup a Receive chain to begin receiving data.
    /// </summary>
    /// <param name="sender">object which posted the AcceptAsync</param>
    /// <param name="e">Information about the Accept call.</param>
    private void Accept_Completed(object sender, SocketAsyncEventArgs e)
    {
        //
        // Socket Options
        //
        e.AcceptSocket.NoDelay = true;

        //
        // Create and setup a new connection object for this user
        //
        Connection connection = new Connection(this, e.AcceptSocket);

        //
        // Tell the client that we will be echo'ing data sent
        //
        DisableEcho(connection);

        //
        // Post the first receive
        //
        SocketAsyncEventArgs args = m_EventArgsPool.Pop();
        args.UserToken = connection;

        //
        // Connect Event
        //
        if (Connected != null)
        {
            Connected(this, args);
        }

        args.Completed += Receive_Completed;
        PostReceive(args);

        //
        // Post another accept
        //
        StartAccept(e);
    }

    //
    // RECEIVE
    //

    /// <summary>
    /// Post an asynchronous receive on the socket.
    /// </summary>
    /// <param name="e">Used to store information about the Receive call.</param>
    private void PostReceive(SocketAsyncEventArgs e)
    {
        Connection connection = e.UserToken as Connection;

        if (connection != null)
        {
            connection.ReceiveBuffer.EnsureCapacity(64);
            e.SetBuffer(connection.ReceiveBuffer.DataBuffer, connection.ReceiveBuffer.Count, connection.ReceiveBuffer.Remaining);

            if (connection.Socket.ReceiveAsync(e) == false)
            {
                Receive_Completed(this, e);
            }
        }
    }

    /// <summary>
    /// Receive completion callback. Should verify the connection, and then notify any event listeners
    /// that data has been received. For now it is always expected that the data will be handled by the
    /// listeners and thus the buffer is cleared after every call.
    /// </summary>
    /// <param name="sender">object which posted the ReceiveAsync</param>
    /// <param name="e">Information about the Receive call.</param>
    private void Receive_Completed(object sender, SocketAsyncEventArgs e)
    {
        Connection connection = e.UserToken as Connection;

        if (e.BytesTransferred == 0 || e.SocketError != SocketError.Success || connection == null)
        {
            Disconnect(e);
            return;
        }

        connection.ReceiveBuffer.UpdateCount(e.BytesTransferred);

        OnDataReceived(e);

        HandleCommand(e);
        Echo(e);

        OnLineReceived(connection);

        PostReceive(e);
    }

    /// <summary>
    /// Handles Event of Data being Received.
    /// </summary>
    /// <param name="e">Information about the received data.</param>
    protected void OnDataReceived(SocketAsyncEventArgs e)
    {
        if (DataReceived != null)
        {
            DataReceived(this, e);
        }
    }

    /// <summary>
    /// Handles Event of a Line being Received.
    /// </summary>
    /// <param name="connection">User connection.</param>
    protected void OnLineReceived(Connection connection)
    {
        if (LineReceived != null)
        {
            int index = 0;
            int start = 0;

            while ((index = connection.ReceiveBuffer.IndexOf('\n', index)) != -1)
            {
                string s = connection.ReceiveBuffer.GetString(start, index - start - 1);
                s = s.Backspace();

                LineReceivedEventArgs args = new LineReceivedEventArgs(connection, s);
                Delegate[] delegates = LineReceived.GetInvocationList();

                foreach (Delegate d in delegates)
                {
                    d.DynamicInvoke(new object[] { this, args });

                    if (args.Handled == true)
                    {
                        break;
                    }
                }

                if (args.Handled == false)
                {
                    connection.CommandBuffer.Enqueue(s);
                }

                start = index;
                index++;
            }

            if (start > 0)
            {
                connection.ReceiveBuffer.Reset(0, start + 1);
            }
        }
    }

    //
    // SEND
    //

    /// <summary>
    /// Overloaded. Sends a string over the telnet socket.
    /// </summary>
    /// <param name="connection">Connection to send data on.</param>
    /// <param name="s">Data to send.</param>
    /// <returns>true if the data was sent successfully.</returns>
    public bool Send(Connection connection, string s)
    {
        if (String.IsNullOrEmpty(s) == false)
        {
            return Send(connection, Encoding.Default.GetBytes(s));
        }

        return false;
    }

    /// <summary>
    /// Overloaded. Sends an array of data to the client.
    /// </summary>
    /// <param name="connection">Connection to send data on.</param>
    /// <param name="data">Data to send.</param>
    /// <returns>true if the data was sent successfully.</returns>
    public bool Send(Connection connection, byte[] data)
    {
        return Send(connection, data, 0, data.Length);
    }

    public bool Send(Connection connection, char c)
    {
        return Send(connection, new byte[] { (byte)c }, 0, 1);
    }

    /// <summary>
    /// Sends an array of data to the client.
    /// </summary>
    /// <param name="connection">Connection to send data on.</param>
    /// <param name="data">Data to send.</param>
    /// <param name="offset">Starting offset of date in the buffer.</param>
    /// <param name="length">Amount of data in bytes to send.</param>
    /// <returns></returns>
    public bool Send(Connection connection, byte[] data, int offset, int length)
    {
        bool status = true;

        if (connection.Socket == null || connection.Socket.Connected == false)
        {
            return false;
        }

        SocketAsyncEventArgs args = m_EventArgsPool.Pop();
        args.UserToken = connection;
        args.Completed += Send_Completed;
        args.SetBuffer(data, offset, length);

        try
        {
            if (connection.Socket.SendAsync(args) == false)
            {
                Send_Completed(this, args);
            }
        }
        catch (ObjectDisposedException)
        {
            //
            // return the SocketAsyncEventArgs back to the pool and return as the
            // socket has been shutdown and disposed of
            //
            m_EventArgsPool.Push(args);
            status = false;
        }

        return status;
    }

    /// <summary>
    /// Sends a command telling the client that the server WILL echo data.
    /// </summary>
    /// <param name="connection">Connection to disable echo on.</param>
    public void DisableEcho(Connection connection)
    {
        byte[] b = new byte[] { 255, 251, 1 };
        Send(connection, b);
    }

    /// <summary>
    /// Completion callback for SendAsync.
    /// </summary>
    /// <param name="sender">object which initiated the SendAsync</param>
    /// <param name="e">Information about the SendAsync call.</param>
    private void Send_Completed(object sender, SocketAsyncEventArgs e)
    {
        e.Completed -= Send_Completed;
        m_EventArgsPool.Push(e);
    }

    /// <summary>
    /// Handles a Telnet command.
    /// </summary>
    /// <param name="e">Information about the data received.</param>
    private void HandleCommand(SocketAsyncEventArgs e)
    {
        Connection c = e.UserToken as Connection;

        if (c == null || e.BytesTransferred < 3)
        {
            return;
        }

        for (int i = 0; i < e.BytesTransferred; i += 3)
        {
            if (e.BytesTransferred - i < 3)
            {
                break;
            }

            if (e.Buffer[i] == (int)TelnetCommand.IAC)
            {
                TelnetCommand command = (TelnetCommand)e.Buffer[i + 1];
                TelnetOption option = (TelnetOption)e.Buffer[i + 2];

                switch (command)
                {
                    case TelnetCommand.DO:
                        if (option == TelnetOption.Echo)
                        {
                            // ECHO
                        }
                        break;
                    case TelnetCommand.WILL:
                        if (option == TelnetOption.Echo)
                        {
                            // ECHO
                        }
                        break;
                }

                c.ReceiveBuffer.Remove(i, 3);
            }
        }
    }

    /// <summary>
    /// Echoes data back to the client.
    /// </summary>
    /// <param name="e">Information about the received data to be echoed.</param>
    private void Echo(SocketAsyncEventArgs e)
    {
        Connection connection = e.UserToken as Connection;

        if (connection == null)
        {
            return;
        }

        //
        // backspacing would cause the cursor to proceed beyond the beginning of the input line
        // so prevent this
        //
        string bs = connection.ReceiveBuffer.ToString();

        if (bs.CountAfterBackspace() < 0)
        {
            return;
        }

        //
        // find the starting offset (first non-backspace character)
        //
        int i = 0;

        for (i = 0; i < connection.ReceiveBuffer.Count; i++)
        {
            if (connection.ReceiveBuffer[i] != '\b')
            {
                break;
            }
        }

        string s = Encoding.Default.GetString(e.Buffer, Math.Max(e.Offset, i), e.BytesTransferred);

        if (connection.Secure)
        {
            s = s.ReplaceNot("\r\n\b".ToCharArray(), '*');
        }

        s = s.Replace("\b", "\b \b");

        Send(connection, s);
    }

    //
    // DISCONNECT
    //

    /// <summary>
    /// Disconnects a socket.
    /// </summary>
    /// <remarks>
    /// It is expected that this disconnect is always posted by a failed receive call. Calling the public
    /// version of this method will cause the next posted receive to fail and this will cleanup properly.
    /// It is not advised to call this method directly.
    /// </remarks>
    /// <param name="e">Information about the socket to be disconnected.</param>
    private void Disconnect(SocketAsyncEventArgs e)
    {
        Connection connection = e.UserToken as Connection;

        if (connection == null)
        {
            throw (new ArgumentNullException("e.UserToken"));
        }

        try
        {
            connection.Socket.Shutdown(SocketShutdown.Both);
        }
        catch
        {
        }

        connection.Socket.Close();

        if (Disconnected != null)
        {
            Disconnected(this, e);
        }

        e.Completed -= Receive_Completed;
        m_EventArgsPool.Push(e);
    }

    /// <summary>
    /// Marks a specific connection for graceful shutdown. The next receive or send to be posted
    /// will fail and close the connection.
    /// </summary>
    /// <param name="connection"></param>
    public void Disconnect(Connection connection)
    {
        try
        {
            connection.Socket.Shutdown(SocketShutdown.Both);
        }
        catch (Exception)
        {
        }
    }

    /// <summary>
    /// Telnet command codes.
    /// </summary>
    internal enum TelnetCommand
    {
        SE = 240,
        NOP = 241,
        DM = 242,
        BRK = 243,
        IP = 244,
        AO = 245,
        AYT = 246,
        EC = 247,
        EL = 248,
        GA = 249,
        SB = 250,
        WILL = 251,
        WONT = 252,
        DO = 253,
        DONT = 254,
        IAC = 255
    }

    /// <summary>
    /// Telnet command options.
    /// </summary>
    internal enum TelnetOption
    {
        Echo = 1,
        SuppressGoAhead = 3,
        Status = 5,
        TimingMark = 6,
        TerminalType = 24,
        WindowSize = 31,
        TerminalSpeed = 32,
        RemoteFlowControl = 33,
        LineMode = 34,
        EnvironmentVariables = 36
    }
}

There used to be a really good discussion 109 of scalable TCP/IP using .NET written by 108 Chris Mullins of Coversant. Unfortunately, it 107 appears his blog has disappeared from its 106 prior location, so I will try to piece together 105 his advice from memory (some useful comments 104 of his appear in this thread: C++ vs. C#: Developing a highly scalable IOCP server)

First and 103 foremost, note that both using Begin/End and the 102 Async methods on the Socket class make use of I/O completion ports (IOCP) to 101 provide scalability. This makes a much bigger 100 difference (when used correctly; see below) to 99 scalability than which of the two methods 98 you actually pick to implement your solution.

Chris 97 Mullins' posts were based on using Begin/End, which 96 is the one I personally have experience 95 with. Note that Chris put together a solution 94 based on this that scaled up to 10,000s 93 of concurrent client connections on a 32-bit 92 machine with 2 GB of memory, and well into 91 100,000s on a 64-bit platform with sufficient 90 memory. From my own experience with this 89 technique (although nowhere near this kind 88 of load) I have no reason to doubt these 87 indicative figures.

IOCP versus thread-per-connection or 'select' primitives

The reason you want to 86 use a mechanism that uses IOCP under the 85 hood is that it uses a very low-level Windows 84 thread pool that does not wake up any threads 83 until there is actual data on the I/O channel 82 that you are trying to read from (note that 81 IOCP can be used for file I/O as well). The 80 benefit of this is that Windows does not 79 have to switch to a thread only to find 78 that there is no data yet anyway, so this 77 reduces the number of context switches your 76 server will have to make to the bare minimum 75 required.

Context switches is what will definitely 74 kill the 'thread-per-connection' mechanism, although 73 this is a viable solution if you are only 72 dealing with a few dozen connections. This 71 mechanism is however by no stretch of the 70 imagination 'scalable'.

Important considerations when using IOCP

Memory

First and foremost 69 it is critical to understand that IOCP can 68 easily result in memory issues under .NET 67 if your implementation is too naive. Every 66 IOCP BeginReceive call will result in "pinning" of 65 the buffer you are reading into. For a good 64 explanation of why this is a problem, see: Yun Jin's Weblog: OutOfMemoryException and Pinning.

Luckily 63 this problem can be avoided, but it requires 62 a bit of a trade-off. The suggested solution 61 is to allocate a big byte[] buffer at application 60 start-up (or close thereto), of at least 59 90 KB or-so (as of .NET 2, required size 58 may be larger in later versions). The reason 57 to do this is that large memory allocations 56 automatically end up in a non-compacting 55 memory segment (the large object heap) that is effectively 54 automatically pinned. By allocating one 53 large buffer at start-up you make sure that 52 this block of unmovable memory is at a relatively 51 'low address' where it will not get in the 50 way and cause fragmentation.

You then can 49 use offsets to segment this one big buffer 48 into separate areas for each connection 47 that needs to read some data. This is where 46 a trade-off comes into play; since this 45 buffer needs to be pre-allocated, you will 44 have to decide how much buffer space you 43 need per connection, and what upper limit 42 you want to set on the number of connections 41 you want to scale to (or, you can implement 40 an abstraction that can allocate additional 39 pinned buffers once you need them).

The simplest 38 solution would be to assign every connection 37 a single byte at a unique offset within 36 this buffer. Then you can make a BeginReceive call for 35 a single byte to be read, and perform the 34 rest of the reading as a result of the callback 33 you get.

Processing

When you get the callback from the 32 Begin call you made, it is very important to 31 realise that the code in the callback will 30 execute on the low-level IOCP thread. It 29 is absolutely essential that you avoid lengthy operations 28 in this callback. Using these threads for 27 complex processing will kill your scalability 26 just as effectively as using 'thread-per-connection'.

The 25 suggested solution is to use the callback 24 only to queue up a work item to process 23 the incoming data, that will be executed 22 on some other thread. Avoid any potentially 21 blocking operations inside the callback 20 so that the IOCP thread can return to its 19 pool as quickly as possible. In .NET 4.0 18 I'd suggest the easiest solution is to spawn 17 a Task, giving it a reference to the client 16 socket and a copy of the first byte that 15 was already read by the BeginReceive call. This task 14 is then responsible for reading all data 13 from the socket that represent the request 12 you are processing, executing it, and then 11 making a new BeginReceive call to queue the socket for 10 IOCP once more. Pre .NET 4.0, you can use 9 the ThreadPool, or create your own threaded 8 work-queue implementation.

Summary

Basically, I'd 7 suggest using Kevin's sample code for this solution, with the 6 following added warnings:

• Make sure the buffer you pass to BeginReceive is already 'pinned'
• Make sure the callback you pass to BeginReceive does nothing more than queue up a task to handle the actual processing of the incoming data

When you do that, I 5 have no doubt you could replicate Chris' results 4 in scaling up to potentially hundreds of 3 thousands of simultaneous clients (given 2 the right hardware and an efficient implementation 1 of your own processing code of course ;)

You already got the most part of the answer 112 via the code samples above. Using asynchronous 111 I/O operations is absolutely the way to 110 go here. Async I/O is the way the Win32 109 is designed internally to scale. The best 108 possible performance you can get is achieved 107 using completion ports, binding your sockets to completion 106 ports and have a thread pool waiting for 105 completion port completion. The common wisdom 104 is to have 2-4 threads per CPU (core) waiting 103 for completion. I highly recommend to go 102 over these three articles by Rick Vicik 101 from the Windows Performance team:

1. Designing Applications for Performance - Part 1
2. Designing Applications for Performance - Part 2
3. Designing Applications for Performance - Part 3

The said 100 articles cover mostly the native Windows 99 API, but they are a must-read for anyone 98 trying to get a grasp at scalability and 97 performance. They do have some briefs on 96 the managed side of things too.

The second 95 thing you'll need to do is make sure you 94 go over the Improving .NET Application Performance and Scalability book, that is available online. You 93 will find pertinent and valid advice around 92 the use of threads, asynchronous calls and 91 locks in Chapter 5. But the real gems are 90 in Chapter 17 where you'll find such goodies 89 as practical guidance on tuning your thread 88 pool. My applications had some serious problems 87 until I adjusted the maxIothreads/maxWorkerThreads 86 as per the recommendations in this chapter.

You 85 say that you want to do a pure TCP server, so 84 my next point is spurious. However, if you find 83 yourself cornered and use the WebRequest 82 class and its derivatives, be warned that 81 there is a dragon guarding that door: the 80 ServicePointManager. This is a configuration class that has 79 one purpose in life: to ruin your performance. Make 78 sure you free your server from the artificial 77 imposed ServicePoint.ConnectionLimit or 76 your application will never scale (I let 75 you discover yourself what the default value 74 is...). You may also reconsider the default 73 policy of sending an Expect100Continue header 72 in the HTTP requests.

Now about the core 71 socket managed API, things are fairly easy 70 on the Send side, but they are significantly 69 more complex on the Receive side. In order 68 to achieve high throughput and scale, you 67 must ensure that the socket is not flow 66 controlled, because you do not have a buffer 65 posted for receive. Ideally for high performance 64 you should post ahead 3-4 buffers and post 63 new buffers as soon as you get one back 62 (before you process the one got back), so you 61 ensure that the socket always has somewhere 60 to deposit the data coming from the network. You'll 59 see why you probably won't be able to achieve 58 this shortly.

After you're done playing with 57 the BeginRead/BeginWrite API and start the 56 serious work, you'll realize that you need 55 security on your traffic, i.e., NTLM/Kerberos 54 authentication and traffic encryption, or 53 at least traffic tampering protection. The 52 way you do this is you use the built in 51 System.Net.Security.NegotiateStream (or 50 SslStream if you need to go cross disparate 49 domains). This means that instead of relying 48 on straight socket asynchronous operations 47 you will rely on the AuthenticatedStream 46 asynchronous operations. As soon as you 45 obtain a socket (either from connect on 44 client or from accept on server) you create 43 a stream on the socket and submit it for 42 authentication, by calling either BeginAuthenticateAsClient 41 or BeginAuthenticateAsServer. After the 40 authentication completes (at least your 39 safe from the native InitiateSecurityContext/AcceptSecurityContext 38 madness...) you will do your authorization 37 by checking the RemoteIdentity property 36 of your Authenticated stream and doing whatever 35 ACL verification your product must support.

After 34 that you will send messages using the BeginWrite 33 and you'll be receiving them with BeginRead. This 32 is the problem I was talking before that 31 you won't be able to post multiple receive 30 buffers, because the AuthenticateStream 29 classes don't support this. The BeginRead 28 operation manages all the I/O internally 27 until you have received an entire frame. Otherwise, it 26 could not handle the message authentication 25 (decrypt frame and validate signature on 24 frame). Though in my experience the job 23 done by the AuthenticatedStream classes 22 is fairly good and shouldn't have any problem 21 with it. I.e., you should be able to saturate 20 a 1 Gbit/s network with only 4-5% CPU. The 19 AuthenticatedStream classes will also impose 18 the protocol-specific frame size limitations 17 on you (16k for SSL, 12k for Kerberos).

This 16 should get you started on the right track. I'm 15 not going to post code here, and there is 14 a perfectly good example on MSDN. I've done many projects like this and 13 I was able to scale to about 1000 users 12 connected without problems. Above that, you'll 11 need to modify registry keys to allow the 10 kernel for more socket handles. And make 9 sure you deploy on a server OS, that is, Windows Server 2003, not 8 Windows XP or Windows Vista (i.e., client OS), it makes 7 a big difference.

BTW, make sure, if you 6 have databases operations on the server 5 or file I/O, you also use the async flavor 4 for them, or you'll drain the thread pool 3 in no time. For SQL Server connections, make 2 sure you add the 'Asyncronous Processing=true' to 1 the connection string.

I've got such a server running in some of 26 my solutions. Here is a very detailed explanation 25 of the different ways to do it in .NET: Get Closer to the Wire with High-Performance Sockets in .NET

Lately 24 I've been looking for ways to improve our 23 code and will be looking into this: "Socket Performance Enhancements in Version 3.5" that 22 was included specifically "for use 21 by applications that use asynchronous network 20 I/O to achieve the highest performance".

"The 19 main feature of these enhancements is the 18 avoidance of the repeated allocation and 17 synchronization of objects during high-volume 16 asynchronous socket I/O. The Begin/End design 15 pattern currently implemented by the Socket 14 class for asynchronous socket I/O requires 13 a System.IAsyncResult object be allocated 12 for each asynchronous socket operation."

You 11 can keep reading if you follow the link. I 10 personally will be testing their sample 9 code tomorrow to benchmark it against what 8 I've got.

Here you can find working code for 7 both client and server using the new 3.5 6 SocketAsyncEventArgs so you can test it 5 within a couple minutes and go through the 4 code. It is a simple approach, but it is 3 the basis for starting a much larger implementation. Also 2 this article from almost two years ago in MSDN 1 Magazine was a interesting read.

Consider just using a WCF net TCP binding and 6 a publish/subscribe pattern. WCF would allow 5 you to focus (mostly) on your domain instead 4 of plumbing...

There are lots of WCF samples 3 and even a publish/subscribe framework available 2 on IDesign's download section which may 1 be useful: http://www.idesign.net

I am wondering about one thing:

I definitely 25 do not want to start a thread for each 24 connection.

Why is that? Windows could handle 23 hundreds of threads in an application since 22 at least Windows 2000. I've done it, it's 21 really easy to work with if the threads 20 don't need to be synchronized. Especially 19 given that you're doing a lot of I/O (so 18 you're not CPU-bound, and a lot of threads 17 would be blocked on either disk or network 16 communication), I don't understand this 15 restriction.

Have you tested the multi-threaded 14 way and found it lacking in something? Do 13 you intend to also have a database connection 12 for each thread (that would kill the database 11 server, so it's a bad idea, but it's easily 10 solved with a 3-tier design). Are you worried 9 that you'll have thousands of clients instead 8 of hundreds, and then you'll really have 7 problems? (Though I'd try a thousand threads 6 or even ten thousand if I had 32+ GB of 5 RAM - again, given that you're not CPU bound, thread 4 switch time should be absolutely irrelevant.)

Here 3 is the code - to see how this looks running, go 2 to http://mdpopescu.blogspot.com/2009/05/multi-threaded-server.html and click on the picture.

Server class:

  public class Server
  {
    private static readonly TcpListener listener = new TcpListener(IPAddress.Any, 9999);

    public Server()
    {
      listener.Start();
      Console.WriteLine("Started.");

      while (true)
      {
        Console.WriteLine("Waiting for connection...");

        var client = listener.AcceptTcpClient();
        Console.WriteLine("Connected!");

        // each connection has its own thread
        new Thread(ServeData).Start(client);
      }
    }

    private static void ServeData(object clientSocket)
    {
      Console.WriteLine("Started thread " + Thread.CurrentThread.ManagedThreadId);

      var rnd = new Random();
      try
      {
        var client = (TcpClient) clientSocket;
        var stream = client.GetStream();
        while (true)
        {
          if (rnd.NextDouble() < 0.1)
          {
            var msg = Encoding.ASCII.GetBytes("Status update from thread " + Thread.CurrentThread.ManagedThreadId);
            stream.Write(msg, 0, msg.Length);

            Console.WriteLine("Status update from thread " + Thread.CurrentThread.ManagedThreadId);
          }

          // wait until the next update - I made the wait time so small 'cause I was bored :)
          Thread.Sleep(new TimeSpan(0, 0, rnd.Next(1, 5)));
        }
      }
      catch (SocketException e)
      {
        Console.WriteLine("Socket exception in thread {0}: {1}", Thread.CurrentThread.ManagedThreadId, e);
      }
    }
  }

Server 1 main program:

namespace ManyThreadsServer
{
  internal class Program
  {
    private static void Main(string[] args)
    {
      new Server();
    }
  }
}

Client class:

  public class Client
  {
    public Client()
    {
      var client = new TcpClient();
      client.Connect(IPAddress.Loopback, 9999);

      var msg = new byte[1024];

      var stream = client.GetStream();
      try
      {
        while (true)
        {
          int i;
          while ((i = stream.Read(msg, 0, msg.Length)) != 0)
          {
            var data = Encoding.ASCII.GetString(msg, 0, i);
            Console.WriteLine("Received: {0}", data);
          }
        }
      }
      catch (SocketException e)
      {
        Console.WriteLine("Socket exception in thread {0}: {1}", Thread.CurrentThread.ManagedThreadId, e);
      }
    }
  }

Client main program:

using System;
using System.Threading;

namespace ManyThreadsClient
{
  internal class Program
  {
    private static void Main(string[] args)
    {
      // first argument is the number of threads
      for (var i = 0; i < Int32.Parse(args[0]); i++)
        new Thread(RunClient).Start();
    }

    private static void RunClient()
    {
      new Client();
    }
  }
}

Using .NET's integrated Async I/O (BeginRead, etc.) is 51 a good idea if you can get all the details 50 right. When you properly set up your socket/file 49 handles it will use the OS's underlying 48 IOCP implementation, allowing your operations 47 to complete without using any threads (or, in 46 the worst case, using a thread that I believe 45 comes from the kernel's I/O thread pool 44 instead of .NET's thread pool, which helps 43 alleviate threadpool congestion.)

The main 42 gotcha is to make sure that you open your 41 sockets/files in non-blocking mode. Most 40 of the default convenience functions (like 39 File.OpenRead) don't do this, so you'll need to write 38 your own.

One of the other main concerns 37 is error handling - properly handling errors 36 when writing asynchronous I/O code is much, much 35 harder than doing it in synchronous code. It's 34 also very easy to end up with race conditions 33 and deadlocks even though you may not be 32 using threads directly, so you need to be 31 aware of this.

If possible, you should try 30 and use a convenience library to ease the 29 process of doing scalable asynchronous I/O.

Microsoft's 28 Concurrency Coordination Runtime is one example of a .NET library designed 27 to ease the difficulty of doing this kind 26 of programming. It looks great, but as I 25 haven't used it, I can't comment on how 24 well it would scale.

For my personal projects 23 that need to do asynchronous network or 22 disk I/O, I use a set of .NET concurrency 21 / I/O tools that I've built over the past 20 year, called Squared.Task. It's inspired by libraries 19 like imvu.task and twisted, and I've included some working examples in the 18 repository that do network I/O. I also have 17 used it in a few applications I've written 16 - the largest publicly released one being 15 NDexer (which uses it for threadless disk I/O). The 14 library was written based on my experience 13 with imvu.task and has a set of fairly comprehensive 12 unit tests, so I strongly encourage you 11 to try it out. If you have any issues with 10 it, I'd be glad to offer you some assistance.

In 9 my opinion, based on my experience using 8 asynchronous/threadless I/O instead of threads 7 is a worthwhile endeavor on the .NET platform, as 6 long as you're ready to deal with the learning 5 curve. It allows you to avoid the scalability 4 hassles imposed by the cost of Thread objects, and 3 in many cases, you can completely avoid 2 the use of locks and mutexes by making careful 1 use of concurrency primitives like futures and promises.

I used Kevin's solution, but he says that solution lacks 2 code for reassembly of messages. Developers 1 can use this code for reassembly of messages:

private static void ReceiveCallback(IAsyncResult asyncResult )
{
    ClientInfo cInfo = (ClientInfo)asyncResult.AsyncState;

    cInfo.BytesReceived += cInfo.Soket.EndReceive(asyncResult);
    if (cInfo.RcvBuffer == null)
    {
        // First 2 byte is lenght
        if (cInfo.BytesReceived >= 2)
        {
            //this calculation depends on format which your client use for lenght info
            byte[] len = new byte[ 2 ] ;
            len[0] = cInfo.LengthBuffer[1];
            len[1] = cInfo.LengthBuffer[0];
            UInt16 length = BitConverter.ToUInt16( len , 0);

            // buffering and nulling is very important
            cInfo.RcvBuffer = new byte[length];
            cInfo.BytesReceived = 0;

        }
    }
    else
    {
        if (cInfo.BytesReceived == cInfo.RcvBuffer.Length)
        {
             //Put your code here, use bytes comes from  "cInfo.RcvBuffer"

             //Send Response but don't use async send , otherwise your code will not work ( RcvBuffer will be null prematurely and it will ruin your code)

            int sendLenghts = cInfo.Soket.Send( sendBack, sendBack.Length, SocketFlags.None);

            // buffering and nulling is very important
            //Important , set RcvBuffer to null because code will decide to get data or 2 bte lenght according to RcvBuffer's value(null or initialized)
            cInfo.RcvBuffer = null;
            cInfo.BytesReceived = 0;
        }
    }

    ContinueReading(cInfo);
 }

private static void ContinueReading(ClientInfo cInfo)
{
    try
    {
        if (cInfo.RcvBuffer != null)
        {
            cInfo.Soket.BeginReceive(cInfo.RcvBuffer, cInfo.BytesReceived, cInfo.RcvBuffer.Length - cInfo.BytesReceived, SocketFlags.None, ReceiveCallback, cInfo);
        }
        else
        {
            cInfo.Soket.BeginReceive(cInfo.LengthBuffer, cInfo.BytesReceived, cInfo.LengthBuffer.Length - cInfo.BytesReceived, SocketFlags.None, ReceiveCallback, cInfo);
        }
    }
    catch (SocketException se)
    {
        //Handle exception and  Close socket here, use your own code
        return;
    }
    catch (Exception ex)
    {
        //Handle exception and  Close socket here, use your own code
        return;
    }
}

class ClientInfo
{
    private const int BUFSIZE = 1024 ; // Max size of buffer , depends on solution
    private const int BUFLENSIZE = 2; // lenght of lenght , depends on solution
    public int BytesReceived = 0 ;
    public byte[] RcvBuffer { get; set; }
    public byte[] LengthBuffer { get; set; }

    public Socket Soket { get; set; }

    public ClientInfo(Socket clntSock)
    {
        Soket = clntSock;
        RcvBuffer = null;
        LengthBuffer = new byte[ BUFLENSIZE ];
    }

}

public static void AcceptCallback(IAsyncResult asyncResult)
{

    Socket servSock = (Socket)asyncResult.AsyncState;
    Socket clntSock = null;

    try
    {

        clntSock = servSock.EndAccept(asyncResult);

        ClientInfo cInfo = new ClientInfo(clntSock);

        Receive( cInfo );

    }
    catch (SocketException se)
    {
        clntSock.Close();
    }
}
private static void Receive(ClientInfo cInfo )
{
    try
    {
        if (cInfo.RcvBuffer == null)
        {
            cInfo.Soket.BeginReceive(cInfo.LengthBuffer, 0, 2, SocketFlags.None, ReceiveCallback, cInfo);

        }
        else
        {
            cInfo.Soket.BeginReceive(cInfo.RcvBuffer, 0, cInfo.BytesReceived, SocketFlags.None, ReceiveCallback, cInfo);

        }

    }
    catch (SocketException se)
    {
        return;
    }
    catch (Exception ex)
    {
        return;
    }

}

You can find a nice overview of techniques 1 at the C10k problem page.

You could try using a framework called Adaptive Communications Environment (ACE) which 15 is a generic C++ framework for network servers. It's 14 a very solid, mature product and is designed 13 to support high-reliability, high-volume 12 applications up to telco-grade.

The framework 11 deals with quite a wide range of concurrency 10 models and probably has one suitable for 9 your application out of the box. This should 8 make the system easier to debug as most 7 of the nasty concurrency issues have already 6 been sorted out. The trade-off here is 5 that the framework is written in C++ and 4 is not the most warm and fluffy of code 3 bases. On the other hand, you get tested, industrial 2 grade network infrastructure and a highly 1 scalable architecture out of the box.

I would use SEDA or a lightweight threading 3 library (Erlang or newer Linux. See NTPL scalability on the server side). Async coding 2 is very cumbersome if your communication 1 isn't :)

Well, .NET sockets seem to provide select() - that's 5 best for handling input. For output I'd 4 have a pool of socket-writer threads listening 3 on a work queue, accepting socket descriptor/object 2 as part of the work item, so you don't need 1 a thread per socket.

I would use the AcceptAsync/ConnectAsync/ReceiveAsync/SendAsync 5 methods that were added in .NET 3.5. I have 4 done a benchmark and they are approximately 3 35% faster (response time and bitrate) with 2 100 users constantly sending and receiving 1 data.

To people copy pasting the accepted answer, you 10 can rewrite the acceptCallback method, removing 9 all calls of _serverSocket.BeginAccept(new 8 AsyncCallback(acceptCallback), _serverSocket); and 7 put it in a finally{} clause, this way:

private void acceptCallback(IAsyncResult result)
    {
       xConnection conn = new xConnection();
       try
       {
         //Finish accepting the connection
         System.Net.Sockets.Socket s = (System.Net.Sockets.Socket)result.AsyncState;
         conn = new xConnection();
         conn.socket = s.EndAccept(result);
         conn.buffer = new byte[_bufferSize];
         lock (_sockets)
         {
           _sockets.Add(conn);
         }
         //Queue recieving of data from the connection
         conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
       }
       catch (SocketException e)
       {
         if (conn.socket != null)
         {
           conn.socket.Close();
           lock (_sockets)
           {
             _sockets.Remove(conn);
           }
         }
       }
       catch (Exception e)
       {
         if (conn.socket != null)
         {
           conn.socket.Close();
           lock (_sockets)
           {
             _sockets.Remove(conn);
           }
         }
       }
       finally
       {
         //Queue the next accept, think this should be here, stop attacks based on killing the waiting listeners
         _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);       
       }
     }

You 6 could even remove the first catch since 5 its content is the same, but it's a template 4 method and you should use a typed exception 3 to better handle the exceptions and understand 2 what caused the error, so just implement 1 those catches with some useful code.

I would recommend to read these books on 5 ACE,

• C++ Network Programming: Mastering Complexity Using ACE and Patterns
• C++ Network Programming: Systematic Reuse with ACE and Frameworks

to get ideas about patterns allowing you 4 to create an efficient server.

Although ACE 3 is implemented in C++, the books cover a 2 lot of useful patterns that can be used 1 in any programming language.