Grid Computing

We developed a simple and fast grid computing system, using .net remoting, particularly adapted to local networks.
This system is composed of :

• Executors which are windows .net services that actually perform the computing, you can install executors on any windows machine with .net framework 2.0.
• A manager (a windows .net service requiring .net 2.0) that maintains a list of available executors and a library of computing routines.
• Clients that performs parallel computation as simply as possible.

How it works :

• Each executor periodically indicates its availability to the manager.
• Executors are automatically updated whenever a change occurs in the manager computation routines library.
• A client creates an array of parameters to be computed, asks the manager for available executors and then asks for each executor to compute items of the parameters array. All of this is done transparently without having to bother about the complexity of network communications and threads synchronization.
  When each parameter has been computed (a parameter may be a complex structure) the client builds the final solution (by aggregating or averaging elementary solutions for instance).

Development framework :

• computing routines are simple, they have to inherit from a base class and redefine a function ‘Calculate’ whose signature is : public override object Calculate(object obj);
  obj is an elementary parameter (it may be an array of other objects) and the function should return an elementary solution (it may also be an array of other objects).


- sample : Pi calculation (Monte Carlo method)



using System;
using ZelGrid;

namespace PI_montecarlo
{
    public class PI_montecarlo : ZelGrid.Std0Calculator
	{
     private Random rndNumbers;

     public PI_montecarlo(object dataIni): base(dataIni)
		{
            rndNumbers = new Random();
		}

     public override object Calculate(object obj)
        {
        Double dres = 0.0;
        try
            {
                long nb = (long)obj;
                long nbin = 0;
                double dx = 0.0;
                double dy = 0.0;
                for (long i = 1; i <= nb; i++)
                {
                    dx = rndNumbers.NextDouble();
                    dy = rndNumbers.NextDouble();
                    if (dx * dx + dy * dy <= 1) 
                       nbin = nbin + 1;
                }
                dres = 4.0 * nbin / nb;
            }
            catch (Exception)
            {
                return null;
            }
            return dres;
        }
	}
}



• developing a client is very easy, here is an example of code :

  using ZelGrid;
  using System.Runtime.Remoting;
  ...
  String ConfigFilePath = 
      Path.GetDirectoryName(Application.ExecutablePath) 
      + @"\Client.exe.config";
  RemotingConfiguration.Configure(ConfigFilePath,false);
  ...
              
  long nbIter = 100000000;
  long nbCalc = 2000;
  zgs = new ZelGrid.CalcArray(@"tcp://{0}:9000/ZelExecutor", 
            null, 
            "PI_montecarlo",  
            "PI_montecarlo.PI_montecarlo", 
            ZelGrid.Calculator.TYPECALC.Standard); 
  
  //Parameters
  long i = 1;
  do
  {
  zgs.AddItem(nbIter);
  i += 1;
  } while (i <= nbCalc);
  
  //Calculation
  CalcArray.CALCRESULT resul = 
       zgs.Calculate(true, 0, 0);
  
  //Result
  Double piEval=0.0;
  for (i = 0; i < nbCalc; i++)
  {
  object lRes = zgs.getResult(i);
  piEval=piEval+(double)lRes;
  }
  piEval = piEval / nbCalc;			
  

Advantages :

• Developing computing routines and clients is very simple.
• You only need .net framework 2.0 to be installed, you don’t have to rely on a database on the manager.
• It uses .Net remoting to communicate between computers and it’s the fastest solution.
• It is stable.
• If an executor is stopped during a calculation the calculation will still perform correctly. If a new computer appears to be available during a calculation it will be included to perform the calculation.
• It is multitasked, if several clients ask for a calculation, those calculation will be performed simultaneously.
• executors run with a default low priority (it can be changed), so you can install them on desktops without slowing them.
• You can easily develop wrappers in C# to use existing C or C++ routines.
• You can develop routines in IronPython (but they will be slower) or use the framework from IronPython.

If you want to test :

Download Manager : ZelManager.msi
Download Executor : ZelExecutor.msi
Download Client sample : Client.zip

Those are demo versions, Executor stops after 1 hour (you have then to restart the service) and automatic update of routines library is not enabled.

Installation instructions :

• First Install the manager on a computer, choose a tcp port (default 9001)
• Install executor on other computers (you can install an executor on the manager computer) specifying the manager name (or ip address) and tcp port (9001), choose tcp port for the executor (default 9000), leave blank "Update directory" (it is useless on demo version) check "use ip" if the client can't resolve executor names but can connect to it's ip.
• Unzip client.zip in a folder. Modify the file Client.exe.config : replace MANAGER by your manager name or ip in the following line : tcp://MANAGER:9001/ZelManager/ZelManagerObj and modify the "execport" key to the executor tcp port if needed.
• Open firewalls on TCP port 9001 between executors and manager and TCP Port 9000 between client and executors.

To develop your own routines:

• Use .net framework 2.0, start from the pi_montecarlo sample above.
• Copy your assembly in ZelExecutor directory on each executor machine.

To develop a client :

• Import ZelGrid assembly
• Configure .net remoting in your client
  sample: you can use a configuration file
  
  <configuration>
  <system.runtime.remoting>
  <application name = "ZelExecutor">
  <client>
  <wellknown
  type="ZelGrid.ZelManagerObj, ZelGrid"
  url="tcp://MANAGER:9001/ZelManager/ZelManagerObj"
  />
  </client>
  <channels>
  <channel ref="tcp client" name="client"/>
  </channels>
  </application>
  </system.runtime.remoting>
  </configuration>
  (MANAGER should be replace by your manager name or ip)
  
  and load it :
  RemotingConfiguration.Configure(ConfigFilePath,false);
  
  
• Create a parameters array and add parameters to it
  zgs = new ZelGrid.CalcArray(@"tcp://{0}:9000/ZelExecutor", //{0} will be automatically replaced by each available executor name or ip.
  null, //object : datas that could be used by your Executor constructor
  "PI_montecarlo", //executor assembly name without extension
  "PI_montecarlo.PI_montecarlo", //executor class containing calculate method
  ZelGrid.Calculator.TYPECALC.Standard); //executor type Standard (assembly that can be loaded dynamically), IronPython (developped in IronPython) and Non Standard (rarely used)
  
  2 other constructors may be used : the first one specifies a number of processors requested (of cores) as last parameter and the second one specifies the name of a given machine as last parameter (it allows you to ask for a given machine to do a particular computation).
  
  Add parameters:
  do { zgs.AddItem(object); ...} while (condition);
  
  
• Start calculation
  zgs.Calculate(true, //Is it a blocking call ?
  0, //Timeout for whole calculation (ms) no timeout if 0
  0);//Timeout for an elementary calculation (ms) no timeout if 0
  return a CALCRESULT {CalcOk,NoExecutorAvailable,TimeoutAllCalc};
  
  
• Analyze results
  for (i = 0; i < nbCalc; i++) { object lRes = zgs.getResult(i); ...}
  


• If you want information during a calculation use infos structure
  for (int i = 0; i < zgs.infos.Count; i++) {long nbItemsCalculated = ((InfosMachine)zgs.infos.GetByIndex(i)).nbItems;}



• Synchronisation: You can start several computation on different array of parameters and synchronise them using the m_Ended property (of type ManualResetEvent) of the array of parameter.
  ex: zgs1.Calculate(false);
  zgs2.Calculate(false);
  zgs1.m_Ended.WaitOne();
  zgs2.m_Ended.WaitOne();
  zgs3.Calculate(false);
  Wait for computation 1 and 2 to be over before launching computation 3.
  

Contact us for more information :