6

I have a .NET function that does some complex calculation. Depending on the parameters that are passed in, the function:

  • Takes anywhere from several minutes to several hours to run
  • Uses 100% of a single core during the computation
  • Requires anywhere from 100s of MB to several GB of memory
  • Writes anywhere from several MB to several GB of data to disk
  • May throw an exception, including an OutOfMemoryException

The amount to data to be written to disk can be accurately predicted from the function parameterisation. There is no easy way to predict the other resource requirements from the function parameterisation.

I need to expose this function via a web service. This service needs to be:

  • Resiliant and gracefully report any problems during the calculation
  • Capable of handling concurrent requests, as long as there are sufficient resources to handle the request without significant performance degradation, and to gracefully deny the request otherwise.

I'm intending to handle the long-running nature by having the initial request return a status resource that can be polled for progress. Once the calculation is complete this resource will provide the location of the output data, which the client can download (probably via FTP).

I'm less clear on how best to handle the other requirements. I'm considering some sort of "calculation pool" that maintains instances of the calculator and keeps track of which ones are currently being used, but I haven't figured out the details.

Does anyone with experience of similar situations have any suggestions? As long as the solution can run on a Windows box, all technology options can be considered.

Akash
  • 2,311
  • 1
  • 20
  • 37
  • Both VladV and Steven have provided very useful answers. I'm only allowed to mark one answer as accepted, so I've chosen VladV's based on the number of votes at the time of writing this comment. – Akash Nov 16 '10 at 11:40

2 Answers2

4

I'd suggest splitting your application in two parts.

  1. The web service itself. It's functionality:
    • Get a work item from a client;
    • Transfer this work to a backend service that performs the actual work;
    • Report progress and the result;
  2. The backend service. It's functionality:
    • Process the requests friom the web service;
    • Perform the actual computation.

The reasons for this design are
1) it's relatively difficult to handle the workload in the hosted application (ASP.NET) because the server (IIS) will manage the resources, while in a separate app you have more direct control;
2) two-tier design is more scalable - for instance, later you could easily move the backend to another physical machine (or several machines).

The web service should be stateless - for instance, after a request is accepted, the user gets back some ID and uses this ID to poll the service for the result.

The backend server, probably, has to maintain a queue of the requests to process and a set of worker threads that process them. The workers should monitor the resources available and take care not to overload the machine (and, of course, gracefully handle all possible error conditions).

VladV
  • 10,093
  • 3
  • 32
  • 48
  • Thanks, the idea of a backend service seems to be key. Now I need to think about how to isolate the calculation instances such that a failure in one doesn't kill the entire service. – Akash Nov 14 '10 at 08:53
  • 1
    Just create a catch-all exception handler in each thread used for calculation, so a background thread never throws an exception outside. You could also use AppDomains to isolate instances from each other, but I don't think it makes much sense here. And if you are concerned with running out of virtual memory, just move to x64 OS. – VladV Nov 15 '10 at 08:19
2

While you may want to provide a web service interface, web services are typically not designed for these kind of processes. What you might want to do is forward the request to a windows service (on a dedicated machine) that can handle this. Windows Services won't get recycled and you have much more control over the process.

About the calculation pool: what you can try is create a calculation queue (for instance a table in the database). This way you can have multiple windows services on dedicated machines processing the calculations. This can allow you to scale more easily.

Steven
  • 166,672
  • 24
  • 332
  • 435
  • Thanks, the idea of a backend service seems to be key. Now I need to think about how to isolate the calculation instances such that a failure in one doesn't kill the entire service. – Akash Nov 14 '10 at 08:56
  • 1
    What you can do is let one Windows service always run one calculation at a time and when a calculation (or the service) fails, you can flag that calculation as failed and log the failure. What we did on failure depended on the task at hand, but normally we would reschedule the task so that it would be picked up again later on. Of course you have to make sure tasks are atomic. When a task succeeded it would be flagged as 'done'. The administrators had an user interface that allowed them to see which tasks were scheduled, which were running, and which were failed and for what reason. – Steven Nov 14 '10 at 11:53