I am currently working on implementing asynchronous processes in draftrack. Aim of asynchronous processes are to increase the response speed of a web application by relieving the page generation from non mandatory computation (I don t understand what I just wrote so let s take an example). For example on twitter, when you post a message, a lot of actions need to be taken (with example timie it takes to perform the action):
- Save the message (200ms)
- Publish the message to the followers (up to several seconds if lot of followers (lates take 1000ms))
- Make the message searchable (200ms)
Issue with this is that it takes a lot of time to perform all the actions. So if you wait all the processes to be finished to return an answer to the client, it may take a lot of time (here 1400ms). And it also burden the servers because you must scale to handle the pick amount of received message. Aim of asynchronous is to differentiate the actions that need to be perform while the user is waiting from processes that can wait a little longer. To take back the twitter example:
- Save the message (must be done will the user is waiting)
- Publish the message to the followers (can wait)
- Make the message searchable (can wait)
So instead of waiting 1400ms, the user only waits 200ms to have the message returned and other processes are done aside, this creating a lot of gains:
- The user has a faster response time
- You need less servers because these non mandatory actions can be queued and wait a server to be free in order being processed
- Easier to process issues as long as the mandatory steps have been accomplished in a good manner (you can retry the asynchronous functionality as long as needed)
So I implemented on draftrack: 2 ways to perform asynchronous process.
- Via a queue where system can post actions to be performed and these actions are called asynchronously after each page generation
- Via a cron lookalike where system post actions that need to be performed from time to time (like cleaning temporary data, improving lucene indexes, generating stats....)
Limitations where that I want the asynchronous process to run on the web server and do not force user to run other things like daemons or cron job.
To perform the asynchronous call I use a controller plugin that calls the asynchronous service access point after the end of the user page generation.
You can find the plugin
here. The only thing it does is calling the page
/asynchronous/message/process/ without the user noticing it (using curl
without waiting for the answer start /b curl '.$websiteURL.$baseURL.'/asynchronous/message/process/)
the
asynchronous module then checks in a Zend_Queue if there is a message waiting to be processed. If yes, the service access point is called and the calling module can then perform its asynchronous service. Publishing a service to the queue is as simple as calling the queue you can find an example on the status module
here line 47 and 48.
If there is no message to process, the system checks if there is a cron job to perform. Creating a new cronjob is as simple as calling via a webservice. The most difficult in cron being able to calculate the next time a job should be run (details available
on the cronjob class).
With these 2 tools, it is possible to have a very efficient system that put aside non critical computing for later (limitation being that if you don t have that many page viewed by the users then the asynchronous does not perform correctly).
