Aggregating responses from multiple asynchronous calls with WSO2 Micro Integrator
Sometimes we come across situations where in order to achieve a certain requirement there are times where we need to make some requests in a asynchronous manner whilst making other requests in a synchronous manner.
Consider this scenario, for a certain customer management system, 2 requests needs to be made to create the customer in an internal database calling a data service as well as an external one calling an external micro service. The order of these two requests are irrelevant and should be made in parallel to save time. Now here’s the tricky part, the next call we need to make should be in a synchronous manner and for this request we need a value sent in the response of one of those asynchronous calls. Also it is required that the customer creation is completed before running this third request. Now the questions arises as to how we can store the responses of those asynchronous requests seamlessly in order to make this third call?
Even though this may seem like a complex requirement it can be easily achieved using the WSO2 Micro Integrator (or the Enterprise Integrator) with the help of mediators. The below diagram depicts what we are trying to achieve in accordance with what was described above.
In order to make the two asynchronous calls to the two different backends we are going to make use of the “Clone Mediator” in WSO2. The clone mediator allows to make any number of asynchronous calls. It also has the ability to make synchronous calls with a configuration change, however we will not be discussing about that in this scenario here.
Once we make the parallel calls using the clone mediator we can then aggregate the responses of the two calls using the “Aggregate Mediator”. The order of the aggregation will not be predicted since the calls are happening asynchronously. And we can then process the rest of the logic which we need to acheive synchronously inside the “onComplete” section of the aggregate mediator.
Our logic will look something like this.
Now let’s take a look at how we can test out a scenario like this using the WSO2 Micro Integrator.
Let’s get started
- Download and run the WSO2 Micro Integrator. You can download it from here. Let’s name the extracted product distribution as MI_HOME. Copy over the mysql java connector to $MI_HOME/lib folder since we will be connecting to a mysql database in the sample.
- Clone this github repository into you machine. Let’s name this as REPO_HOME.
- Now we need to execute the database script needed to run this sample. Navigate to. $REPO_HOME/async-calls-aggregation-with-wso2-mi/databaseScript and execute the script against your mysql server. This will create database named “retail” which contains the two tables “customers” and “creditLimit”.
mysql -u <dbUsername> -p < retaildatabase.sql
4. Next we need to run our external customer microservice. This microservice returns a customer Identifier and the customer Type as outputs once a customer creation payload is passed. Navigate to the location $REPO_HOME/async-calls-aggregation-with-wso2-mi/backend-microservice/ExternalCustomerService and run the microservice using the below command.
java -jar External-Customer-Service-1.0.jar
5. Next we need to build the artifacts related to this sample. Navigate to the folder $REPO_HOME/async-calls-aggregation-with-wso2-mi/source/CustomerMgtProject.
6. Build the project by passing the database connection details as part of the build command as shown below.
mvn clean install -Ddb.host=<databaseHost> -Ddb.port=<databasePort> -Ddb.username=<databaseUsername> -Ddb.password=<databasePassword>
eg: mvn clean install -Ddb.host=localhost -Ddb.port=3306 -Ddb.username=dbuser -Ddb.password=dbpassword
7. Once the build is complete we need to copy the composite archive containing all of our artifacts into the micro integrator.
Copy the file found in $REPO_HOME/async-calls-aggregation-with-wso2-mi/source/CustomerMgtProject/CustomerMgtProjectCompositeExporter/target/CustomerMgtProjectCompositeExporter_1.0.0.car to the $MI_HOME/repository/deployment/server/carbonapps/ folder
8. Next let’s start the micro integrator. Navigate to $MI_HOME/bin and execute the below command. You should see an output similar to below which shows all the artifacts being deployed.
sh micro-integrator.sh
Next we can test out our sample. For that you can import the postman collection found in the location $REPO_HOME/async-calls-aggregation-with-wso2-mi/postman. If you do not have postman you can simply make a POST REST call using any REST client passing a payload similar to the one in the image.
Once you get the successful response from this call, you will be able to notice three things which happened behind the scenes.
- A customer record will be added to the customers table in the retail database. This record gets added as a result of the first asynchronous call to the dataservice.
2. Next, if you observe the microservice logs, you will notice an identifier has been generated by the External Customer microservice . This is a result of the second asynchronous call which was made to the micro service.
3. And finally you will notice a new record in the “creditLimit” database table which has a mapping for the credit Limit alongside the unique id returned from the asynchronous call above. This is the logic which was executed inside the “onComplete” method of the aggregate mediator in the sequence and has been executed once all the async calls were completed.
The id will be extracted from the aggregated response of the async calls and stored in the creditLimit table of the database. Here another call will be made to another endpoint which returns an activation message for the user account.
So there you have it. This is how easy it is to use responses from asynchronous calls and impelement use cases with a mix of both asynchronous and synchronous requests through the WSO2 platform.
References
[1] https://wso2.com/integration/micro-integrator/
[2] https://apim.docs.wso2.com/en/latest/integrate/integration-overview/
[3] https://apim.docs.wso2.com/en/latest/reference/mediators/clone-mediator/
[4] https://apim.docs.wso2.com/en/latest/reference/mediators/aggregate-mediator/
