Build Third party multi-service provider integration using Event-Driven based architecture & Reactive programming

 Objective

• This document highlights the solution approach for multi telecom service provider integration on AWS platform using Event-Driven based Architecture & reactive programming using the Java based Spring Boot framework & Java RX.
• In telecom, there are two types of incident management of creating the ticket, CM i.e. change management & TM i.e. trouble management 
• CM  Creating the request/ticket for the infrequent events. This can be happened one/two times in a year, as a production release plan. 
• For the CM category, there are many service providers who are giving support for creating the ticket CM.
• TM  Trouble Management, request/ticket would be created frequently in a year, like creating the ticket for the network router, cable, device, etc.
• About to crate TM ticket, there is so many TM category of multi-service provider.

Key challenges

• On-boarding the multi telecom service provider in a common platform is really challenging due to each service provider has their own semantic APIs which can pose a challenge.
• Latency  While calling multiple service provider API, which increases the latency of service even in the case when calls are made parallel.
• Failure  One service provider API failure could make a major impact on the performance degradation of the overall service.
• Timeouts  For any service provider’s API  time out, which could impact overall service performance and availability.

Conceptual solution approach

To build a multi-service provider platform, it is required to find out the exact architecture, here EDA,  which can be help full for the fulfillment of desire goal.

Building the solution on EDA enabled us to deliver a consistent customer experience while addressing failures and performance issues.

Conceptual HL solution design

Key components solution

User Interface: There are mainly three categories of users like web, mobile, IOS users. About user interface follows the single page application which is running on browser. Here, UI makes an API call to Service Orchestration MS with required service category provider parameters & receives a unique identifier for a specific telecom service provider.

Service Orchestration MS: This service basically generates the event message for a specific service provider it might be CM or TM provider. And the message sends to EVENT BUS. A callback to submit the response from the Ticket Processing MS (CM || TM) in a reactive manner. This MS will also play the role of both event producer & consumer.

Event Bus: It will play a major role in EVD-based architecture, it enables transportation, and routing, and redirect the event to the right services. Service orchestration service raises a set of events to the EVENT BUS for multi-service provider-based requests, there is one event for each partner that is listened to by the reactive services.

Reactive Services: All the ticket processing MSs will consume events from their respective EVENT BUS, after that, it will do the required processing i.e. transforming the API request/response, And making calls to third-party API (multi-service provider) in a parallel way. Each service also implements the JMS listener to consume the event from the corresponding QUEUE. After getting the response from the third-party service provider, it will PUSH the notification again EVENT BUS.

Caching: Service orchestration MS will consume the response for any specific third-party service provider from EVENT BUS, after that, it will commit the response to CACHE (Here REDIS). UI interface will poll the data from this CACHE and load it into an angular single-page application.

Technology Stack

Mapping conceptual components on AWS

AWS provides cloud-native services enabling how to do brain mapping with logical components & real components in AWS.

User Interface: Built the load balancing interface in Angular and host it on Apache Web Servers running (AWS ECS) AWS Elastic Container Service. AWS load balancing distributes traffic evenly across the containers. Running a container gives the required flexibility and scalability needed.

Service Orchestration MS: This microservice is built using Spring Boot and running on AWS ECS containers. The choice of Spring Boot and AWS ECS gives flexibility and scalability through cluster auto-scaling.

Event Bus: To build multi-service-provider integration on AWS using EDA, AWS SNS (Amazon Simple Notification) notification service is used. This offers the right capability, implemented by defining topics, this topic is subscribed by reactive services for each service provider integration, here, AWS SQS (Amazon Simple Queue Service) is used for each service provider-specific messaging QUEUE to consume the message from AWS SNS for asynchronous processing.

Reactive Services: These MSs are built using two different technologies, Spring Boot MS running on AWS ECS container and AWS Lambda functions. AWS Lambda is one of the key components which can be help full for building the multiservice provider integration using an event-driven, server-less computing platform in this scenario.

Sequential Workflow

Sequential workflow highlights.

The Above sequence of flow (Slide 13) is summarized below.

1 ) UI: If the end-user crates the request/ticket for Change management (CM), then it will send the request with the below parameters like

                       UserId, Mobile No, Ticket category (Whether CM or TM), service provider Id. And it sends the request to Service Orchestration.

2 ) Service Orchestration: It will generate the HashKey using a combination of the below parameters.  (UserId/Mobile No, Ticket category, Service provider ID).

       Ex: HashKey ID: 234D9705G  (Using Java simple HashCode method or    

        Apache common utility to generate the hash code). 

• This Hash code is a unique identifier to identify any user-specific third-party CM/TM  provider service-related request/ticket. This service will basically play two key important roles here, first of all, it will generate the message with hash key (234D9705G ) & PUSH the message to AWS SNS.

• While UI calls this service first time, then Service Orchestration will return the response with Hash Code & parallels it also do the other necessary operations

3) CM/TM Ticket processing MS: This service will play two key roles here, 

• It will first consume the message from CM/TM AWS SQS which is already subscribed with AWS SNS, 
• Do the required transformation, 
• After that using the REST client, it will call a third-party provider (CM/TM) service.
• Get the response from the third-party service provider & commit the response in NOSQL MongoDB against the Hash Key (234D9705G ).
• PUSH the response message also to AWS SNS against Hash Key (234D9705G ).

4) Service Orchestration

• This service also takes the responsibility of consuming the message from AWS SNS (i.e. this message is a combination of third-party service provider response & hash key (234D9705G )
• After successfully consuming the message from AWS SNS, it will commit the message to REDIS CACHE with key-value pair like  Key: 234D9705G, Value:  CM/TM third-party service provider response.

 5 ) UI: Using the polling mechanism, it will every time check whether data is available in REDISH CACHE against unique Hash Code/Key 234D9705G  for any third-party CM/TM provider response. If the data is available then it will be rendered on the UI page.

Service Catalogue

API Endpoint (Service catalog details)

In Service Orchestration, basically, two API Endpoints are used. One Endpoint is taking the responsibility of initiating the request and creating the CM/TM service provider ticket & another endpoint will take care of  Polling the records from  REDIS which we have received from third-party service providers.

1) Identify the third-party service provider & call their key details. (Using this endpoint, UI will initiate the request for third-party service provider data.

       Http method : POST

       Endpoint: /v1/telecom/tickets.

       Request payload: User Id, Mobile No,  Ticket category, service provider Id

       Response  payload: send HashCode : 234D9705G  as a response.

2) Poll CM/TM specific response data from REDIS. (UI will use this endpoint to fetch the records from REDIS.

      HTTP method: GET.

      EndPoint :  /v1/telecom/ticket/{HashCode}   (Here HashCode : 234D9705G )

Data Storage

AWS PostgressSql 9.6 Database: 

Postgres is a powerful, open-source, object-relational database system that uses and extends the SQL language combined with many features that safely store and scale the most complicated data workloads.

Benefits:  1) we can easily manage and deployment, for high-performance OLTP applications, automated backup and recovery, The automated backup feature of Amazon RDS enables recovery of your PostgreSQL database instance to any point in time within your specified retention period of up to thirty five days

AWS MongoDB 3.2 : Using this database for dumping the huge volume of third-party service provider responses, which can be help full for data analysis or reconciliation.

Its flexible data model enables us to store data of any structure, and it provides full index support, sharding, and replication

Third Party Remote API tracking

Functional Requirements: Using third-party Remote API tracking, we can able to track external service provider requests & responses, if any issues in external service provider then we will generate the log message for same in Production  ENV, in this way we can troubleshoot any service provider specific issues.

Non-functional Requirements: All the services logs should be required on an immediate basis in the ELK/Kibana. Indexes should be there for every third party service provider

Solution: Using the Event-Driven based architecture to fulfill the above challenges.  Regarding that below components are used.

1) CM/TM ticket processing MS:  This domain service is using one RemoteAPI tracking lib/dependency, using this lib/artifact each domain service will send the log message like REST API request/response, an error message to AWS SQS.

RemoteAPITrackingLib: This component is taking care of transforming the rest API request/response, success/error & structuring the message, configuring AWS SQS, and pushing the message to AWS SQS.

RemoteApiTrackingProcessor: This is a Lambda based server-less event-driven service which is taking the responsibility of consuming the message from AWS SQS, configuring AWS ElasticSerach & PUSH the data to AWS ElasticSerach. Using the KIBANA here for data visualization, application monitoring, log & time-series.

Event Notification Solution Approach

Sending the event notification to multiple mediums or modes (Mail, SMS, IVR) is a really challenging task in MS service-based ARCH platform, our below event Event-driven based ARCH can overcome this problem. To implement this ARCH, we are using one third-party tool CBUS  which will take care about forward the SMS, Email, IVR notifications to the respective end-users.

Using the below logical components which are help full to fulfill the EDA ARCH solution.

CM/TM ticket processing MS: Using the REST client making a call to NotificatioGateWay

MS with templateDataMap parameters and other necessary details.

Here, templateDataMap  is a key value pair i.e templateDataMap : {ticketNo: “2345”}

NotificatioGateWay MS: This service will take care of reading the template Id from the database and do rest of other necessary actions, crate the message for SMS/Email and send to AWS SQS. Here, template Id is the template or text file name. Example: SMSNotificationTemplate.txt

AWS S3: AWS S3 bucket contains all the template or text files, each file body contains some static value and placeholder value like below. Example: SMSNotificationTemplate.txt

“This ticket number {ticketNo} is not valid, please recheck” 

We are going to replace this placeholder ticketNo with templateDataMap  key value ticketNo: “2345.

SMS or Email MS: To make the event notification service more robust, for each notification mode, we have to create two separate MS modules i.e. Email and SMS. Each service will take care of its own functionality, so, in the future, if there is any enhancement comes, it will not make any impact on the existing system, hence, in this way we can decouple the whole system. Basically, these two modules are taking care below actions, 

• One is read the template id or text file from AWS S3 bucket, 
• Consume templateDataMap related message from AWS SQS.
• Transforming the request for third party CBUS tool.
• Using the REST client to make a connection to the CBUS tool, and send the request to this tool.

CBUS : This is a third party event notification tool, that provides API and visual tools to communicate with customers worldwide through various channels, including SMS, MMS, RCS, WhatsApp for Business, Video, and Voice.

Handling third party API Security

Before accessing the third-party API service, the external service provider service IP  has to register in the current platform.  They will provide a partner Key and secret Key which will be required to login to get AUTH token and that will be passed in subsequent requests in headers. These two partner Partner Key and secret Key should be keep in database or application properties file (external). From a security point of view, we should also maintain these in AWS parameter store. (Will update more on this very soon)

Artifacts

Artifactory is designed to mimic a wide variety of package registers, including Java repositories which contain different types of internal and external lib or artifact and Docker. Docker registry will be used to store docker images before they are deployed to Amazon Elastic Container service for Kubernetes (EKS), EKS is an AWS cloud-based container management service that natively integrates with Kubernetes to deploy Microservice based applications.

Code Quality

Checkstyle : it is an open source tool that checks code against a configurable set of rules.

• It will be integrated into Java project  via Maven and by IDE plugins
• Rules will be used to check for violations.
• Its integrated into CI/CD pipeline as well as developer environment .

Spotbugs : This is a program which uses static analysis to look for bugs in java code. It is the spiritual successor of FindBugs.

SonarQube : it is an open source platform for continuous inspection of code quality to perform automatic reviews.

DevOps CI/CD

• AWS managed continuous integration & delivery service used to automatically build & deploy code. 
• It's based on JENKINS.
• The team defines the build, test, and deployment pipelines in the  AWS TERAFORM file and registers the project with the platform, when changes are pushed to the Source REPO , the platform will run the appropriate pipeline jobs.
• Unit tests and component tests are typically run as part of the build phase. AWS platform collects, tracks and displays these test results and static analysis reports.
• Build logs are also available to see the status of the build.

Load testing or performance engineering

From the Load testing point of view, first, we have to benchmark and consider the vertical scaling parameters first after that consider Horizontal scale parameters.

The horizontal scale is something we can define for different env. And it depends on traffic.

The vertical scaling parameters should be independent of the tasks running.

From a performance or load testing point of view, we have used the below technology components. 

1) NewRelic

2) JMeter

Below are key parameters need to consider at the time of  service or system performance engineering.

Check the database connection time, SQL execution time, Cluster, Service, No. of containers, CPU units, Hard Memory (MiB), Soft Memory (MiB), thread dumps, Memory Allocation. CPU allocation, Heap Max Count, Heap Min Count

How this solution approach address multi-service provider integration

This solution is built on the foundation of Event-Driven based architecture with reactive design and it will identify the below challenges.

Latency: Due to UI interface asynchronously polling the response, it receives the partner response as soon as it arrives. This makes the response latency on the fastest service provider third-party API rather than slowest.

Timeout: This timeout is set in the UI layer for polling, hence, while a service provider doesn’t respond, it time it out without ant degradation. This timeouts are set based on the established user experience benchmark, i.e. how long do we let a user wait to see all service provider response before degrading their experience.

Failures: If there is any third-party API failure, the UI will time out, And system API tracking can also enable a circuit breaker pattern to take a service provider offline in the event of continuous failure.

Customer Experience: Above approach will give a consistent experience to the users, users can make their choice from the third party service provider in a reactive way as received, rather than waiting for all responses to be displayed.

Conclusion

In the IT digital field, APIs are the most common key mechanism for system integration. Building a solution which is resilient, scalable and share the best customers experience is one of the biggest challenge. Here, Event Driven based Architecture with Reactive programming offers a solution to address this issues. Using AWS cloud platform, now able to achieve the availability of over 99.9%, while maintaining a positive customer experience on this platform.