Expérience réussie d’Oracle Big Data et Busit

Retour d’expérience réussie d’Oracle Big Data et Busit : l’ intégration facile à l’aide de ORDS et NoSQL

 

Propulsé par l’initiative #WeLoveStartups d’Oracle France, Busit a décuplé ses possibilités Big Data en utilisant ORDS et Oracle NoSQL.

Texte complet en version originale :

 

Boosted by the Oracle France #WeLoveStartups initiative and sponsored personally by Christophe Bonenfant, Busit tremendously increased its Big Data capabilities using ORDS to leverage the Oracle NoSQL storage engine.

 

The Oracle universe is very large and may seem complex at first sight because it is so opulent with precise documentation, tutorials and examples. All is a matter of identifying the proper resource and the right method to access it. Thanks to Dario Vega for his expertise and Olivia Faucheux for her advices throughout the We Love Startups program, in a matter of hours, we were able to setup a fully functional Big Data infrastructure. We used the Big Data Lite virtual machine to inject raw values from IoT into Oracle NoSQL using a dedicated BusApp.

 

Busit is a software solution to process and manage workflows of asynchronous real time events focused around IoT, services, and interoperability. The diversity and evolutive unpredictability of this type of data, require a flexible database storage mechanism able to handle the load. Oracle NoSQL is the perfect match because it is storing data dynamically in the form of documents while preserving physical data integrity. By design, the data model remains simple while offering powerful ACID transactions and JSON support. Moreover, its storage engine can scale horizontally as the demand for performance or massive storage requirements increase. As we can expect from Oracle, we benefit from a scalable throughput with bounded latency using both local and remote failover and synchronization.

 

The next challenging question is about the communication channel between components. The Busit solution may be deployed on large cloud infrastructures as well as embedded in limited gateways close to the source of data. Therefore, maintaining an open connection to the database is a costly operation. Using ORDS, many independent instances can send stateless requests to Oracle NoSQL with little overhead using standard HTTP requests. Thus, the memory footprint and CPU cycles spared can be fully dedicated to the business logic. Nonetheless, this design allows for standard networking capabilities such as load balancing, failover and DNS aliases to perform natively regardless of the higher-level infrastructure without any extra configuration. Using this communication scheme, new cloud instances of both tiers can pop on the fly at runtime even if they are managed independently.

 

No more than three simple steps are required to run ORDS and access the Oracle NoSQL Cluster (running on commodity hardware, on a Big Data appliance or in a cloud). First, download and installORDS and unzip it:

 

unzip ords.3.0.6.176.08.46.zip

 

Second, configure ORDS to hook on the NoSQL key-value store:

 

java -jar ords.war nosqladd  nosql_kvstore kvstore bigdatalite:5000

 

Third, start ORDS in standalone mode (or using a supported Java EE application server for production use: WebLogic, GlassFish or Apache Tomcat)

 

java -jar ords.war standalone

 

Finally, test the configuration by loading some data:

curl -X POST --data-binary @data.json -H "Content-Type: application/json" "http://bigdatalite:1234/ords/nosql_kvstore/tables/test/"

 

On its end, the Busit application model allows for very easy integration of any service or API based on the natural data flow actors: producers, consumers and transformers. BusApps are composed of a declarative configuration file in JSON and a single code file in PHP or as a JAR file. Once validated, the application is injected in the Busit ecosystem and is directly compatible with all other applications in order to build event-driven workflows. The Oracle ORDS application is considered as a data consumer as well as a data producer because it can both absorb data to insert in the database, or retrieve a specific query periodically. Therefore, it has two data channels:

 

{
    "channels":
    {
        "insert":
        {
            "pattern": "consumer"
        },
        "query":
        {
            "pattern": "producer",
            "timer": "E-E-E-E-E"
        }
    }
}

 

 

Behind the scenes, the BusApp will call the ORDS API to insert new values or perform a query using an ORDS Filter syntax. In less than 80 lines of trivial code, we have a generic read-write bridge to Oracle NoSQL.

 

class OracleORDS extends App implements Consumer, Producer
{
    public function consume($message, $in)
    {
        $payload = array();
        foreach( $message->content() as $k=>$v )
            $payload[$k] = $v;
        $payload = json_encode($payload);
        
        HTTP::send($this->config('url') . '/tables/' . $this->config('table') . '/', array(
            'method' => 'PUT',
            'params' => $payload,
            'headers'=> array('Content-Type' => 'application/json')
            ));

       [...]
    }

    [...]
}

 

Final code and config files

 

 

When using the Oracle ORDS BusApp, users will specify the url endpoint and the table name; optionally, use a query filter to retrieve data. That’s it, really !

 

Gather data from various sources, display them on a map and get instant notifications.

 

 

 

Using the combined power of Oracle NoSQL and Busit, prototyping a real life project is a matter of days. If a new protocol is encountered, integrating the missing piece into the ecosystem can be done in a few lines of code. Overall, operators and managers benefit from a robust hands-on solution to design the workflows and decide proficiently with the ability to perform Big Data analytics the enterprise-grade way.