Stream Data from Twitter API with OAuth using Kettle

Streaming data from Twitter Api is really important from the data analytic perspective. Getting the pulse of your user community on the web and across different geographics gets really important in terms of making big decisions. Pentaho Kettle does provide you with few steps to read or stream data from Twitter. In fact there is already a sample example present in the installation directory of the PDI on the twitter. But that sample code might not work due to changes in the Authentication system of twitter api’s. Currently Twitter uses OAuth now for the third party users to access the data.

So in this blog will share few steps to actually stream the twitter data using OAuth:

STEP 1: Register an Application in Twitter (if you haven’t done yet):

Very first step is to register an application on Twitter. Click on this link and register yourself an application.

STEP 2: The Authentication details of the App:

Once you have registered your app on twitter, you will find few details shown. Check images below:

In the above images, my application name is : EnigmaRishu and twitter provides with various keys and access tokens. These keys and tokens are required when calling the twitter api from PDI in the request header.

STEP 3: Building a Transformation:

Define Parameters: This data grid step is where we would define all the parameters required as a part of the authentication process. The various parameters are documented on the twitter developer space. The token values and key values needs to be as per your registered application and it differs user wise.

Generate Header: This is a JS Step which includes codes developed by Paul Johnston (A JavaScript implementation of the Secure Hash Algorithm, SHA-1) and Netflix to handle the encryption of the keys and token. All we need to do is to pass the secure data to the header along with the query.

Calling Twitter REST Client: This step called a REST Client and we are actually using REST API of Twitter to read the twitter streams. The URL would be the combination of twitter search url and the query string. Check the sample code (as below) for more details.

Output – Twitter Result: The final search result and all the twitter. The raw format of the data is in JSON. If you want to further analyze the JSON format, use the steps like JSON Input file to analyse each section of the data.

Having said, twitter limits the streaming data for security reasons. More details here in the official document.

---

PDI Sample code:

The sample PDI as explained above is in the github repo here.

Advertisements