Using External Jars import in Pentaho Data Integration


Sometimes during the phase of the development, we might need to import some External JAR to the Pentaho Kettle Code. Pentaho DI has a step like “User Defined Java Class” where we can use some part of the Java code. Though this step doesn’t allow you to do full scale Java development, but still you can use some java features like methods, imports, etc. For more you can Pentaho Wiki. Now suppose, you have to get some java methods imported from a third party jar/ external jar, you need to do some changes as below:

STEP I: Create a library folder libext

Inside the /design-tools/data-integration/ folder of your pentaho installation, create a folder named libext. We would use this folder for storing the external jars / third party jars.

Capture22

STEP II: Configure launcher.properties file

Inside the same directory as above, you would find a folder called launcher. Open that folder and there you would find a property file as launcher.properties.

You need to add the above folder configuration to the classpath and libraries section of the code.

Capture

Note i have added :../libext to the above file since i want Kettle to read the content from that file too.

STEP III: Restart Spoon

Once all the configurations are done, simply start/restart the spoon in order to reflect the changes done to the properties.

STEP IV: The meal is ready

Now simply open “User Defined Java Class” Step and you can easily use the import your new jar files. In the image below, my class is stored inside the package code.rishu.kettle.test.*

Capture2Hence you can easily use an external jar.

Please feel to comment or suggest 🙂

Advertisements

24 thoughts on “Using External Jars import in Pentaho Data Integration

  1. Hi Reshu,

    I am trying to implement the same. But when I run the transformation, it keeps on running. Below given is my log trace:
    2015/09/21 16:06:15 – Spoon – Transformation opened.
    2015/09/21 16:06:15 – Spoon – Launching transformation [Experiment]…
    2015/09/21 16:06:15 – Spoon – Started the transformation execution.
    2015/09/21 16:06:16 – Experiment – Dispatching started for transformation [Experiment]

    Experiment is the name of my transformation. Could you please help?

    Thanks.

    Like

    1. hi Bhavya, it seems something is wrong in the transformation. I cannot say it unless i analyze ur ktr file. if possible send me the ktr file, i will try to see if i can get any issue. (rishu.shrivastava@gmail.com)

      Like

      1. Thank you for your response, Rishu. I created another transformation and tried again. The function in the jar file merges a number of xml files into groups of five to another location. When executing the tramsformation, it is creating the output files but the transformation keeps on running and doesnt stop. When I run the jar file through cmd, it takes only few seconds. I have sent the .ktr file to your gmail ID.

        Like

  2. Rishu,

    Thanks for the helpful post. I tired this but am unable to call methods inside the class imported from the external jar file. My java code in Eclipse that I am exporting into a jar file looks like this:

    public class importTest {
    // Return a test string
    public static String getTestString() {
    return new String(“This is a test…”);
    }
    }

    My UDJC in Pentaho looks like this:

    import java.util.*;
    import importTest.*;

    public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws KettleException
    {
    // Test importing class
    importTest t = new importTest();
    String s = t.getTestString();

    return true;
    }

    I imported the jar file successfully into the UDJC per above; however, when I test the class I get the following error:

    org.codehaus.janino.CompileException: Line 9, Column 28: A method named “getTestString” is not declared in any enclosing class nor any supertype, nor through a static import

    Can you see any reason why this doesn’t work?

    Thanks,
    Imanuel

    Like

    1. Have you configured the launcher.properties file properly, if you have placed your file in some seperate folder. This issue seems due to the UDJC step not being able to find the Class. Just recheck the properties. Also you may try to give the complete package path instead of the “import importtest.*”

      Like

      1. Rishu,

        Thanks for the kind reply. I did have launcher.property file set up correctly. The package name specification did make a difference, though. Initially I created my test class as part of the default package in Eclipse so there was no package name. When I moved the class to a newly created package and updated the UDJC with the full package name, it finally worked.

        Again, thanks for the reply and the help.

        -Imanuel

        Like

  3. Hi Rishu,
    I’m facing a similar situation to the one that Bhavya described where a transformation keeps on running and never stops. Do you have any idea of what could be the problem there? Have you ever faced an issue like that before?

    Any help will be great and highly appreciated.

    Thanks!

    Like

    1. Hi Juan,

      I was able to fix that issue. The reason why UDJC was throwing the error was because, my Table Input step itself was failing, as it was unable to connect to the DB due to firewall. Apparently, UDJC will throw error if any step in the transformation fails. After I opened the port for PDI, that issue was resolved.

      Hope it helps you.

      Regards,
      Bhavya

      Like

  4. Please i have two questions:
    1)- i want to know what is the need of if(first){ }
    2)-i want to deploy my pentaho di in a server, so i am asking if i should change somthing in the configuration file (luancher.properties).
    thank you.

    Like

    1. Hi Mostafa…first of all thanks for reading my blogs. Now as per your question,
      1) the “first” flag is inherited from the base step implementation of kettle class. It is used to guard some processing tasks, like figuring out field indexes in the row structure that only need to be done once.
      2) No you don’t have to make any changes to launcher.properties if you are deploying di in the servers. It will automatically detect the settings it requires.

      Like

  5. Hi,
    I have followed exact steps but my user defined java steps keeps on running indefinitely .
    Can you let me know the solution to this problem if you have ever encountered it ?

    Like

  6. Hi,

    thanks for the post, I did it already by “try and errors” so happy somebody tells it really works this way!
    I a have a question:
    I want to make the libext directory outside of the /data-integration installation.
    Moreover, I want the path to be dependent on some Windows env. variable, so that I can launch Spoon with different libext path for different projects, without touching the launcher.properties file.

    I tried to set
    librairies=./test:../lib:../libswt:${PENTAHO_LIBEXT_PATH}
    classpath=../:../ui:../ui/images:../lib:${PENTAHO_LIBEXT_PATH}

    But it doesn’t seem to replace the value of env. variable PENTAHO_LIBEXT_PATH.
    I thought it should works like the PENTAHO_INSTALLED_LICENSE_PATH which seems to be catched.

    Do you know if we can do such a thing?

    Thanks,

    Best regars

    Like

  7. Hi Reeshu,

    how we can pass parameter from kettle.properties in a launcher.properties file?
    i need to do something like this:
    libraries=../test:../lib:../libswt:../${devlib}
    but it is not picking the jar from devlib folder. could you please suggest

    Like

  8. Good day , Ive written code to read pcap files , and i am trying to use it as a jar file for a transformation .the class is using a third party jar file jnetpcap . ive added the third party jar file that my code uses to decode pcap files and my jar file to libext and added the imports . jnetpcap jar file has headers which i think it uses . However i cant seem to use the jar file in spoon i get the following exception

    poon – Using legacy execution engine
    2017/10/02 11:20:52 – Spoon – Transformation opened.
    2017/10/02 11:20:52 – Spoon – Launching transformation [pcap]…
    2017/10/02 11:20:52 – Spoon – Started the transformation execution.
    2017/10/02 11:20:53 – Spoon – The transformation has finished!!
    2017/10/02 11:26:37 – getPcapFiles – PREVIEW – Dispatching started for transformation [getPcapFiles – PREVIEW]
    2017/10/02 11:26:37 – ## TEST DATA ##.0 – Finished processing (I=0, O=0, R=0, W=10, U=0, E=0)
    2017/10/02 11:26:37 – getPcapFiles.0 – ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Unexpected error
    2017/10/02 11:26:37 – getPcapFiles.0 – ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : java.lang.NoClassDefFoundError: Could not initialize class org.jnetpcap.Pcap
    2017/10/02 11:26:37 – getPcapFiles.0 – at com.cornastone.pcap_reader.PcapFileReader.readPcapOfflineFiles(PcapFileReader.java:37)

    Like

  9. I am not sure where you are opening the “User Defined Java Class” Step is this in one of the data-integration files which folder?

    Like

  10. Thanks, Its running fine for spoon.

    But resulting into error when I run it on pentaho carte server. I have attached the screenshot of error. Its giving HTTP Status 500 – /pentaho-di/kettle/prepareExec/ – Internal Server Error.

    I have placed required libraries on pentaho server at /pdi_appl01/pentaho/pdi/server/data-integration-server/tomcat/lib and /pdi_appl01/pentaho/pdi/server/data-integration-server/tomcat/libext.
    Is there anything on server level that we need to change like you did in launcher.properties file.

    Can u please help me in this matter. That how to run it on Pentaho Carte Server ?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s