MSDN Blogs: SCP.Net with HDInsight Linux Storm clusters

SCP.Net is now available on HDInsight Linux clusters 3.4 and above.

Versions

Note: HDInsight Storm team recommends HDI 3.5 clusters for users looking to migrate their SCP.Net topologies from Windows to Linux.

Development of SCP.Net Topology

Pre-Steps

Azure Datalake Tools for Visual Studio

HDInsight tools for Visual Studio does not support submission of SCP.Net topologies to HDI Linux Storm clusters.

The latest Azure Datalake Tools for Visual Studio is needed to develop and submit SCP.Net topologies to HDI Linux Storm clusters.

The tools are available for Visual Studio 2013 and 2015.

Please note: Azure DataLake tools has compatibility issues with other/older extensions to Visual Studio.

One known issue is where no clusters are shown in the drop-down for topology submission.

If you encounter this issue, please uninstall all extensions to Visual studio, and re-install Azure Datalake tools.

Java

SCP.Net generates a zip file consisting of the topology DLLs and dependency jars.

It uses Java (if found in the PATH) or .net to generate the zip. Unfortunately, zip files generated with .net are not compatible with Linux clusters.

Java installation requirements:

• Java should be installed on the machine (JDK1.7+) (example: C:JDK1.7)
• JAVA_HOME system variable should be set to the installation path. (C:JDK1.7)
• PATH system variable should include %JAVA_HOME%bin

One way to verify if your Java setup is good is to:

1. Launch a command window
2. execute: java -version
   The above command should output java usage.
3. Execute: jar
   The above command should output jar usage

Creating a SCP.Net topology

• Open visual studio, and the create new project dialog should have storm templates.

• Select Storm Project from Azure Datalake Templates

• For HDI 3.5, the SCP.Net nuget package needs to be updated to 1.0.0.1 version (or the latest available).

• After update the package.config should look as below

Submission of Topology

• Right click on the project and select Submit to Storm on HDInsight … option

• Choose the Storm Linux cluster from the drop down.
  
   

  Java File Path

  If you have java jar dependencies, you can include their full paths as a ; string.
  or
  you can use * to indicate all jars in a given directory.

Investigating Submission failures

 
Topology submissions can fail due to many reasons:

• Required java dependencies are not included
• Incompatible java jar dependencies. Example: Storm-eventhub-spouts-9.jar is incompatible with Storm 1.0.1. If you submit a jar with that dependency, topolopgy submission will fail.
• Duplicate names for topologies

Logs

Topology submission operations are logged into /var/log/hdinsight-scpwebapi/hdinsight-scpwebapi.out on the active head node.
Users can look into the above file on head node to identify causes for submission failures. (In case where the output from the tool is not helpful).