Sqoop配置安裝 |
發(fā)布時間: 2012/8/27 17:21:08 |
Sqoop是一個用來將Hadoop和關系型數(shù)據(jù)庫中的數(shù)據(jù)相互轉移的工具,可以將一個關系型數(shù)據(jù)庫(例如 : MySQL ,Oracle ,Postgres等)中的數(shù)據(jù)導入到Hadoop的HDFS中,也可以將HDFS的數(shù)據(jù)導入到關系型數(shù)據(jù)庫中。 Sqoop的User Guide地址:http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html#_introduction 1:tar zxvf sqoop-1.1.0.tar.gz 2:修改配置文件 /home/Hadoopuser/sqoop-1.1.0/conf/sqoop-site.xml 一般只需要修改如下幾個項: sqoop.metastore.client.enable.autoconnect 3: bin/sqoop help bin/sqoop help import 4: [Hadoopuser@master sqoop-1.1.0]$ bin/sqoop import --connect jdbc:mysql://localhost/ppc --table data_ip --username kwps -P Enter password: 11/02/18 10:51:58 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not find appropriate Hadoop shim for 0.20.2 java.lang.RuntimeException: Could not find appropriate Hadoop shim for 0.20.2 at com.cloudera.sqoop.shims.ShimLoader.loadShim(ShimLoader.java:190) at com.cloudera.sqoop.shims.ShimLoader.getHadoopShim(ShimLoader.java:109) at com.cloudera.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:173) at com.cloudera.sqoop.tool.ImportTool.init(ImportTool.java:81) at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:411) at com.cloudera.sqoop.Sqoop.run(Sqoop.java:134) at org.apache.Hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.Hadoop.util.ToolRunner.run(ToolRunner.java:79) at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:170) at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:196) at com.cloudera.sqoop.Sqoop.main(Sqoop.java:205) 解決辦法: 默認情況下: ./Hadoop-0.20.2/conf/hadoop-env.sh # Extra Java runtime options. Empty by default. 需要更改成: export Hadoop_OPTS="-Djava.net.preferIPv4Stack=true -Dsqoop.shim.jar.dir=/home/hadoopuser/sqoop-1.1.0/shims" 特別需要注意的是: Sqoop目前在Apache 版本的Hadoop 0.20.2上是無法使用的。 目前只支持CDH 3 beta 2版本。所以如果想使用的話,得升級到 CDH 3 beta 2版本了。 “Sqoop does not run with Apache Hadoop 0.20.2. The only supported platform is CDH 3 beta 2. It requires features of MapReduce not available in the Apache 0.20.2 release of Hadoop. You should upgrade to CDH 3 beta 2 if you want to run Sqoop 1.0.0.” 這個問題 已經(jīng)被Cloudera 標記為 Major Bug,希望能盡快解決吧。
本文出自:億恩科技【mszdt.com】 |