大数据平台docker部署

fs.permissions.umask-modeCD0拜客生活常识网

000 修改022为000CD0拜客生活常识网

CD0拜客生活常识网

vim /etc/hadoop/conf/hdfs-site.xmlCD0拜客生活常识网

CD0拜客生活常识网

fs.permissions.umask-modeCD0拜客生活常识网

000 修改022为000CD0拜客生活常识网

CD0拜客生活常识网

CD0拜客生活常识网

CD0拜客生活常识网

登录配置节点http://192.168.1.156:7180 用户名 admin 密码 adminCD0拜客生活常识网

CD0拜客生活常识网

登录管理hdfs http://192.168.1.156:50070/CD0拜客生活常识网

CD0拜客生活常识网

CD0拜客生活常识网

python测试CD0拜客生活常识网

pip install hdfs 安装 hdfscli 包CD0拜客生活常识网

from hdfs.client import Client

client = Client("http://127.0.0.1:50070", root="/", timeout=100)

CD0拜客生活常识网

print(client.list("/"))

client.upload("/", "D:/Users/piperck/Desktop/About_me/dragen.wma")

CD0拜客生活常识网

查看    hadoop fs -ls /

CD0拜客生活常识网

sudo -u hdfs hadoop fs -mkdir /test

CD0拜客生活常识网

CDH测试   http://ip:7180

CD0拜客生活常识网

http://quickstart.cloudera:7180

    Username: cloudera

    Password: cloudera

CD0拜客生活常识网

docker扩容

https://www.cnblogs.com/HD/p/4807088.html

https://blog.csdn.net/qq_22083251/article/details/80417097  

https://blog.csdn.net/kwame211/article/details/76691003 

二、host模式启动docker    （docker与宿主机有相同IP）

以host模式启动镜像

docker run --name cdh --hostname=quickstart.cloudera --net=host --privileged=true -t -i -p 8020:8020 -p 8022:8022 -p 7180:7180 -p 21050:21050 -p 50070:50070 -p 50075:50075 -p 50010:50010 -p 50020:50020 -p 8890:8890 -p 60010:60010 -p 10002:10002 -p 25010:25010 -p 25020:25020 -p 18088:18088 -p 8088:8088 -p 19888:19888 -p 7187:7187 -p 44444:44444 -p 11000:11000 cloudera/quickstart /bin/bash -c '/usr/bin/docker-quickstart && /home/cloudera/cloudera-manager --express && service ntpd start'

如果启动报错：Starting cloudera-scm-agent:    [FAILED]

解决办法： /etc/hosts文件中加入   127.0.0.1 quickstart.cloudera

ps -ef|grep oracle-hdfs.conf |awk  '{print $2}'|xargs kill -9 

配置flume采集oracle数据至hdfs

vim /etc/flume-ng/conf

tier1.sources  = srctest

tier1.channels = chtest

tier1.sinks    = sinktest

tier1.sources.srctest.type = org.keedio.flume.source.SQLSource

tier1.sources.srctest.hibernate.connection.url = jdbc:oracle:thin:@10.12.4.145:1521/orcl

tier1.sources.srctest.hibernate.connection.user = dr

tier1.sources.srctest.hibernate.connection.password = root

tier1.sources.srctest.hibernate.connection.autocommit = true

tier1.sources.srctest.hibernate.dialect = org.hibernate.dialect.Oracle10gDialect

tier1.sources.srctest.hibernate.connection.driver_class = oracle.jdbc.driver.OracleDriver

tier1.sources.srctest.table = enterprise

tier1.sources.srctest.run.query.delay=10000

tier1.sources.srctest.status.file.path = /flume/oracle/flume-ng

tier1.sources.srctest.status.file.name = sqlSource.status

#tier1.sources.srctest.start.from = 0

#tier1.sources.srctest.custom.query = select * from enterprise

tier1.sources.srctest.batch.size = 1000