marathon mesos zookeeper docker

Posted on Posted in docker

marathon mesos zookeeper docker

环境

Distributor ID: CentOS
Description:    CentOS Linux release 7.5.1804 (Core) 
Release:    7.5.1804
Codename:   Core

软件包

三主 n 从

  • master 10.10.8.11
  • master 10.10.8.12
  • master 10.10.8.13

防火墙

iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 2181 -j ACCEPT
iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 2888 -j ACCEPT
iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 38888 -j ACCEPT
iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 5050 -j ACCEPT
iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 5051 -j ACCEPT
iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT
iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 9090 -j ACCEPT

安装使用

zookeeper

master 上操作

[root@localhost zookeeper]# pwd
/home/data/services/zookeeper
[root@localhost zookeeper]# cp -rf conf/zoo_sample.cfg conf/zoo.cfg
[root@localhost zookeeper]# cat conf/zoo.cfg  | grep -v "#"
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/data/services/zookeeper/zoodata/data
dataLogDir=/home/data/services/zookeeper/zoodata/logs
clientPort=2181
server.1=10.10.8.11:2888:38888
server.2=10.10.8.12:2888:38888
server.3=10.10.8.13:2888:38888

3 台 master 上执行

[root@localhost zookeeper]# echo 1 > zoodata/data/myid 
[root@localhost zookeeper]# echo 2 > zoodata/data/myid 
[root@localhost zookeeper]# echo 3 > zoodata/data/myid 

3 台 master 启动

[root@localhost zookeeper]# ./bin/zkServer.sh start
[root@localhost zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/data/services/zookeeper/bin/../conf/zoo.cfg
Mode: follower

mesos 安装

https://mesos.apache.org/documentation/latest/building/ 安装相应系统的依赖

[root@localhost mesos-1.7.0]# pwd
/usr/local/src/mesos-1.7.0
[root@localhost mesos-1.7.0]# ./bootstrap && mkdir -p build && cd $_
[root@localhost mesos-1.7.0]# ../configure --prefix=/home/data/services/mesos && make -j4 V=0 && make install

启动主 mesos-master

/home/data/services/mesos/sbin/mesos-master --zk=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --port=5050 --log_dir=/var/log/mesos/master --quorum=2 --hostname=10.10.8.11 --ip=10.10.8.11 --work_dir=/var/lib/mesos/master --cluster=mesos
/home/data/services/mesos/sbin/mesos-master --zk=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --port=5050 --log_dir=/var/log/mesos/master --quorum=2 --hostname=10.10.8.12 --ip=10.10.8.12 --work_dir=/var/lib/mesos/master --cluster=mesos
/home/data/services/mesos/sbin/mesos-master --zk=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --port=5050 --log_dir=/var/log/mesos/master --quorum=2 --hostname=10.10.8.13 --ip=10.10.8.13 --work_dir=/var/lib/mesos/master --cluster=mesos

在所有 mesos-slave 上安装docker

docker 安装参考 https://docs.docker.com/install/linux/docker-ce/centos/

启动从 mesos-slave

/home/data/services/mesos/sbin/mesos-slave --master=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --log_dir=/var/log/mesos/slave --hostname=10.10.8.10 --ip=10.10.8.10 --work_dir=/home/data/services/mesos/data/slave --containerizers=docker,mesos
/home/data/services/mesos/sbin/mesos-slave --master=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --log_dir=/var/log/mesos/slave --hostname=10.10.8.11 --ip=10.10.8.11 --work_dir=/home/data/services/mesos/data/slave --containerizers=docker,mesos
/home/data/services/mesos/sbin/mesos-slave --master=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --log_dir=/var/log/mesos/slave --hostname=10.10.8.12 --ip=10.10.8.12 --work_dir=/home/data/services/mesos/data/slave --containerizers=docker,mesos
/home/data/services/mesos/sbin/mesos-slave --master=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --log_dir=/var/log/mesos/slave --hostname=10.10.8.13 --ip=10.10.8.13 --work_dir=/home/data/services/mesos/data/slave --containerizers=docker,mesos
/home/data/services/mesos/sbin/mesos-slave --master=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --log_dir=/var/log/mesos/slave --hostname=10.10.8.14 --ip=10.10.8.14 --work_dir=/home/data/services/mesos/data/slave --containerizers=docker,mesos
/home/data/services/mesos/sbin/mesos-slave --master=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --log_dir=/var/log/mesos/slave --hostname=10.10.8.15 --ip=10.10.8.15 --work_dir=/home/data/services/mesos/data/slave --containerizers=docker,mesos
/home/data/services/mesos/sbin/mesos-slave --master=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --log_dir=/var/log/mesos/slave --hostname=10.10.8.16 --ip=10.10.8.16 --work_dir=/home/data/services/mesos/data/slave --containerizers=docker,mesos
/home/data/services/mesos/sbin/mesos-slave --master=zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --log_dir=/var/log/mesos/slave --hostname=10.10.8.17 --ip=10.10.8.17 --work_dir=/home/data/services/mesos/data/slave --containerizers=docker,mesos

启动从 marathon

3 台 master 上执行

MESOS_NATIVE_JAVA_LIBRARY=/home/data/services/mesos/lib/libmesos.so /home/data/services/marathon/bin/marathon --hostname 10.10.8.11 --master zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --zk zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/marathon
MESOS_NATIVE_JAVA_LIBRARY=/home/data/services/mesos/lib/libmesos.so /home/data/services/marathon/bin/marathon --hostname 10.10.8.12 --master zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --zk zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/marathon
MESOS_NATIVE_JAVA_LIBRARY=/home/data/services/mesos/lib/libmesos.so /home/data/services/marathon/bin/marathon --hostname 10.10.8.13 --master zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/mesos --zk zk://10.10.8.11:2181,10.10.8.12:2181,10.10.8.13:2181/marathon

现有页面

  • http://10.10.8.11:5050
  • http://10.10.8.11:8080
  • http://10.10.8.11:5050/state mesos状态

marathon-lb 负载均衡

指定到某个主机或多台主机

"constraints": [["hostname", "CLUSTER", "10.10.8.17|10.10.8.18"]],

marathon-lb json

  • network 采用 HOST
  • args 后面是 marathonip
  • groupexternal
{
  "id":"marathon-lb",
  "cpus": 1,
  "mem": 128,
  "instances": 1,
  "constraints": [["hostname", "UNIQUE"]],
  "container": {
  "type":"DOCKER",
  "docker": {
     "image": "mesosphere/marathon-lb",
     "privileged": true,
     "network": "HOST"
    }
  },
    "args": [
    "sse",
    "-m",
    "http://10.10.8.11:8080",
    "-m",
    "http://10.10.8.12:8080",
    "-m",
    "http://10.10.8.13:8080",
    "--group",
    "external"
  ]
}

创建 docker marathon-lb

curl -i -H 'Content-Type: application/json' 10.10.8.11:8080/v2/apps -d@marathon-lb.json

nginx json

  • HAPROXY_GROUP 是上面 marathon-lbexternal
  • HAPROXY_0_VHOST 是标签名
  • instances 2个实例, 就是2个docker容器
  • containerPort 容器内部端口
  • hostPort 宿主机映射端口 0 表示随机
  • servicePort 表示访问 marathon-lb 的端口, 来访问 nginx, 0 表示随机
{
  "id":"web",
  "labels": {
     "HAPROXY_GROUP":"external",
     "HAPROXY_0_VHOST":"web.nginx.com"
  },
  "cpus":0.2,
  "mem":20.0,
  "instances": 2,
  "healthChecks": [{ "path": "/" }],
  "container": {
    "type":"DOCKER",
    "docker": {
     "image": "nginx",
     "network": "BRIDGE",
     "portMappings":[{"containerPort":80,"hostPort":0,"servicePort":0,"protocol":"tcp"}]
    }
  }
}

创建 nginx docker

curl -i -H 'Content-Type: application/json' 10.10.8.11:8080/v2/apps -d@nginx.json

访问

假设 marathon-lb 被分配到了 10.10.8.14

  • 监控页面:http://10.10.8.14:9090/haproxy?stats
  • csv格式的状态监控:http://10.10.8.14:9090/haproxy?stats;csv
  • 健康检查:http://10.10.8.14:9090/_haproxy_health_check
  • haproxy配置获取:http://10.10.8.14:9090/_haproxy_getconfig
  • haproxy的pid:http://10.10.8.14:9090/_haproxy_getpids

keepalive

把所有 marathon-lb 加到 keepalive

vim keepalived.conf

vrrp_script chk_marathonlb {
    script "/usr/local/keepalived/etc/chk_marathonlb.sh"
    interval 1
    weight 2          //脚本检测返回值为”0”时,权重上升
    rise 1
}
track_script {
        chk_marathonlb
    }

vim /usr/local/keepalived/etc/chk_marathonlb.sh

MARATHONLB_LOG="/tmp/marathon-lb.log"
MARATHONLB=`netstat -tunlp | grep "haproxy" | grep ":80" | awk '{print $4}' | awk  'BEGIN{FS=":"} {print $2}'`

#检测marathon-lb服务的80端口,若存在则返回”0”,不存在返回”1”;
#记录日志,非必须项
if [ ${MARATHONLB} -eq 80 ]; then
     echo -e "`date +%F\ %T`: Matathon-lb is here, exit!\n" >> $MARATHONLB_LOG
     exit 0       
else
     echo -e "`date +%F\ %T`: Matathon-lb is not here, the keepalived weight will be downgraded." >> $MARATHONLB_LOG
     exit 1
fi
» 转载请注明来源:呢喃 » marathon mesos zookeeper docker

Leave a Reply

Your email address will not be published. Required fields are marked *

2 × 4 =