marathon/mesos 以容器方式启动,访问私有仓库(https)报找不到镜像问题
为了方便快速部署,将mesos,marathon 进行了容器化部署, 但是容器化完后发现在marathon 上创建应用使用的是私有仓库镜像一直创建不成功
分析过程
mesos-slave 错误信息
E0106 17:17:58.250929 6 slave.cpp:4423] Container '1a75080c-6af6-47c8-8d15-9930197a52d6' for executor 'micadvisor.036cf537-d3f1-11e6-9da6-0242fc24b10b' of framework c52ff3c2-1675-472e-8ac2-b0151195a8ef-0000 failed to start: Failed to run 'docker -H unix:///var/run/docker.sock pull 10.135.183.13:5000/micadvisor:1.0': exited with status 1; stderr='Error: image micadvisor:1.0 not found
因为我们是容器化部署的mesos-slave和marathon,是不是因为没有找到证书和登录信息导致的,随后在mesos-salve容器手动的进行docker login 操作并将证书进行挂载到对应目录,手工执行命令docker -H unix:///var/run/docker.sock pull 10.135.183.13:5000/micadvisor:1.0 能正常下载私有仓库镜像,但通过marathon创建应用问题依旧,此时我们查看了一下镜像仓库日志发现一些踪迹
镜像仓库日志
time="2017-01-06T09:27:07Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry Realm\": invalid authorization credential" go.version=go1.6.3 http.request.host="10.135.183.13:5000" http.request.id=680e0d4a-1938-4de7-9c85-ab08010b6082 http.request.method=GET http.request.remoteaddr="10.104.239.20:46752" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/3.10.0-327.36.3.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=4b4e5b56-1bdf-4b9f-adc3-4a481bae568d version=v2.5.1
镜像仓库报错很明显,认证不通过,但我们明明在mesos-slave节点进行了验证操作,手工执行没有问题,通过查找marathon 官方文档,我们发现官方说明在使用Private Docker Registry 时候需要额外做一些配置,参考 ,配置简单的步骤如下,所有slave节点和marathon节点都需要配置
- 宿主机手工进行docker login
docker login
- 登录信息会保存在/root/.docker 目录下,将其进行打包
cd /roottar -cvf docker.tar.gz .docker/
- 将打包的文件复制到所有节点目录,注意: 放置在一个公共目录,因为marathon和mesos-slave都需要调用到
scp docker.tar.gz 所有节点:/一个公共目录
- 容器启动marathon 和mesos-slave时候进行挂载对应公共目录
#mesos-slavedocker run -v /tmp/docker.tar.gz:/tmp/docker.tar.gz ..... mesos-salve#marathondocker run -v /tmp/docker.tar.gz:/tmp/docker.tar.gz ..... marathon
- 创建应用时候,配置urls参数
"uris": [ "file:///tmp/docker.tar.gz" ]
应用启动后,我们登录到mesos-slave 容器上查看资源stderr日志可以发现,资源节点如何下载和使用docker.tar.gz 包
[root@dcos03 e87e4d7c-c86f-45e7-98cf-dcabbeb5da38]# cat stderrI0106 17:33:19.300180 935 fetcher.cpp:498] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/c52ff3c2-1675-472e-8ac2-b0151195a8ef-S6","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"executable":false,"extract":true,"value":"file:\/\/\/root\/.docker\/docker.tar.gz"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/c52ff3c2-1675-472e-8ac2-b0151195a8ef-S6\/frameworks\/c52ff3c2-1675-472e-8ac2-b0151195a8ef-0000\/executors\/micadvisor.28907cad-d3f3-11e6-9da6-0242fc24b10b\/runs\/e87e4d7c-c86f-45e7-98cf-dcabbeb5da38"}I0106 17:33:19.302255 935 fetcher.cpp:409] Fetching URI 'file:///root/.docker/docker.tar.gz'I0106 17:33:19.302276 935 fetcher.cpp:250] Fetching directly into the sandbox directoryI0106 17:33:19.302304 935 fetcher.cpp:187] Fetching URI 'file:///root/.docker/docker.tar.gz'I0106 17:33:19.302328 935 fetcher.cpp:167] Copying resource with command:cp '/root/.docker/docker.tar.gz' '/tmp/mesos/slaves/c52ff3c2-1675-472e-8ac2-b0151195a8ef-S6/frameworks/c52ff3c2-1675-472e-8ac2-b0151195a8ef-0000/executors/micadvisor.28907cad-d3f3-11e6-9da6-0242fc24b10b/runs/e87e4d7c-c86f-45e7-98cf-dcabbeb5da38/docker.tar.gz'I0106 17:33:19.304684 935 fetcher.cpp:84] Extracting with command: tar -C '/tmp/mesos/slaves/c52ff3c2-1675-472e-8ac2-b0151195a8ef-S6/frameworks/c52ff3c2-1675-472e-8ac2-b0151195a8ef-0000/executors/micadvisor.28907cad-d3f3-11e6-9da6-0242fc24b10b/runs/e87e4d7c-c86f-45e7-98cf-dcabbeb5da38' -xf '/tmp/mesos/slaves/c52ff3c2-1675-472e-8ac2-b0151195a8ef-S6/frameworks/c52ff3c2-1675-472e-8ac2-b0151195a8ef-0000/executors/micadvisor.28907cad-d3f3-11e6-9da6-0242fc24b10b/runs/e87e4d7c-c86f-45e7-98cf-dcabbeb5da38/docker.tar.gz'I0106 17:33:19.307724 935 fetcher.cpp:92] Extracted '/tmp/mesos/slaves/c52ff3c2-1675-472e-8ac2-b0151195a8ef-S6/frameworks/c52ff3c2-1675-472e-8ac2-b0151195a8ef-0000/executors/micadvisor.28907cad-d3f3-11e6-9da6-0242fc24b10b/runs/e87e4d7c-c86f-45e7-98cf-dcabbeb5da38/docker.tar.gz' into '/tmp/mesos/slaves/c52ff3c2-1675-472e-8ac2-b0151195a8ef-S6/frameworks/c52ff3c2-1675-472e-8ac2-b0151195a8ef-0000/executors/micadvisor.28907cad-d3f3-11e6-9da6-0242fc24b10b/runs/e87e4d7c-c86f-45e7-98cf-dcabbeb5da38'I0106 17:33:19.307751 935 fetcher.cpp:547] Fetched 'file:///root/.docker/docker.tar.gz' to '/tmp/mesos/slaves/c52ff3c2-1675-472e-8ac2-b0151195a8ef-S6/frameworks/c52ff3c2-1675-472e-8ac2-b0151195a8ef-0000/executors/micadvisor.28907cad-d3f3-11e6-9da6-0242fc24b10b/runs/e87e4d7c-c86f-45e7-98cf-dcabbeb5da38/docker.tar.gz' 原因分析
查看官方文档可以发现urls 定义是在cmd 命令执行之前进行一些资源加载操作,urls支持常见压缩包会自动进行解压,前面我们在容器上进行手工添加login信息和证书信息没有作用,应该是mesos-slave 容器默认没有引用/root/.docker 目录下登录配置导致报没有发现镜像问题,但测试rpm包安装mesos-slave 是不存在此问题,需要翻一下源码看一下实现