Mesos DNS server

One of the most important, and difficult, tasks during an app deployment is knowing the IP and port the different services are using to their respective jobs. We have to balance requests to a bunch of web server nodes which have to access to a containerized database and all this services has to send their logs to an analytic server, and we don't know the IP and ports of all this stuff.

Ok, calm down, there are some solutions to manage all this dynamic information.

• Distributed configuration systems as etcd or zookeeper let us know all the environment configuration, like IPs and ports of all services deployed. 
• Technologies like kubernetes uses its own proxy solution for services: kubernetes says us there is a service in a fixed IP and port and all the different services behind it are automatically manage
• We can even use haproxy in mesos to publish services in a fixed IP and port, such as kubernetes does with its proxy solution.
• Finally we can use a dynamically generated DNS services: Mesos DNS. This is what we are going to talk about

Mesos DNS

Mesos offers us a DNS server to save the name of all the services along its IP and service port. This is a diagram from the mesos-dns docs:

Mesos DNS gets all the information from mesos master, so can generate dynamically all the records. In case of a new application, services, change and forth on, this new state will be captured by mesos DNS with its consequent reconfiguration.

This DNS can be used to resolve names in our infrastructure and forward queries to external DNS servers for names outside our environment, so can be the primary DNS in all our nodes. Mesos recommends to deploy it as an app in a mesos slave to take advantage of the high availability and resources mesos offers us.

Another advantage of Mesos DNS it's implements its own and simple REST API to query it from our applications:

/v1/version: lists the Mesos-DNS version
/v1/config: lists the Mesos-DNS configuration info
/v1/hosts/{host}: lists the IP address of a host
/v1/services/{service}: lists the host, IP address, and port for a service

It's a simple and clean solution to manage all the applications within our environment and can be used mixing applications from marathon with other frameworks or from our applications through API.


Installation
We can use mesos DNS as a binary executable or a container

From binary
We must download it from github and getting tge right binary for our architecture: 

https://github.com/mesosphere/mesos-dns/releases 

In my case:

# archx86_64

# curl -L https://github.com/mesosphere/mesos-dns/releases/download/v0.5.1/mesos-dns-v0.5.1-linux-amd64 -o /usr/local/bin/mesos-dns

# chmod +x /usr/local/bin/mesos-dns

Now we must create the config file in, for example, /etc/mesos-dns/config.json with our custom content:

{

  "zk": "zk://172.31.27.67:2181/mesos",
  "masters": ["172.31.27.67:5050"],

  "refreshSeconds": 60,
  "ttl": 60,

  "domain": "mesos",

  "port": 53,

  "resolvers": ["8.8.8.8","4.4.4.4"],

  "timeout": 5,

  "httpon": true,

  "dnson": true,

  "httpport": 8123,

  "externalon": true,

  "SOAMname": "ns1.mesos",

  "SOARname": "root.ns1.mesos",

  "SOARefresh": 60,

  "SOARetry":   600,

  "SOAExpire":  86400,

  "SOAMinttl": 60,

  "IPSources": ["netinfo", "mesos", "host"]

}

And now we can launch it

# /usr/local/bin/mesos-dns -config /etc/mesos-dns/config.json &

From docker

We will use the same config file from the binary part and place it in /etc/mesos-dns/config.json as well.

Now we must pull and run the mesos-dns container

# sudo docker run --net=host -d -v "/etc/mesos-dns/config.json:/config.json" mesosphere/mesos-dns /mesos-dns -config /config.json

Testing Mesos DNS

Testing external resolution

# dig @127.0.0.1 www.google.com[..];; ANSWER SECTION:www.google.com. 299 IN A 74.125.136.147www.google.com. 299 IN A 74.125.136.99www.google.com. 299 IN A 74.125.136.106www.google.com. 299 IN A 74.125.136.105www.google.com. 299 IN A 74.125.136.103www.google.com. 299 IN A 74.125.136.104[..]

Testing internal resolution

# dig @127.0.0.1 master.mesos[..];; ANSWER SECTION:master.mesos. 60 IN A 172.31.27.67
[..]

Testing API

# curl -X GET http://127.0.0.1:8123/v1/hosts/master.mesos -H "Content-type: application/json"
[
  {
   "host": "master.mesos.",
   "ip": "172.31.27.67"
  }

If we launch a simple python's web server (python -m SimpleHTTPServer) in a 5 instances /webserver application, we get the following result:

# dig @127.0.0.1 _webserver._tcp.marathon.mesos SRV[..]
;; ANSWER SECTION:_webserver._tcp.marathon.mesos. 60 IN SRV 0 0 31940 webserver-ijjsn-s1.marathon.slave.mesos._webserver._tcp.marathon.mesos. 60 IN SRV 0 0 31446 webserver-66bp6-s1.marathon.slave.mesos._webserver._tcp.marathon.mesos. 60 IN SRV 0 0 31854 webserver-5gr6z-s0.marathon.slave.mesos._webserver._tcp.marathon.mesos. 60 IN SRV 0 0 31211 webserver-ihsjm-s0.marathon.slave.mesos._webserver._tcp.marathon.mesos. 60 IN SRV 0 0 31964 webserver-p6drh-s2.marathon.slave.mesos.
;; ADDITIONAL SECTION:webserver-5gr6z-s0.marathon.slave.mesos. 60 IN A 172.31.29.193webserver-ijjsn-s1.marathon.slave.mesos. 60 IN A 172.31.27.67webserver-ihsjm-s0.marathon.slave.mesos. 60 IN A 172.31.29.193webserver-66bp6-s1.marathon.slave.mesos. 60 IN A 172.31.27.67webserver-p6drh-s2.marathon.slave.mesos. 60 IN A 172.31.27.66
[..]

# curl -X GET http://127.0.0.1:8123/v1/services/_webserver._tcp.marathon.mesos -H "Content-type: application/json"

[

  {

   "service": "_webserver._tcp.marathon.mesos",   "host": "webserver-5gr6z-s0.marathon.slave.mesos.",   "ip": "172.31.29.193",   "port": "31854"  },  {   "service": "_webserver._tcp.marathon.mesos",   "host": "webserver-ijjsn-s1.marathon.slave.mesos.",   "ip": "172.31.27.67",   "port": "31940"  },  {   "service": "_webserver._tcp.marathon.mesos",   "host": "webserver-ihsjm-s0.marathon.slave.mesos.",   "ip": "172.31.29.193",   "port": "31211"  },  {   "service": "_webserver._tcp.marathon.mesos",   "host": "webserver-66bp6-s1.marathon.slave.mesos.",   "ip": "172.31.27.67",   "port": "31446"  },  {   "service": "_webserver._tcp.marathon.mesos",   "host": "webserver-p6drh-s2.marathon.slave.mesos.",   "ip": "172.31.27.66",   "port": "31964"  }

Clients

To use it we must change our /etc/resolv.conf file in the master (or masters) and slaves to point to the IP where our mesos dns server is now installed.

Using mesos DNS from marathon

Using constraints in marathon we can deploy mesos dns (using binary or container) in fixed servers which we now their IP. Using this way of deployment we can ensure the high availability of the service and not depend of so many human tasks, what it is, at the end, what is all this about.