Visits: 4701
Running your MongoDB in replica set is a really great idea, using Docker is a quick way to deploy it.
The simplest way to create a multi region MongoDB Replica set using Docker Swarm is to use our scripts that are already written. They launch a CloudFormation and the scripts written below.
Please leave a comment below with any issues or thank yous.
https://aws.amazon.com/marketplace/pp/B01N9N0KFZ/
Then run the CloudFormation Script:
https://console.aws.amazon.com/cloudformation/
http://float.i.ng/replicated-mongodb-installation-instructions/
News for this project: Nov. 16 , 2017.
I have mostly completed the bash script and redacted it so that it can use Multiple AWS regions. I have also created a CloudFormation file to run this easily.
IN order to get distributed Mongo to talk to each other from their docker containers in other regions, the /etc/hosts file of each container needs to the real Public IP address.
I have been running the commands that are below with docker swarm to create a MongoDB Replica set. It builds incredibly fast on AWS on separate instances. The AMI number is not accessible, so replace it with the AMI of your desire. The default of Ubuntu doesn’t work so great. I have also added an XFS volume.
This solution is now a simple “one-click” solution in the AWS Marketplace, it might save you dozens of hours of work.
The script works on AWS if you give IAM priveges to the first instance. I used a custom Ubuntu, but will try with the plain AWS AMI.
Another issue that I encountered was that Docker Swarm could not get the replicas to communicate without opening all possible Docker and MongoDB Ports in the AWS EC2 Security Groups. Here is the settings that I used, You might be able to make it neater.
Type
|
Protocol
|
Port Range
|
Source
|
Custom TCP Rule
|
TCP (6)
|
2377
|
0.0.0.0/0
|
Custom TCP Rule
|
TCP (6)
|
2377
|
::/0
|
HTTP (80)
|
TCP (6)
|
80
|
0.0.0.0/0
|
HTTP (80)
|
TCP (6)
|
80
|
::/0
|
Custom TCP Rule
|
TCP (6)
|
4789
|
0.0.0.0/0
|
Custom TCP Rule
|
TCP (6)
|
4789
|
::/0
|
Custom TCP Rule
|
TCP (6)
|
8000
|
0.0.0.0/0
|
Custom TCP Rule
|
TCP (6)
|
8000
|
::/0
|
SSH (22)
|
TCP (6)
|
22
|
0.0.0.0/0
|
Custom UDP Rule
|
UDP (17)
|
7946
|
0.0.0.0/0
|
Custom UDP Rule
|
UDP (17)
|
7946
|
::/0
|
Custom TCP Rule
|
TCP (6)
|
2376
|
0.0.0.0/0
|
Custom TCP Rule
|
TCP (6)
|
27017-27019
|
0.0.0.0/0
|
Custom TCP Rule
|
TCP (6)
|
27017-27019
|
::/0
|
Custom TCP Rule
|
TCP (6)
|
7946
|
0.0.0.0/0
|
Custom TCP Rule
|
TCP (6)
|
7946
|
::/0
|
Custom UDP Rule
|
UDP (17)
|
4789
|
0.0.0.0/0
|
Custom UDP Rule
|
UDP (17)
|
4789
|
::/0
|
The issues still left open for this project include
- making a cloudformation script to work so that clients easily launch MongoDB with replica set.
- making it work with lots more security since MongoDB defaults to work without a password.
- making it store the MongoDB Database files in the pre-created XFS volume
- making it run in Multiple Regions and even multiple clouds since Docker Swarm can serve as the DNS.
Here is the script as of Oct 31 2017
#!/bin/bash set -v set -p #in Couldformation label the first one CharmingManager #fixed SEARCH RESULTS FOR: DISABLE TRANSPARENT HUGE PAGES (THP) — MONGODB MANUAL 3.4" Disable Transparent Huge Pages # delete keys that docker machine makes # when running the script it needs the AMI number to run #Directions - In order to connect the Primary use or the URL to put in your MOngo Client mongo mongodb://PublicDNS:27017/test #You can connect to any of the REplicas by accessing the Public IP of the Swarm Manager and changing the port XXXXXXX #Directions - In order to connect to the Read Replicas go to the charming1 or 2 Instances and use their PublicDNS for the following command or point your mongo client to the following URL mongo mongodb://PublicDNS:port/test for charming 1 use port 27018 for charming2 use port 27019 #Our installation has auto complete so to help with options for docker and docker-machine # create docker-machines #make the volume of first machine 20 gb default #DONT use cloudstor - use aws ec2 cli # add apt update and upgrade to scripts # think of adding docker-machine env to cloudformation userdata instead of whole complication with leaving the script on server. #Variable that will be needed are instance type , key , volume size. voliume type, IOPS needed. PRETTY MUCH LIKE cloudformation for docker for Aws. use https://docs.docker.com/machine/drivers/aws/#default-amis for vars fields #sleep 50 n=0 while [[ $n -lt 3 ]] do aws ec2 create-volume --size 5 --region us-east-1 --availability-zone us-east-1b --volume-type gp2 --tag-specifications 'ResourceType=volume,Tags=[{Key=charmingvol,Value='$n'}]' n=$((n+1)) done n=0 while [[ $n -lt 3 ]] do docker-machine create --driver amazonec2 --amazonec2-iam-instance-profile fullAccessEC2 --amazonec2-zone b --amazonec2-ami $1 charming$n n=$((n+1)) done sleep 40 # regenerate-certs since Ubuntu does its thing and misses Docker commands n=0 while [[ $n -lt 3 ]] do docker-machine regenerate-certs charming$n -f n=$((n+1)) done n=0 while [[ $n -lt 3 ]] do aws ec2 attach-volume --volume-id $(aws ec2 describe-volumes --filters Name=tag-key,Values="charmingvol" Name=tag-value,Values="$n" --region us-east-1 | sed -n 's/.*"VolumeId": "\(.*\)",/\1/p') --instance-id $( aws ec2 describe-instances --region us-east-1 --filters Name=instance-state-code,Values=16 Name=tag-key,Values="Name" Name=tag-value,Values="charming$n" | sed -n 's/.*"InstanceId": "\(.*\)",/\1/p') --device /dev/sdc --region us-east-1 n=$((n+1)) done echo "LABEL=cloudimg-rootfs / ext4 defaults,discard 0 0" > fstab echo "/dev/xvdc /mnt/charming xfs rw,user,auto 0 0" >> fstab n=0 while [[ $n -lt 3 ]] do docker-machine ssh charming$n sudo mkfs.xfs /dev/xvdc docker-machine ssh charming$n sudo mkdir /mnt/charming docker-machine scp fstab charming$n: docker-machine ssh charming$n sudo /bin/cp -f fstab /etc/fstab docker-machine ssh charming$n sudo mount -a n=$((n+1)) done #From create-mongo-replicas-7.sh # create swarm docker swarm init --listen-addr $(/bin/hostname -i):2377 --advertise-addr $(hostname -i):2377 # get token for joining in script Also make 1 instead of 0 # join swarm - n=0 while [[ $n -lt 1 ]] do export JOIN_TOKEN=`docker swarm join-token -q worker` n=$((n+1)) done n=0 while [[ $n -lt 3 ]] do eval `docker-machine env charming$n` docker swarm join --token $JOIN_TOKEN $(/bin/hostname -i):2377 n=$((n+1)) done # make network n=0 while [[ $n -lt 1 ]] do eval `docker-machine env -u` docker network create --driver overlay --internal charmingnet n=$((n+1)) done n=0 while [[ $n -lt 3 ]] do docker node update --label-add mongo.rpl=$n $(docker node ls -q -f name=charming$n) n=$((n+1)) done #run docker service - chANGE port number each time - change name too- #Following didnt work Oct - 19 -17 - could be issue with label n=0 while [[ $n -lt 3 ]] do docker service create --detach=false --network charmingnet --publish 270$((16+n+1)):27017 --mount type=bind,src=/mnt/charming,target=/data/db --constraint 'node.labels.mongo.rpl=='$n'' --name mongo$n mongo:3.4 mongod --replSet charming n=$((n+1)) done #Create Replicas in Mongo - will need number of replicas - skip this figure it out after running the rest of the set. eval `docker-machine env charming0` sleep 2 docker exec $(docker ps -q) mongo --eval 'rs.initiate({ _id: "charming", members: [{ _id: 0, host: "mongo0:27017" }, { _id: 1, host: "mongo1:27017" }, { _id: 2, host: "mongo2:27017" }], settings: { getLastErrorDefaults: { w: "majority", wtimeout: 30000 }}})'
This is simpler older script, they all require an AMI that has docker-machine
#!/bin/bash
#put upgrade back on!#remove volumes as well containers and images#volume didnt take up!#Deal with Mongo errors – lower down#VERY clean images, containers, etc
# The folowing works on AWS if you give IAM preliges to the first instance. I used a custom Ubuntu, but will try with the plain AWS AMI.
# create docker-machines
n=0 while [[ $n -lt 3 ]] do docker-machine create –driver amazonec2 –amazonec2-zone b –amazonec2-ami ami-7873be02 charming$n n=$((n+1)) done
sleep 40
# regenerate-certs since Ubuntu does its thing and misses Docker commands n=0 while [[ $n -lt 3 ]] do docker-machine regenerate-certs charming$n -f n=$((n+1))done
# create swarmdocker swarm init –listen-addr $(hostname -i):2377 –advertise-addr $(hostname -i):2377
# get token for joining in script Also make 1 instead of 0
JOIN_TOKEN=`docker swarm join-token -q worker`
#join swarm –
n=0 while [[ $n -lt 3 ]] do eval `docker-machine env charming$n` docker swarm join –token $JOIN_TOKEN $(hostname -i):2377 n=$((n+1)) done
#make networkeval `docker-machine env -u`docker network create –driver overlay –internal charmingnetn=0 while [[ $n -lt 3 ]] do docker node update –label-add mongo.rpl=$n $(docker node ls -q -f name=charming$n) n=$((n+1)) done
#run docker service – chANGE port number each time – change name too-
#Following didnt work Oct – 19 -17 – could be issue with label
n=0 while [[ $n -lt 3 ]] do docker service create –network charmingnet –publish 270$((16+n+1)):27017 –mount type=bind,src=/mnt/charming,target=/data/db –constraint ‘node.labels.mongo.rpl==’$n” –name mongo$n mongo:3.4 mongod –replSet charming n=$((n+1)) done
#Create Replicas in Mongo – will need number of replicas – skip this figure it out after running the rest of the set.# eval `docker-machine env charming0`
# docker exec -it $(docker ps -qf label=com.docker.swarm.service.name=mongo0) $(docker ps –format “{{.Names}}”) mongo –eval ‘rs.initiate({ _id: “charming”, members: [{ _id: 0, host: “mongo0:27017” }, { _id: 1, host: “mongo1:27017” }, { _id: 2, host: “mongo2:27017” }], settings: { getLastErrorDefaults: { w: “majority”, wtimeout: 30000 }}})’ exit
Also see:
- http://charmingwebdesign.com/bootstrapping-a-docker-swarm-mode-cluster-semaphore/
- http://charmingwebdesign.com/bash-how-do-i-undo-the-command-eval-docker-machine-env-blog-stack-overflow/
- http://charmingwebdesign.com/docker-swarm-intro-tutorial-httpcharming-co-ilaws-marketplacelabsreadme-md-at-master-%c2%b7-dockerlabs/
- http://charmingwebdesign.com/docker-basics-amazon-ec2-container-service-fun/
- https://eladnava.com/deploy-a-highly-available-mongodb-replica-set-on-aws/
This is yet another option to manage networking so that Replica containers can talk to each other.
It seems like there are 4 options
hard code it with hosts file on main manager server
use kubernetes to manage the DNS
Use Docker Swarm – the link below discusses using Swarm
Amazon ECS also has an option with an agent that configure route 53 DNS
Azure is starting to push their Container Service, but it is based on Open Source Kubernetes or Swarm
You can read about how this scripts is written.
Source: Running a MongoDB Replica Set on Docker 1.12 Swarm Mode: Step by Step
http://charmingwebdesign.com/running-a-mongodb-replica-set-on-docker-1-12-swarm-mode-step-by-step/
Relate Blog articles
http://charmingwebdesign.com/make-xfs-faield-mkfs-xfs-no-such-file-or-directory/
http://charmingwebdesign.com/kubernetes-running-mongodb-on-kubernetes-with-statefulsets/