Visits: 1001

I need to change a username in mongodb 3.4 due to re-branding to floatingcloud.io

db.system.users.update({"user":"oldname"}, {$set:{"user":"newname"}})  From: https://stackoverflow.com/questions/35265277/how-to-rename-a-user-in-mongodb

Visits: 2650

In Windows PowerShell, run the following command to schedule the script to run as a Windows Scheduled Task. The script runs one time during the next boot and then disables these tasks from running again.

PS C:\> C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule

Also see from our Blog:

Prepare AMI for AWS Marketplace – Reset Linux ec2 Image – delete public keys, etc

https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html

Visits: 4525

Running your MongoDB in replica set is a really great idea, using Docker is a quick way to deploy it.

The simplest way to create a multi region MongoDB Replica set using Docker Swarm is to use our scripts that are already written. They launch a CloudFormation and the scripts written below.

Please leave a comment below with any issues or thank yous.

https://aws.amazon.com/marketplace/pp/B01N9N0KFZ/

Then run the CloudFormation Script:

https://console.aws.amazon.com/cloudformation/

http://float.i.ng/replicated-mongodb-installation-instructions/

News for this project: Nov. 16 , 2017.

I have mostly completed the  bash script and redacted it so that it can use Multiple AWS regions. I have also created a CloudFormation file to run this easily.

IN order to get distributed Mongo to talk to each other from their docker containers in other regions, the /etc/hosts file of each container needs to the real Public IP address.

I have been running the commands that are below with docker swarm  to create a  MongoDB Replica set. It builds incredibly fast on AWS on separate instances. The AMI number is not accessible, so replace it with the AMI of your desire. The default of Ubuntu doesn’t work so great. I have also added an XFS volume.

This solution is now a simple  “one-click” solution in the AWS Marketplace, it might save you dozens of hours of work.

The script works on AWS  if you give  IAM priveges to the first instance. I used a custom Ubuntu, but will try with the plain AWS AMI.

Another issue that I encountered was that Docker Swarm could not get the replicas to communicate without opening all possible Docker and MongoDB Ports in the AWS EC2 Security Groups. Here is the settings that I used, You might be able to make it neater.

The issues still left open for this project include

• making a cloudformation script to work so that  clients easily launch MongoDB with replica set.
• making it work with lots more security since MongoDB defaults to work without a password.
• making it store the MongoDB Database files  in the pre-created XFS volume
• making it run in Multiple Regions and even multiple clouds since Docker Swarm can serve as the DNS.

Here is the script as of Oct 31 2017

#!/bin/bash
set -v
set -p 
#in Couldformation label the first one CharmingManager
#fixed SEARCH RESULTS FOR:  DISABLE TRANSPARENT HUGE PAGES (THP) — MONGODB MANUAL 3.4"
Disable Transparent Huge Pages
# delete keys that docker machine makes
# when running the script it needs the AMI number to run 
#Directions - In order to connect the Primary use or the URL to put in your MOngo Client  mongo  mongodb://PublicDNS:27017/test
#You can connect to any of the REplicas by accessing the Public IP of the Swarm Manager and changing the port XXXXXXX
#Directions - In order to connect to the Read Replicas go to the charming1 or 2 Instances and use their PublicDNS for the following command or point your mongo client to the following URL mongo  mongodb://PublicDNS:port/test for charming 1 use port 27018 for charming2 use port 27019
#Our installation has auto complete so to help with options for docker and docker-machine 
# create docker-machines
#make the volume of first machine 20 gb default
#DONT use cloudstor - use aws ec2 cli

# add apt update and upgrade to scripts
# think of adding docker-machine env to cloudformation userdata instead of whole complication with leaving the script on server.
#Variable that will be needed are instance type , key , volume size. voliume type, IOPS needed. PRETTY MUCH LIKE cloudformation for docker for Aws. use https://docs.docker.com/machine/drivers/aws/#default-amis for vars fields


#sleep 50
n=0
	while [[ $n -lt 3 ]]
	do 
aws ec2 create-volume --size 5 --region us-east-1 --availability-zone us-east-1b   --volume-type gp2  --tag-specifications 'ResourceType=volume,Tags=[{Key=charmingvol,Value='$n'}]'
n=$((n+1))
 done
 


n=0
	while [[ $n -lt 3 ]]
	do docker-machine create --driver amazonec2   --amazonec2-iam-instance-profile fullAccessEC2 --amazonec2-zone b  --amazonec2-ami  $1  charming$n 
	n=$((n+1))
 done


sleep 40
# regenerate-certs since Ubuntu does its thing and misses Docker commands
	n=0
	while [[ $n -lt 3 ]]
	do docker-machine regenerate-certs charming$n -f
	n=$((n+1))
done 

 
 n=0
	while [[ $n -lt 3 ]]
	do aws ec2 attach-volume --volume-id $(aws ec2 describe-volumes --filters Name=tag-key,Values="charmingvol" Name=tag-value,Values="$n"  --region us-east-1 | sed -n 's/.*"VolumeId": "\(.*\)",/\1/p') --instance-id $( aws ec2 describe-instances  --region us-east-1 --filters Name=instance-state-code,Values=16  Name=tag-key,Values="Name" Name=tag-value,Values="charming$n" | sed -n 's/.*"InstanceId": "\(.*\)",/\1/p')  --device /dev/sdc  --region us-east-1
	n=$((n+1))
 done
 
  echo "LABEL=cloudimg-rootfs   /        ext4   defaults,discard        0 0" > fstab
echo "/dev/xvdc /mnt/charming xfs rw,user,auto 0 0" >> fstab

 n=0
	while [[ $n -lt 3 ]]
	do 
    docker-machine ssh charming$n  sudo mkfs.xfs /dev/xvdc 
	docker-machine ssh charming$n sudo mkdir /mnt/charming	
	docker-machine scp fstab charming$n:
	docker-machine ssh charming$n sudo /bin/cp -f fstab /etc/fstab
    docker-machine ssh charming$n sudo mount -a 
	n=$((n+1))
 done
 
 #From create-mongo-replicas-7.sh
 
 # create swarm
docker swarm init --listen-addr $(/bin/hostname -i):2377 --advertise-addr $(hostname -i):2377
# get token for joining in script Also make 1 instead of 0
# join swarm - 
 n=0
	while [[ $n -lt 1 ]]
do export JOIN_TOKEN=`docker swarm join-token -q worker`
n=$((n+1))
 done
 n=0
	while [[ $n -lt 3 ]]
	do eval `docker-machine env charming$n`
	docker swarm join --token $JOIN_TOKEN $(/bin/hostname -i):2377
	n=$((n+1))
 done
# make network
 n=0
	while [[ $n -lt 1 ]]
       do eval `docker-machine env -u`
       docker network create --driver overlay --internal charmingnet
n=$((n+1))
 done
n=0
	while [[ $n -lt 3 ]]
	do docker node update --label-add mongo.rpl=$n $(docker node ls -q -f name=charming$n)
	n=$((n+1))
 done
#run docker service  - chANGE port number each time - change name too- 
#Following didnt work Oct - 19 -17 - could be issue with label
n=0
	while [[ $n -lt 3 ]]
	do docker service create --detach=false --network charmingnet --publish 270$((16+n+1)):27017  --mount type=bind,src=/mnt/charming,target=/data/db  --constraint 'node.labels.mongo.rpl=='$n'' --name mongo$n mongo:3.4  mongod   --replSet charming
	n=$((n+1))
 done
#Create Replicas in Mongo - will need number of replicas - skip this figure it out after running the rest of the set.
eval `docker-machine env charming0`
sleep 2
 docker exec  $(docker ps -q) mongo  --eval 'rs.initiate({ _id: "charming", members: [{ _id: 0, host: "mongo0:27017" }, { _id: 1, host: "mongo1:27017" }, { _id: 2, host: "mongo2:27017" }], settings: { getLastErrorDefaults: { w: "majority", wtimeout: 30000 }}})'

This is simpler older script, they all require an AMI that has docker-machine

#!/bin/bash

#put upgrade back on!#remove volumes as well containers and images#volume didnt take up!#Deal with Mongo errors – lower down#VERY clean images, containers, etc
# The folowing works on AWS  if you give  IAM preliges to the first instance. I used a custom Ubuntu, but will try with the plain AWS AMI.
# create docker-machines
n=0 while [[ $n -lt 3 ]] do docker-machine create –driver amazonec2 –amazonec2-zone b  –amazonec2-ami ami-7873be02 charming$n n=$((n+1)) done
sleep 40
# regenerate-certs since Ubuntu does its thing and misses Docker commands n=0 while [[ $n -lt 3 ]] do docker-machine regenerate-certs charming$n -f  n=$((n+1))done

 

# create swarmdocker swarm init –listen-addr $(hostname -i):2377 –advertise-addr $(hostname -i):2377
# get token for joining in script Also make 1 instead of 0

 

JOIN_TOKEN=`docker swarm join-token -q worker`

#join swarm –

n=0 while [[ $n -lt 3 ]] do eval `docker-machine env charming$n` docker swarm join –token $JOIN_TOKEN $(hostname -i):2377 n=$((n+1)) done
#make networkeval `docker-machine env -u`docker network create –driver overlay –internal charmingnet

n=0 while [[ $n -lt 3 ]] do docker node update –label-add mongo.rpl=$n $(docker node ls -q -f name=charming$n) n=$((n+1)) done
#run docker service  – chANGE port number each time – change name too-
#Following didnt work Oct – 19 -17 – could be issue with label
n=0 while [[ $n -lt 3 ]] do docker service create –network charmingnet –publish 270$((16+n+1)):27017  –mount type=bind,src=/mnt/charming,target=/data/db  –constraint ‘node.labels.mongo.rpl==’$n” –name mongo$n mongo:3.4  mongod   –replSet charming n=$((n+1)) done

 

#Create Replicas in Mongo – will need number of replicas – skip this figure it out after running the rest of the set.# eval `docker-machine env charming0`
# docker exec -it $(docker ps -qf label=com.docker.swarm.service.name=mongo0) $(docker ps –format “{{.Names}}”) mongo  –eval ‘rs.initiate({ _id: “charming”, members: [{ _id: 0, host: “mongo0:27017” }, { _id: 1, host: “mongo1:27017” }, { _id: 2, host: “mongo2:27017” }], settings: { getLastErrorDefaults: { w: “majority”, wtimeout: 30000 }}})’ exit

Also see:

•  http://charmingwebdesign.com/bootstrapping-a-docker-swarm-mode-cluster-semaphore/
• http://charmingwebdesign.com/bash-how-do-i-undo-the-command-eval-docker-machine-env-blog-stack-overflow/
• http://charmingwebdesign.com/docker-swarm-intro-tutorial-httpcharming-co-ilaws-marketplacelabsreadme-md-at-master-%c2%b7-dockerlabs/
• http://charmingwebdesign.com/docker-basics-amazon-ec2-container-service-fun/
• https://eladnava.com/deploy-a-highly-available-mongodb-replica-set-on-aws/

This is yet another option to manage networking so that Replica containers can talk to each other.

It seems like there are 4 options

hard code it with hosts file on main manager server

use kubernetes to manage the DNS

Use Docker Swarm – the link below discusses using Swarm

Amazon ECS  also has an option with an agent that configure route 53 DNS

Azure is starting to push their Container Service, but it is based on Open Source Kubernetes or Swarm

You can read about how this scripts is written.

Source: Running a MongoDB Replica Set on Docker 1.12 Swarm Mode: Step by Step

http://charmingwebdesign.com/running-a-mongodb-replica-set-on-docker-1-12-swarm-mode-step-by-step/

Relate Blog articles

http://charmingwebdesign.com/make-xfs-faield-mkfs-xfs-no-such-file-or-directory/

http://charmingwebdesign.com/kubernetes-running-mongodb-on-kubernetes-with-statefulsets/

Visits: 5104

Postfix Email Servers can filter or Reject Spam

Open Source spam solution

This is the simplest solution for rejecting 90% of spam with no false positives For your Postfix Email Server.

smtpd_recipient_restrictions =
permit_sasl_authenticated,
reject_non_fqdn_recipient,
reject_unknown_recipient_domain,
permit_mynetworks,
reject_unauth_destination,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client bl.spamcop.net

This makes your Postfix email server reject mail from servers that are known to send spam. Spamhaus and spamcop aren’t self righteous.

Make your life easy and perform a simple installation method, Install Postfix Email Server with Webmail, Https Cert, Easy User Admin

Also see

https://rspamd.com/doc/quickstart.html

https://www.vultr.com/docs/simple-mailserver-postfix-dovecot-sieve-centos-7

https://www.vultr.com/docs/simple-mailserver-postfix-dovecot-sieve-debian

More ideas for getting rid of Spam on your Postfix Email Server

I am using

Amazon Linux 2
dovecot 2.2.33.2
spamassassin 3.4.1
amavisd

The goal is to run an e-mail server, which I achieved already. I can access the e-mails at the server using Thunderbird and the imap protocol.

For the postfix configuration I followed this tutorial: https://help.ubuntu.com/community/PostfixBasicSetupHowto but using Maildir instead.

Dovecot got configured following:

https://www.linuxbabe.com/mail-server/secure-email-server-ubuntu-16-04-postfix-dovecothttps://help.ubuntu.com/community/Dovecot

In addition I installed fail2ban, which got tested successfully.

The next step is e-mail filtering. Following https://help.ubuntu.com/lts/serverguide/mail-filtering.html.en worked out nicely. Spamassassin is blocking all spam. But acutally I do not want to block it, I just want that spamassassin marks it as spam and that the spam gets redirected into my spam-folder. This is just in case something gets filtered out that was not a spam.

For that I set /etc/amavis/conf.d/21-ubuntu_defaults:

$final_spam_destiny = D_PASS;

and the subject gets added ****SPAM****

The next step is that dovecot automatically moves this mail to my junk folder. And there I get stuck. I followed this tutorial: https://workaround.org/ispmail/stretch/filtering-out-spam-with-rspamd

and there the part “Sending spam to the Junk folder”. But it doesn’t work. I have seen that sieve is not working for imap. But I cannot find any tutorial or manual on imap_sieve, that would solve my problem. Does anyone of you has an idea? I also do not find any log entry where I could see that sieve is working (or not)?

down vote

I got a bit further in my problem:

By setting conf.d/10-logging.conf

mail_debug = yes

and conf.d/90-sieve.conf

sieve_plugins = sieve_imapsieve sieve_global_dir = /etc/dovecot/sieve/`

as well as in:

conf.d/90-plugin.conf `

plugin { sieve_plugins = sieve_imapsieve sieve_extprograms imapsieve_mailbox1_name = INBOX imapsieve_mailbox1_after = file:/etc/dovecot/sieve/default.sieve sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment }

and sieve/default.sieve: `

require [“fileinto”, “mailbox”, “imap4flags”]; if header :contains “subject” [“*SPAM*”] { setflag “\\Seen”; #setflag “\\Deleted”; fileinto :create “Junk”; stop; }

as well as compiling by sievec sieve/default.sieve I made it that junk mail gets copied into the junk folder. But unfortunately the mail stays as well in the Inbox. It seems it is just copied?

Visits: 1143

Install Horde Web Mail to check your email

1. Configuring Horde for IMP [OPTIONAL]If you would prefer that your users authenticate directly with IMP, without having to authenticate through Horde first, load
2. Roundcube with full postfix email server is also extremely easy to intall
3. theAdministration/Configuration/Authentication page and from the What backend should we use for authenticating users to Hordepulldown menu select Let a Horde application handle authentication (see the note below about new installs). Select imp from the The application which is providing authentication pulldown menu.NoteYou will have to log in twice if you don’t do this — Once to Horde and a second time to IMP.NoteIf this is a new install, you will not be able to configure IMP using the Horde Administration/Configuration page if you first enabled IMP authentication for Horde. You must set Horde to use another authentication method (refer to the horde/docs/INSTALL file), configure IMP, then reset Horde to use IMP authentication. One way to reset Horde in order to reach the Administration page is to replace the Horde configuration file conf.php with the original in horde/config/conf.php.dist. You should of course back up your current settings since they will otherwise be permanently lost.

1. Configuring IMPYou must login to Horde as a Horde Administrator to finish the configuration of IMP. Use the Horde Administration menu item to get to the administration page, and then click on the Configuration icon to get the configuration page. Select Mail from the selection list of applications. Fill in or change any configuration values as needed. When done click on Generate Mail Configuration to generate the conf.php file. If your web server doesn’t have write permissions to the IMP configuration directory or file, it will not be able to write the file. In this case, go back to Configuration and choose one of the other methods to create the configuration file imp/config/conf.php.Documentation on the format and purpose of the other configuration files in the config/ directory can be found in each file. You may create *.local.phpversions of these files if you wish to customize IMP’s appearance and behavior. See the header of the configuration files for details and examples. The defaults will be correct for most sites.
   • By default, IMP is configured to NOT display text/html message parts inline. This is done for various security reasons. If you would like to see text/html parts inline, you must create a imp/config/mime_drivers.local.php file (or add to the existing mime_drivers.local.php file) with the following content:<?php $mime_drivers[‘html’][‘inline’] = true;
2. Creating the database tablesOnce you finished the configuration in the previous step, you can create all database tables by clicking the DB schema is out of date link in the IMP row of the configuration screen.Alternatively, creating the IMP database tables can be accomplished with horde’s horde-db-migrate utility. If your database is properly setup in the Horde configuration, you can run the following command:horde/bin/horde-db-migrate imp
3. Securing IMPBefore you can secure IMP, you need a secure Horde installation. Please read the file in horde/docs/SECURITY for Horde security information before proceeding.Unless steps are taken to avoid it, there are two channels by which IMP can cause users to pass their IMAP/POP3 passwords across the network unencrypted.The first channel is between the browser and the Web server. We strongly recommend using an SSL-capable Web server to give users the option of encrypting communications between their browser and the Web server on which IMP is running. Some sites may wish to disable non-SSL access entirely.The second channel is between the Web server and their IMAP or POP3 server. The simplest way to avoid this is to have the mail server running on the same system as the Web server, and configuring IMP to connect to the IMAP or POP3 server on localhost instead of on the Internet hostname. In cases where that is not possible, it is highly recommended that the mail server be located on a private, secure network. Alternatively, the mail server can be accessed via TLS to ensure that users’ passwords remain safe after they have entrusted them to IMP (this is the default configuration).Other security steps you can take to increase security include:
   • Use session cookies instead of URL based sessions.
   • Set your php session.entropy_length to a larger value (e.g. 16) and session.entropy_file to a random source (e.g. /dev/urandom)
   • If your database, mail server, and web server are on the same host machine, then:
     • Use unix socket database access and disable TCP database access.
     • Use localhost for all TCP/IP connections to avoid the network, or run all services on a local, private network.
4. Testing IMPOnce you have configured IMP, bring up the Horde test page in your Web browser to ensure that all necessary prerequisites have been met. See thehorde/docs/INSTALL document for further details on the Horde test script.The test script will also allow you to test your connection to the mail server and provide some auto-detected configuration parameters that can be used to configure the mail server in imp/config/backends.local.php.Next, use IMP to login to a known working IMAP or POP3 server. Test at least the following:
   • Sending mail (via the Compose item in the menu bar).
   • Setting preferences (check to see if they survive after logging out and back in, if you are using an SQL or LDAP preferences system).
   • Reading mail.
   • Deleting mail.
   • Flagging mail (if using IMAP).
   • Changing mailboxes (if using IMAP).
5. Tuning IMP (Performance)See docs/PERFORMANCE.