Merge Development (#79)

Includes:

* Reduced memory in bulk operations (#56)
* Native x509 authentication (#55)
* Better connection recovery (#69)
* Example usage (#75 and #78)

Thanks to:

* @bachue
* @csucu 
* @feliixx


---
[Throughput overview](https://user-images.githubusercontent.com/9275968/34954403-3d3253dc-fa18-11e7-8eef-0f2b0f21edc3.png)

Select throughput has increased by ~600 requests/second with slightly increased variance:
```
x => r2017.11.06-select-zipfian-throughput.log 
y => 9acbd68-select-zipfian-throughput.log 

     n   min   max median  average   stddev      p99
x 3600 49246 71368  66542 66517.26 2327.675 70927.01
y 3600 53304 72005  67151 67145.36 2448.534 71630.00

          62000       64000       66000        68000       70000       72000    
 |----------+-----------+-----------+------------+-----------+-----------+-----|
                              +---------+--------+                              
1          -------------------|         |        |--------------------          
                              +---------+--------+                              
                                 +---------+---------+                          
2    ----------------------------|         |         |--------------------      
                                 +---------+---------+                          
Legend: 1=data$x, 2=data$y

At 95% probablitiy:
===> average is statistically significant     (p=0.000000, diff ~628.094444)
===> variance is statistically significant    (p=0.002398)

```

* [insert-latency.txt](https://github.com/globalsign/mgo/files/1632474/insert-latency.txt)
* [insert-throughput.txt](https://github.com/globalsign/mgo/files/1632475/insert-throughput.txt)
* [select-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632476/select-zipfian-latency.txt)
* [select-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632477/select-zipfian-throughput.txt)
* [update-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632478/update-zipfian-latency.txt)
* [update-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632479/update-zipfian-throughput.txt)

Note: latencies are approximations calculated from grouped data

Author: domodwyer

.travis.yml

@@ -2,27 +2,23 @@ language: go
 
 go_import_path: github.com/globalsign/mgo
 
+go:
+    - 1.8.x
+    - 1.9.x
+
 env:
     global:
         - BUCKET=https://s3.eu-west-2.amazonaws.com/globalsign-mgo
+        - FASTDL=https://fastdl.mongodb.org/linux
     matrix:
-        - GO=1.7   MONGODB=x86_64-2.6.11
-        - GO=1.8.x MONGODB=x86_64-2.6.11
-        - GO=1.7   MONGODB=x86_64-3.0.9
-        - GO=1.8.x MONGODB=x86_64-3.0.9
-        - GO=1.7   MONGODB=x86_64-3.2.3-nojournal
-        - GO=1.8.x MONGODB=x86_64-3.2.3-nojournal
-        - GO=1.7   MONGODB=x86_64-3.2.12
-        - GO=1.8.x MONGODB=x86_64-3.2.12
-        - GO=1.7   MONGODB=x86_64-3.2.16
-        - GO=1.8.x MONGODB=x86_64-3.2.16
-        - GO=1.7   MONGODB=x86_64-3.4.8
-        - GO=1.8.x MONGODB=x86_64-3.4.8
+        - MONGODB=x86_64-ubuntu1404-3.0.15
+        - MONGODB=x86_64-ubuntu1404-3.2.17
+        - MONGODB=x86_64-ubuntu1404-3.4.10
+        - MONGODB=x86_64-ubuntu1404-3.6.0
 
 install:
-    - eval "$(gimme $GO)"
 
-    - wget $BUCKET/mongodb-linux-$MONGODB.tgz
+    - wget $FASTDL/mongodb-linux-$MONGODB.tgz
     - tar xzvf mongodb-linux-$MONGODB.tgz
     - export PATH=$PWD/mongodb-linux-$MONGODB/bin:$PATH
 

README.md

@@ -11,7 +11,7 @@ Further PR's (with tests) are welcome, but please maintain backwards compatibili
 * Fixes attempting to authenticate before every query ([details](https://github.com/go-mgo/mgo/issues/254))
 * Removes bulk update / delete batch size limitations ([details](https://github.com/go-mgo/mgo/issues/288))
 * Adds native support for `time.Duration` marshalling ([details](https://github.com/go-mgo/mgo/pull/373))
-* Reduce memory footprint / garbage collection pressure by reusing buffers ([details](https://github.com/go-mgo/mgo/pull/229))
+* Reduce memory footprint / garbage collection pressure by reusing buffers ([details](https://github.com/go-mgo/mgo/pull/229), [more](https://github.com/globalsign/mgo/pull/56))
 * Support majority read concerns ([details](https://github.com/globalsign/mgo/pull/2))
 * Improved connection handling ([details](https://github.com/globalsign/mgo/pull/5))
 * Hides SASL warnings ([details](https://github.com/globalsign/mgo/pull/7))
@@ -32,10 +32,13 @@ Further PR's (with tests) are welcome, but please maintain backwards compatibili
 * GetBSON correctly handles structs with both fields and pointers ([details](https://github.com/globalsign/mgo/pull/40))
 * Improved bson.Raw unmarshalling performance ([details](https://github.com/globalsign/mgo/pull/49))
 * Minimise socket connection timeouts due to excessive locking ([details](https://github.com/globalsign/mgo/pull/52))
+* Natively support X509 client authentication ([details](https://github.com/globalsign/mgo/pull/55))
+* Gracefully recover from a temporarily unreachable server ([details](https://github.com/globalsign/mgo/pull/69))
 
 ---
 
 ### Thanks to
+* @bachue
 * @bozaro
 * @BenLubar
 * @carter2000

auth_test.go

@@ -28,6 +28,7 @@ package mgo_test
 
 import (
 	"crypto/tls"
+	"crypto/x509"
 	"flag"
 	"fmt"
 	"io/ioutil"
@@ -38,7 +39,7 @@ import (
 	"sync"
 	"time"
 
-	mgo "github.com/globalsign/mgo"
+	"github.com/globalsign/mgo"
 	. "gopkg.in/check.v1"
 )
 
@@ -963,6 +964,73 @@ func (s *S) TestAuthX509Cred(c *C) {
 	c.Assert(len(names) > 0, Equals, true)
 }
 
+func (s *S) TestAuthX509CredRDNConstruction(c *C) {
+	session, err := mgo.Dial("localhost:40001")
+	c.Assert(err, IsNil)
+	defer session.Close()
+	binfo, err := session.BuildInfo()
+	c.Assert(err, IsNil)
+	if binfo.OpenSSLVersion == "" {
+		c.Skip("server does not support SSL")
+	}
+
+	clientCertPEM, err := ioutil.ReadFile("harness/certs/client.pem")
+	c.Assert(err, IsNil)
+
+	clientCert, err := tls.X509KeyPair(clientCertPEM, clientCertPEM)
+	c.Assert(err, IsNil)
+
+	clientCert.Leaf, err = x509.ParseCertificate(clientCert.Certificate[0])
+	c.Assert(err, IsNil)
+
+	tlsConfig := &tls.Config{
+		InsecureSkipVerify: true,
+		Certificates:       []tls.Certificate{clientCert},
+	}
+
+	var host = "localhost:40003"
+	c.Logf("Connecting to %s...", host)
+	session, err = mgo.DialWithInfo(&mgo.DialInfo{
+		Addrs: []string{host},
+		DialServer: func(addr *mgo.ServerAddr) (net.Conn, error) {
+			return tls.Dial("tcp", addr.String(), tlsConfig)
+		},
+	})
+	c.Assert(err, IsNil)
+	defer session.Close()
+
+	cred := &mgo.Credential{
+		Username:    "root",
+		Mechanism:   "MONGODB-X509",
+		Source:      "$external",
+		Certificate: tlsConfig.Certificates[0].Leaf,
+	}
+	err = session.Login(cred)
+	c.Assert(err, NotNil)
+
+	err = session.Login(&mgo.Credential{Username: "root", Password: "rapadura"})
+	c.Assert(err, IsNil)
+
+	// This needs to be kept in sync with client.pem
+	x509Subject := "CN=localhost,OU=Client,O=MGO,L=MGO,ST=MGO,C=GO"
+
+	externalDB := session.DB("$external")
+	var x509User = mgo.User{
+		Username:     x509Subject,
+		OtherDBRoles: map[string][]mgo.Role{"admin": {mgo.RoleRoot}},
+	}
+	err = externalDB.UpsertUser(&x509User)
+	c.Assert(err, IsNil)
+
+	session.LogoutAll()
+
+	cred.Username = ""
+	c.Logf("Authenticating...")
+	err = session.Login(cred)
+	c.Assert(err, IsNil)
+	c.Logf("Authenticated!")
+}
+
 var (
 	plainFlag = flag.String("plain", "", "Host to test PLAIN authentication against (depends on custom environment)")
 	plainUser = "einstein"

bulk.go

@@ -3,6 +3,7 @@ package mgo
 import (
 	"bytes"
 	"sort"
+	"sync"
 
 	"github.com/globalsign/mgo/bson"
 )
@@ -118,6 +119,15 @@ func (e *BulkError) Cases() []BulkErrorCase {
 	return e.ecases
 }
 
+var actionPool = sync.Pool{
+	New: func() interface{} {
+		return &bulkAction{
+			docs: make([]interface{}, 0),
+			idxs: make([]int, 0),
+		}
+	},
+}
+
 // Bulk returns a value to prepare the execution of a bulk operation.
 func (c *Collection) Bulk() *Bulk {
 	return &Bulk{c: c, ordered: true}
@@ -145,7 +155,9 @@ func (b *Bulk) action(op bulkOp, opcount int) *bulkAction {
 		}
 	}
 	if action == nil {
-		b.actions = append(b.actions, bulkAction{op: op})
+		a := actionPool.Get().(*bulkAction)
+		a.op = op
+		b.actions = append(b.actions, *a)
 		action = &b.actions[len(b.actions)-1]
 	}
 	for i := 0; i < opcount; i++ {
@@ -288,6 +300,9 @@ func (b *Bulk) Run() (*BulkResult, error) {
 		default:
 			panic("unknown bulk operation")
 		}
+		action.idxs = action.idxs[0:0]
+		action.docs = action.docs[0:0]
+		actionPool.Put(action)
 		if !ok {
 			failed = true
 			if b.ordered {

cluster_test.go

@@ -2083,8 +2083,8 @@ func (s *S) TestDoNotFallbackToMonotonic(c *C) {
 	if !s.versionAtLeast(3, 0) {
 		c.Skip("command-counting logic depends on 3.0+")
 	}
-	if s.versionAtLeast(3, 4) {
-		c.Skip("failing on 3.4+")
+	if s.versionAtLeast(3, 2, 17) {
+		c.Skip("failing on 3.2.17+")
 	}
 
 	session, err := mgo.Dial("localhost:40012")

example_test.go

@@ -0,0 +1,136 @@
+package mgo
+
+import (
+	"crypto/tls"
+	"crypto/x509"
+	"io/ioutil"
+	"net"
+	"sync"
+)
+
+func ExampleCredential_x509Authentication() {
+	// MongoDB follows RFC2253 for the ordering of the DN - if the order is
+	// incorrect when creating the user in Mongo, the client will not be able to
+	// connect.
+	//
+	// The best way to generate the DN with the correct ordering is with
+	// openssl:
+	//
+	// 		openssl x509 -in client.crt -inform PEM -noout -subject -nameopt RFC2253
+	// 		subject= CN=Example App,OU=MongoDB Client Authentication,O=GlobalSign,C=GB
+	//
+	//
+	// And then create the user in MongoDB with the above DN:
+	//
+	//		db.getSiblingDB("$external").runCommand({
+	//			createUser: "CN=Example App,OU=MongoDB Client Authentication,O=GlobalSign,C=GB",
+	//			roles: [
+	//				{ role: 'readWrite', db: 'bananas' },
+	//				{ role: 'userAdminAnyDatabase', db: 'admin' }
+	//			],
+	//			writeConcern: { w: "majority" , wtimeout: 5000 }
+	//		})
+	//
+	//
+	// References:
+	// 		- https://docs.mongodb.com/manual/tutorial/configure-x509-client-authentication/
+	// 		- https://docs.mongodb.com/manual/core/security-x.509/
+	//
+
+	// Read in the PEM encoded X509 certificate.
+	//
+	// See the client.pem file at the path below.
+	clientCertPEM, err := ioutil.ReadFile("harness/certs/client.pem")
+
+	// Read in the PEM encoded private key.
+	clientKeyPEM, err := ioutil.ReadFile("harness/certs/client.key")
+
+	// Parse the private key, and the public key contained within the
+	// certificate.
+	clientCert, err := tls.X509KeyPair(clientCertPEM, clientKeyPEM)
+
+	// Parse the actual certificate data
+	clientCert.Leaf, err = x509.ParseCertificate(clientCert.Certificate[0])
+
+	// Use the cert to set up a TLS connection to Mongo
+	tlsConfig := &tls.Config{
+		Certificates: []tls.Certificate{clientCert},
+
+		// This is set to true so the example works within the test
+		// environment.
+		//
+		// DO NOT set InsecureSkipVerify to true in a production
+		// environment - if you use an untrusted CA/have your own, load
+		// its certificate into the RootCAs value instead.
+		//
+		// RootCAs: myCAChain,
+		InsecureSkipVerify: true,
+	}
+
+	// Connect to Mongo using TLS
+	host := "localhost:40003"
+	session, err := DialWithInfo(&DialInfo{
+		Addrs: []string{host},
+		DialServer: func(addr *ServerAddr) (net.Conn, error) {
+			return tls.Dial("tcp", host, tlsConfig)
+		},
+	})
+
+	// Authenticate using the certificate
+	cred := &Credential{Certificate: tlsConfig.Certificates[0].Leaf}
+	if err := session.Login(cred); err != nil {
+		panic(err)
+	}
+
+	// Done! Use mgo as normal from here.
+	//
+	// You should actually check the error code at each step.
+	_ = err
+}
+
+func ExampleSession_concurrency() {
+	// This example shows the best practise for concurrent use of a mgo session.
+	// 
+	// Internally mgo maintains a connection pool, dialling new connections as
+	// required. 
+	// 
+	// Some general suggestions:
+	// 		- Define a struct holding the original session, database name and 
+	// 			collection name instead of passing them explicitly.
+	// 		- Define an interface abstracting your data access instead of exposing
+	// 			mgo to your application code directly.
+	// 		- Limit concurrency at the application level, not with SetPoolLimit().
+
+	// This will be our concurrent worker
+	var doStuff = func(wg *sync.WaitGroup, session *Session) {
+		defer wg.Done()
+
+		// Copy the session - if needed this will dial a new connection which
+		// can later be reused.
+		// 
+		// Calling close returns the connection to the pool.
+		conn := session.Copy()
+		defer conn.Close()
+
+		// Do something(s) with the connection
+		_, _ = conn.DB("").C("my_data").Count()
+	}
+
+	///////////////////////////////////////////////
+
+	// Dial a connection to Mongo - this creates the connection pool
+	session, err := Dial("localhost:40003/my_database")
+	if err != nil {
+		panic(err)
+	}
+
+	// Concurrently do things, passing the session to the worker
+	wg := &sync.WaitGroup{}
+	for i := 0; i < 10; i++ {
+		wg.Add(1)
+		go doStuff(wg, session)
+	}
+	wg.Wait()
+
+	session.Close()
+}

harness/daemons/.env

@@ -22,10 +22,8 @@ versionAtLeast() {
 
 COMMONDOPTSNOIP="
 	--nohttpinterface
-	--noprealloc
 	--nojournal
 	--smallfiles
-	--nssize=1
 	--oplogSize=1
 	--dbpath ./db
 	"
@@ -55,14 +53,12 @@ if versionAtLeast 3 2; then
     # 3.2 doesn't like --nojournal on config servers.
 	COMMONCOPTS="$(echo "$COMMONCOPTS" | sed '/--nojournal/d')"
 
-
     if versionAtLeast 3 4; then
       # http interface is disabled by default, this option does not exist anymore
       COMMONDOPTSNOIP="$(echo "$COMMONDOPTSNOIP" | sed '/--nohttpinterface/d')"
       COMMONDOPTS="$(echo "$COMMONDOPTS" | sed '/--nohttpinterface/d')"
       COMMONCOPTS="$(echo "$COMMONCOPTS" | sed '/--nohttpinterface/d')"
 
-
       # config server need to be started as replica set 
       CFG1OPTS="--replSet conf1"
       CFG2OPTS="--replSet conf2"
@@ -71,12 +67,6 @@ if versionAtLeast 3 2; then
       MONGOS1OPTS="--configdb conf1/127.0.0.1:40101"
       MONGOS2OPTS="--configdb conf2/127.0.0.1:40102"
       MONGOS3OPTS="--configdb conf3/127.0.0.1:40103" 
-    else 
-
-	  # Go back to MMAPv1 so it's not super sluggish. :-(
-	  COMMONDOPTSNOIP="--storageEngine=mmapv1 $COMMONDOPTSNOIP"
-	  COMMONDOPTS="--storageEngine=mmapv1 $COMMONDOPTS"
-	  COMMONCOPTS="--storageEngine=mmapv1 $COMMONCOPTS"
 	fi
 fi
 

harness/daemons/db2/run

@@ -3,6 +3,5 @@
 . ../.env
 
 exec mongod $COMMONDOPTS \
-	--shardsvr \
 	--port 40002 \
 	--auth

harness/daemons/db3/run

@@ -3,7 +3,6 @@
 . ../.env
 
 exec mongod $COMMONDOPTS \
-	--shardsvr \
 	--port 40003 \
 	--auth \
 	--sslMode preferSSL \