StrSearcher: Update substring search to use the Two Way algorithm

To improve our substring search performance, revive the two way searcher
and adapt it to the Pattern API.

Fixes #25483, a performance bug: that particular case now completes faster
in optimized rust than in ruby (but they share the same order of magnitude).

Much thanks to @gereeter who helped me understand the reverse case
better and wrote the comment explaining `next_back` in the code.

I had quickcheck to fuzz test forward and reverse searching thoroughly.

The two way searcher implements both forward and reverse search,
but not double ended search. The forward and reverse parts of the two
way searcher are completely independent.

The two way searcher algorithm has very small, constant space overhead,
requiring no dynamic allocation. Our implementation is relatively fast,
especially due to the `byteset` addition to the algorithm, which speeds
up many no-match cases.

A bad case for the two way algorithm is:

```
let haystack = (0..10_000).map(|_| "dac").collect::<String>();
let needle = (0..100).map(|_| "bac").collect::<String>());
```

For this particular case, two way is not much faster than the naive
implementation it replaces.

Author: Ulrik Sverdrup

src/libcollectionstest/str.rs

@@ -705,7 +705,7 @@ fn test_split_at() {
 #[should_panic]
 fn test_split_at_boundscheck() {
     let s = "ศไทย中华Việt Nam";
-    let (a, b) = s.split_at(1);
+    s.split_at(1);
 }
 
 #[test]
@@ -1820,6 +1820,14 @@ mod pattern {
         Match (4, 6),
         Reject(6, 7),
     ]);
+    make_test!(str_searcher_ascii_haystack_seq, "bb", "abbcbbbbd", [
+        Reject(0, 1),
+        Match (1, 3),
+        Reject(3, 4),
+        Match (4, 6),
+        Match (6, 8),
+        Reject(8, 9),
+    ]);
     make_test!(str_searcher_empty_needle_ascii_haystack, "", "abbcbbd", [
         Match (0, 0),
         Reject(0, 1),

src/libcore/str/mod.rs

@@ -15,13 +15,12 @@
 #![doc(primitive = "str")]
 #![stable(feature = "rust1", since = "1.0.0")]
 
-use self::OldSearcher::{TwoWay, TwoWayLong};
 use self::pattern::Pattern;
 use self::pattern::{Searcher, ReverseSearcher, DoubleEndedSearcher};
 
 use char::CharExt;
 use clone::Clone;
-use cmp::{self, Eq};
+use cmp::Eq;
 use convert::AsRef;
 use default::Default;
 use fmt;
@@ -33,7 +32,6 @@ use option::Option::{self, None, Some};
 use raw::{Repr, Slice};
 use result::Result::{self, Ok, Err};
 use slice::{self, SliceExt};
-use usize;
 
 pub mod pattern;
 
@@ -870,301 +868,6 @@ impl<'a> DoubleEndedIterator for LinesAny<'a> {
     }
 }
 
-/// The internal state of an iterator that searches for matches of a substring
-/// within a larger string using two-way search
-#[derive(Clone)]
-struct TwoWaySearcher {
-    // constants
-    crit_pos: usize,
-    period: usize,
-    byteset: u64,
-
-    // variables
-    position: usize,
-    memory: usize
-}
-
-/*
-    This is the Two-Way search algorithm, which was introduced in the paper:
-    Crochemore, M., Perrin, D., 1991, Two-way string-matching, Journal of the ACM 38(3):651-675.
-
-    Here's some background information.
-
-    A *word* is a string of symbols. The *length* of a word should be a familiar
-    notion, and here we denote it for any word x by |x|.
-    (We also allow for the possibility of the *empty word*, a word of length zero).
-
-    If x is any non-empty word, then an integer p with 0 < p <= |x| is said to be a
-    *period* for x iff for all i with 0 <= i <= |x| - p - 1, we have x[i] == x[i+p].
-    For example, both 1 and 2 are periods for the string "aa". As another example,
-    the only period of the string "abcd" is 4.
-
-    We denote by period(x) the *smallest* period of x (provided that x is non-empty).
-    This is always well-defined since every non-empty word x has at least one period,
-    |x|. We sometimes call this *the period* of x.
-
-    If u, v and x are words such that x = uv, where uv is the concatenation of u and
-    v, then we say that (u, v) is a *factorization* of x.
-
-    Let (u, v) be a factorization for a word x. Then if w is a non-empty word such
-    that both of the following hold
-
-      - either w is a suffix of u or u is a suffix of w
-      - either w is a prefix of v or v is a prefix of w
-
-    then w is said to be a *repetition* for the factorization (u, v).
-
-    Just to unpack this, there are four possibilities here. Let w = "abc". Then we
-    might have:
-
-      - w is a suffix of u and w is a prefix of v. ex: ("lolabc", "abcde")
-      - w is a suffix of u and v is a prefix of w. ex: ("lolabc", "ab")
-      - u is a suffix of w and w is a prefix of v. ex: ("bc", "abchi")
-      - u is a suffix of w and v is a prefix of w. ex: ("bc", "a")
-
-    Note that the word vu is a repetition for any factorization (u,v) of x = uv,
-    so every factorization has at least one repetition.
-
-    If x is a string and (u, v) is a factorization for x, then a *local period* for
-    (u, v) is an integer r such that there is some word w such that |w| = r and w is
-    a repetition for (u, v).
-
-    We denote by local_period(u, v) the smallest local period of (u, v). We sometimes
-    call this *the local period* of (u, v). Provided that x = uv is non-empty, this
-    is well-defined (because each non-empty word has at least one factorization, as
-    noted above).
-
-    It can be proven that the following is an equivalent definition of a local period
-    for a factorization (u, v): any positive integer r such that x[i] == x[i+r] for
-    all i such that |u| - r <= i <= |u| - 1 and such that both x[i] and x[i+r] are
-    defined. (i.e. i > 0 and i + r < |x|).
-
-    Using the above reformulation, it is easy to prove that
-
-        1 <= local_period(u, v) <= period(uv)
-
-    A factorization (u, v) of x such that local_period(u,v) = period(x) is called a
-    *critical factorization*.
-
-    The algorithm hinges on the following theorem, which is stated without proof:
-
-    **Critical Factorization Theorem** Any word x has at least one critical
-    factorization (u, v) such that |u| < period(x).
-
-    The purpose of maximal_suffix is to find such a critical factorization.
-
-*/
-impl TwoWaySearcher {
-    #[allow(dead_code)]
-    fn new(needle: &[u8]) -> TwoWaySearcher {
-        let (crit_pos_false, period_false) = TwoWaySearcher::maximal_suffix(needle, false);
-        let (crit_pos_true, period_true) = TwoWaySearcher::maximal_suffix(needle, true);
-
-        let (crit_pos, period) =
-            if crit_pos_false > crit_pos_true {
-                (crit_pos_false, period_false)
-            } else {
-                (crit_pos_true, period_true)
-            };
-
-        // This isn't in the original algorithm, as far as I'm aware.
-        let byteset = needle.iter()
-                            .fold(0, |a, &b| (1 << ((b & 0x3f) as usize)) | a);
-
-        // A particularly readable explanation of what's going on here can be found
-        // in Crochemore and Rytter's book "Text Algorithms", ch 13. Specifically
-        // see the code for "Algorithm CP" on p. 323.
-        //
-        // What's going on is we have some critical factorization (u, v) of the
-        // needle, and we want to determine whether u is a suffix of
-        // &v[..period]. If it is, we use "Algorithm CP1". Otherwise we use
-        // "Algorithm CP2", which is optimized for when the period of the needle
-        // is large.
-        if &needle[..crit_pos] == &needle[period.. period + crit_pos] {
-            TwoWaySearcher {
-                crit_pos: crit_pos,
-                period: period,
-                byteset: byteset,
-
-                position: 0,
-                memory: 0
-            }
-        } else {
-            TwoWaySearcher {
-                crit_pos: crit_pos,
-                period: cmp::max(crit_pos, needle.len() - crit_pos) + 1,
-                byteset: byteset,
-
-                position: 0,
-                memory: usize::MAX // Dummy value to signify that the period is long
-            }
-        }
-    }
-
-    // One of the main ideas of Two-Way is that we factorize the needle into
-    // two halves, (u, v), and begin trying to find v in the haystack by scanning
-    // left to right. If v matches, we try to match u by scanning right to left.
-    // How far we can jump when we encounter a mismatch is all based on the fact
-    // that (u, v) is a critical factorization for the needle.
-    #[inline]
-    fn next(&mut self, haystack: &[u8], needle: &[u8], long_period: bool)
-            -> Option<(usize, usize)> {
-        'search: loop {
-            // Check that we have room to search in
-            if self.position + needle.len() > haystack.len() {
-                return None;
-            }
-
-            // Quickly skip by large portions unrelated to our substring
-            if (self.byteset >>
-                    ((haystack[self.position + needle.len() - 1] & 0x3f)
-                     as usize)) & 1 == 0 {
-                self.position += needle.len();
-                if !long_period {
-                    self.memory = 0;
-                }
-                continue 'search;
-            }
-
-            // See if the right part of the needle matches
-            let start = if long_period { self.crit_pos }
-                        else { cmp::max(self.crit_pos, self.memory) };
-            for i in start..needle.len() {
-                if needle[i] != haystack[self.position + i] {
-                    self.position += i - self.crit_pos + 1;
-                    if !long_period {
-                        self.memory = 0;
-                    }
-                    continue 'search;
-                }
-            }
-
-            // See if the left part of the needle matches
-            let start = if long_period { 0 } else { self.memory };
-            for i in (start..self.crit_pos).rev() {
-                if needle[i] != haystack[self.position + i] {
-                    self.position += self.period;
-                    if !long_period {
-                        self.memory = needle.len() - self.period;
-                    }
-                    continue 'search;
-                }
-            }
-
-            // We have found a match!
-            let match_pos = self.position;
-            self.position += needle.len(); // add self.period for all matches
-            if !long_period {
-                self.memory = 0; // set to needle.len() - self.period for all matches
-            }
-            return Some((match_pos, match_pos + needle.len()));
-        }
-    }
-
-    // Computes a critical factorization (u, v) of `arr`.
-    // Specifically, returns (i, p), where i is the starting index of v in some
-    // critical factorization (u, v) and p = period(v)
-    #[inline]
-    #[allow(dead_code)]
-    #[allow(deprecated)]
-    fn maximal_suffix(arr: &[u8], reversed: bool) -> (usize, usize) {
-        let mut left: usize = !0; // Corresponds to i in the paper
-        let mut right = 0; // Corresponds to j in the paper
-        let mut offset = 1; // Corresponds to k in the paper
-        let mut period = 1; // Corresponds to p in the paper
-
-        while right + offset < arr.len() {
-            let a;
-            let b;
-            if reversed {
-                a = arr[left.wrapping_add(offset)];
-                b = arr[right + offset];
-            } else {
-                a = arr[right + offset];
-                b = arr[left.wrapping_add(offset)];
-            }
-            if a < b {
-                // Suffix is smaller, period is entire prefix so far.
-                right += offset;
-                offset = 1;
-                period = right.wrapping_sub(left);
-            } else if a == b {
-                // Advance through repetition of the current period.
-                if offset == period {
-                    right += offset;
-                    offset = 1;
-                } else {
-                    offset += 1;
-                }
-            } else {
-                // Suffix is larger, start over from current location.
-                left = right;
-                right += 1;
-                offset = 1;
-                period = 1;
-            }
-        }
-        (left.wrapping_add(1), period)
-    }
-}
-
-/// The internal state of an iterator that searches for matches of a substring
-/// within a larger string using a dynamically chosen search algorithm
-#[derive(Clone)]
-// NB: This is kept around for convenience because
-// it is planned to be used again in the future
-enum OldSearcher {
-    TwoWay(TwoWaySearcher),
-    TwoWayLong(TwoWaySearcher),
-}
-
-impl OldSearcher {
-    #[allow(dead_code)]
-    fn new(haystack: &[u8], needle: &[u8]) -> OldSearcher {
-        if needle.is_empty() {
-            // Handle specially
-            unimplemented!()
-        // FIXME: Tune this.
-        // FIXME(#16715): This unsigned integer addition will probably not
-        // overflow because that would mean that the memory almost solely
-        // consists of the needle. Needs #16715 to be formally fixed.
-        } else if needle.len() + 20 > haystack.len() {
-            // Use naive searcher
-            unimplemented!()
-        } else {
-            let searcher = TwoWaySearcher::new(needle);
-            if searcher.memory == usize::MAX { // If the period is long
-                TwoWayLong(searcher)
-            } else {
-                TwoWay(searcher)
-            }
-        }
-    }
-}
-
-#[derive(Clone)]
-// NB: This is kept around for convenience because
-// it is planned to be used again in the future
-struct OldMatchIndices<'a, 'b> {
-    // constants
-    haystack: &'a str,
-    needle: &'b str,
-    searcher: OldSearcher
-}
-
-impl<'a, 'b>  OldMatchIndices<'a, 'b> {
-    #[inline]
-    #[allow(dead_code)]
-    fn next(&mut self) -> Option<(usize, usize)> {
-        match self.searcher {
-            TwoWay(ref mut searcher)
-                => searcher.next(self.haystack.as_bytes(), self.needle.as_bytes(), false),
-            TwoWayLong(ref mut searcher)
-                => searcher.next(self.haystack.as_bytes(), self.needle.as_bytes(), true),
-        }
-    }
-}
-
 /*
 Section: Comparing strings
 */