Require an Order instance for NonEmptyList's groupBy function by igstan · Pull Request #1964 · typelevel/cats

igstan · 2017-10-11T16:37:45Z

This addresses: #1959

I didn't see a way to require just an Eq instance for B while preserving the performance characteristics. A mutable.Map uses more than just universal equality, it also uses the built-in JVM hashing. So, what I did was to switch to a mutable.TreeMap which requires a scala.math.Ordering instance, and we get that by constraining B to have a cats.Order instance.

kailuowang · 2017-10-11T17:13:17Z

Hmmm, Order helps performance but is over strict. How about we use the newly added Hash which is what map uses anyway right?
we can use the hashcode from Hash for map keys.

johnynek · 2017-10-11T17:25:07Z

We could use Hash but we don’t yet have containers that use Hash.

kailuowang · 2017-10-11T17:30:33Z

@johnynek I suggested that we could use the hashcode from Hash as key in Map, the extra cost will be map calling the hashCode() on the int hashcode, WDYT?

johnynek · 2017-10-11T17:48:49Z

we have to return a Map[K, NonEmptyList[V]] right? So if we key by the hash, how do we return that? Do you want to implement a little wrapper that wraps Map[Int, List[(K, NonEmptyList[V])]] and then fetches it out?

This seems a bit complex to me, but it could work. So something like:

class MapFromHashBuckets[K, V](h: Hash[K], m: Map[Int, List[(K, NonEmptyList[V])]) extends Map[K, V] {
  def get(k: K): Option[NonEmptyList[V]] = m.get(h.hash(k)).flatMap {
    case Nil => None
    case candidates => candidates.collectFirst { case (k0, vs) if h.eqv(k, k0) => vs}
  }
}

then we just need to implement the other 3 methods of Map.

I guess it might not be so bad if we have tests. Probably better to use LongMap or something to avoid boxing the Int.

kailuowang · 2017-10-11T18:12:45Z

Update: Please disregard. the two maps below aren't consistent as @igstan pointed out.

~~Not sure if we need to return a wrapper, since there is already an intermediate map of Map[B, Builder] , just need to do the conversion in the last step, something like~~

def groupBy[B](f: A => B)(implicit B: Hash[B]): Map[B, NonEmptyList[A]] = {
    val m = mutable.LongMap.empty[(B, mutable.Builder[A, List[A]])]
    for { elem <- toList } {
     val b = f(elem)
      m.getOrElseUpdate(B.hash(b), (b, List.newBuilder[A]))._2 += elem
    }
    val b = immutable.Map.newBuilder[B, NonEmptyList[A]]
    for { (k, bs) <- m.values } {
      val head :: tail = bs.result // we only create non empty list inside of the map `m`
      b += (k, NonEmptyList(head, tail)))
    }
    b.result
  }

~~Not sure if this' going to be more performant than the wrapper, but I don't know how I feel about adding tests against a Map either.~~

igstan · 2017-10-11T18:31:17Z

I was going to propose the exact same thing that @kailuowang did in the last snippet. However, I'm thinking now that whatever constraint we're going to put on B (Hash or Order), it must be used for both maps we're producing, mutable and immutable. In the last snippet, the immutable Map uses implicitly universal equality and hashing. I don't think we want that, right?

kailuowang · 2017-10-11T18:36:16Z

@igstan ah you are right. I guess we have to return a wrapper or a new implementation of Map if we really want to do this.

igstan · 2017-10-11T18:37:52Z

@kailuowang I'm actually thinking of overriding elemHashCode for both kind of maps. That would allow us to use our own hashing function.

kailuowang · 2017-10-11T18:42:33Z

Sounds like we are moving into creating our own version of Map and MapBuilder just for this groupBy method. On one hand, these could be useful, on the other hand, to be honest, I am not sure if they should go to cats.core. stew/Dog already has a Map that uses Eq and Order. Maybe this groupBy method should be moved to Dog?

johnynek · 2017-10-11T18:47:25Z

why don't we do the Order based one and punt the Hash based one until we have a Map that uses Hash.

kailuowang · 2017-10-11T18:51:23Z

The Order based one isn't consistent either, the map it returns uses universal hashing and equality.

igstan · 2017-10-11T18:53:46Z

Right. I can change to an immutable TreeMap for that as well.

igstan · 2017-10-11T18:54:56Z

BTW, I've tried the overriding version and can't do it because immutable.HashMap is sealed.

This addresses: typelevel#1959

johnynek · 2017-10-11T19:17:28Z

Yeah, I was assuming we would use TreeMap

On Wed, Oct 11, 2017 at 08:55 Ionuț G. Stan ***@***.***> wrote: BTW, I've tried the overriding version and can't do it because immutable.HashMap is sealed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1964 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdpbD4KHtCQIRAV0zz-R6lKwEBsZBks5srQ8BgaJpZM4P1x49> .

-- P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin

igstan · 2017-10-11T19:20:16Z

I'm going to force-push the changes as soon as I finish running the tests locally.

igstan · 2017-10-11T19:28:31Z

Force-pushed.

igstan · 2017-10-12T10:36:37Z

Not sure why the build is failing, but seems to be a spurious one.

LukaJCB · 2017-10-12T10:39:42Z

There doesn't seem to be a mutable.TreeMap in Scala 2.10.x or 2.11.x.

igstan · 2017-10-12T10:43:09Z

Oh, silly me. I didn't see that error message when I looked earlier. Suddenly this "low-hanging fruit" bug fix seems to be very high :)

igstan · 2017-10-12T10:51:50Z

Ok, so here's a summary of the issues so far.

If we want to go ahead with Order, we can't for Scala < 2.12 as there's no mutable.TreeMap, but there is an immutable.TreeMap.

If we want to go ahead with Hash, we can't for Scala >= 2.12, as we can't override immutable.HashMap to use our own hashing function. But we can do this override for mutable.HashMap.

Not sure how to reconcile these. I'm out of ideas right now.

igstan · 2017-10-12T11:13:37Z

Well, we could of course drop the mutable.TreeMap and build the immutable map directly. Not sure how much that will affect performance, though.

kailuowang · 2017-10-12T11:20:40Z

That brings back my point of such map already exists in dog

igstan · 2017-10-12T11:22:37Z

Then maybe groupBy should be removed altogether from cats? And let people use toList if they want to groupBy on it?

codecov-io · 2017-10-12T11:43:02Z

Codecov Report

Merging #1964 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1964      +/-   ##
==========================================
+ Coverage   96.07%   96.08%   +<.01%     
==========================================
  Files         273      273              
  Lines        4539     4541       +2     
  Branches      122      119       -3     
==========================================
+ Hits         4361     4363       +2     
  Misses        178      178

Impacted Files	Coverage Δ
core/src/main/scala/cats/NonEmptyTraverse.scala	`100% <ø> (ø)`	⬆️
core/src/main/scala/cats/data/NonEmptyList.scala	`100% <100%> (ø)`	⬆️
core/src/main/scala/cats/syntax/list.scala	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46d0b17...2f0a165. Read the comment docs.

igstan · 2017-10-12T11:48:00Z

I've changed to use just an immutable TreeMap. I don't think we're losing anything with these changes. The previous version still had to functionally build an immutable map while traversing the mutable one. However, I kept the mutable List builder, so things should be even.

kailuowang · 2017-10-12T11:56:15Z

-    b.result
-  }

+    m.mapValues(v => NonEmptyList.fromListUnsafe(v.result))


IIRC, mapValues is by-name and full of surprises and not something you would ever want to use.

The alternative would be to construct another TreeMap. Calling map instead of mapValues wouldn't work because we'd have to call toMap on it at the end, and that uses the universal equality and hashing we're trying to get rid off.

What do you think?

BTW, I never had problems with mapValues, which is why I used it.

Another surprise of .mapValues is that the result is not Serializable.

Why don't you use Functor[Map].map(m)(v => NonEmptyList.fromListUnsafe(v.result))? :)

Erm, of course meant:

Functor[Map[B, ?]].map(m)(v => NonEmptyList.fromListUnsafe(v.result))

Ok, I think was wrong when I said we can't use map instead of mapValues. The result of TreeMap.map is a TreeMap as well, which means the ordering is preserved. I'll change to that.

I don't think we need to redundant Functor[Map[B, ?]] summoning here.

typelevel#1964

johnynek · 2017-10-12T17:13:33Z

-    b.result
-  }

+    m.map { case (k, v) => (k, NonEmptyList.fromListUnsafe(v.result)) }


can we ascribe a type to this to make sure we are not converting to using hashCode and equals at the end?

m.map { case (k, v) => (k, NonEmptyList.fromListUnsafe(v.result)) } : TreeMap[B, NonEmptyList[A]]

Sure. One further thought. Shall we also change the return type to SortedMap? That would explain why we require an Order instance for B.

kailuowang · 2017-10-12T17:17:48Z

    */
-  def groupBy[B](f: A => B): Map[B, NonEmptyList[A]] = {
-    val m = mutable.Map.empty[B, mutable.Builder[A, List[A]]]
+  def groupBy[B](f: A => B)(implicit B: Order[B]): Map[B, NonEmptyList[A]] = {


Can we add to the scaladoc a brief description of why Order is required?

johnynek · 2017-10-12T18:49:12Z

actually, yes, I think that's a good idea. It makes it clear the `Order` is being used and it allows callers to leverage that fact, which is nice. Incidentally, `SortedMap` is a pretty nice type for other reasons. It doesn't have some of the issues `Map` and `Set` have (see #1831)

…

On Thu, Oct 12, 2017 at 8:46 AM, Ionuț G. Stan ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In core/src/main/scala/cats/data/NonEmptyList.scala <#1964 (comment)>: > + m.map { case (k, v) => (k, NonEmptyList.fromListUnsafe(v.result)) } Sure. One further thought. Shall we also change the return type to SortedMap? That would explain why we require an Order instance for B. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1964 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdipmWVpWqDKlR_XbYKQKvv6r4fxfks5srl5rgaJpZM4P1x49> .

-- P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin

johnynek · 2017-10-12T21:19:19Z

    */
  def toNel: Option[NonEmptyList[A]] = NonEmptyList.fromList(la)
-  def groupByNel[B](f: A => B): Map[B, NonEmptyList[A]] =
+  def groupByNel[B : Order](f: A => B): Map[B, NonEmptyList[A]] =


shouldn't the type here also be SortedMap?

Yes, it should. Working on it.

Well... current status: yak-shaving inside NonEmptyTraverse doctests, where SortedMap needs an Apply instance.

I've pushed something that addresses this. I've fixed the NonEmptyTraverse doctests by upcasting to a vanilla Map for which there's an Apply instance. I'm not completely satisfied with this approach, but it's the best I've got right now.

I think making an Apply and Traversable for SortedMap is probably out of scope, so I think that's okay.

igstan force-pushed the nel-group-by-order branch from 2516f36 to f5397be Compare October 11, 2017 16:38

igstan changed the title ~~Require and Order instance for NonEmptyList's groupBy function~~ Require an Order instance for NonEmptyList's groupBy function Oct 11, 2017

kailuowang added the Source Breaking label Oct 11, 2017

Require an Order instance for NonEmptyList's groupBy function

685e949

This addresses: typelevel#1959

igstan force-pushed the nel-group-by-order branch from f5397be to 685e949 Compare October 11, 2017 19:27

There's no mutable TreeMap before Scala 2.12; use only immutable

24ff926

kailuowang reviewed Oct 12, 2017

View reviewed changes

Don't use mapValues, as per code review comments

9f116d6

typelevel#1964

LukaJCB previously approved these changes Oct 12, 2017

View reviewed changes

johnynek requested changes Oct 12, 2017

View reviewed changes

kailuowang reviewed Oct 12, 2017

View reviewed changes

Have NonEmptyList.groupBy return a SortedMap instead of just a Map

cace36c

igstan dismissed LukaJCB’s stale review via cace36c October 12, 2017 19:30

johnynek previously approved these changes Oct 12, 2017

View reviewed changes

kailuowang previously approved these changes Oct 12, 2017

View reviewed changes

johnynek reviewed Oct 12, 2017

View reviewed changes

Have List.groupByNel return a SortedMap as well

2f0a165

igstan dismissed stale reviews from kailuowang and johnynek via 2f0a165 October 12, 2017 22:22

johnynek approved these changes Oct 12, 2017

View reviewed changes

johnynek mentioned this pull request Oct 12, 2017

Add Apply and Traverse for SortedMap #1966

Closed

kailuowang approved these changes Oct 13, 2017

View reviewed changes

kailuowang merged commit c121f4f into typelevel:master Oct 13, 2017

kailuowang added this to the 1.0.0-RC1 milestone Oct 13, 2017

durban mentioned this pull request Oct 16, 2017

NonEmptyList#groupBy uses universal equality #1959

Closed

kailuowang added the enhancement label Dec 14, 2017

Uh oh!

Conversation

igstan commented Oct 11, 2017

Uh oh!

kailuowang commented Oct 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnynek commented Oct 11, 2017

Uh oh!

kailuowang commented Oct 11, 2017

Uh oh!

johnynek commented Oct 11, 2017

Uh oh!

kailuowang commented Oct 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

igstan commented Oct 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kailuowang commented Oct 11, 2017

Uh oh!

igstan commented Oct 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kailuowang commented Oct 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnynek commented Oct 11, 2017

Uh oh!

kailuowang commented Oct 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

igstan commented Oct 11, 2017

Uh oh!

igstan commented Oct 11, 2017

Uh oh!

johnynek commented Oct 11, 2017 via email

Uh oh!

igstan commented Oct 11, 2017

Uh oh!

igstan commented Oct 11, 2017

Uh oh!

igstan commented Oct 12, 2017

Uh oh!

LukaJCB commented Oct 12, 2017

Uh oh!

igstan commented Oct 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

igstan commented Oct 12, 2017

Uh oh!

igstan commented Oct 12, 2017

Uh oh!

kailuowang commented Oct 12, 2017

Uh oh!

igstan commented Oct 12, 2017

Uh oh!

codecov-io commented Oct 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

igstan commented Oct 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kailuowang commented Oct 11, 2017 •

edited

Loading

kailuowang commented Oct 11, 2017 •

edited

Loading

igstan commented Oct 11, 2017 •

edited

Loading

igstan commented Oct 11, 2017 •

edited

Loading

kailuowang commented Oct 11, 2017 •

edited

Loading

kailuowang commented Oct 11, 2017 •

edited

Loading

igstan commented Oct 12, 2017 •

edited

Loading

codecov-io commented Oct 12, 2017 •

edited

Loading

igstan commented Oct 12, 2017 •

edited

Loading