Add Enumerable.SkipLast, TakeLast to get the last N elements of a sequence.

[jamesqo]

Background

Currently, Linq does not offer an API for getting the last N elements of a sequence. This has resulted in no less than 5 StackOverflow questions (with ~300 votes) popping up as the first result when you Google "skip last n elements of enumerable."

Proposal

We should add SkipLast and TakeLast to Enumerable, which will skip/take the last N elements of the sequence, respectively. If there are less than N elements, SkipLast will return an empty sequence, and TakeLast will return a sequence with contents equivalent to the original enumerable.

namespace System.Linq
{
    public static class Enumerable
    {
        public static IEnumerable<TSource> SkipLast<TSource>(this IEnumerable<TSource> source, int count);
        public static IEnumerable<TSource> TakeLast<TSource>(this IEnumerable<TSource> source, int count);
    }
    public static class Queryable
    {
        // Lots of existing methods for parity with Enumerable
   
        public static IQueryable<T> SkipLast<T>(this IQueryable<T> source, int count);
        public static IQueryable<T> TakeLast<T>(this IQueryable<T> source, int count);
    }
}

Remarks

• In SkipLast instead of evaluating the whole thing at once, we will read in the first count items into a circular buffer. Then we will interleave between yield returning the oldest element & overwriting that with a new one.

• In TakeLast we will still evaluate the whole sequence during the first iteration, but again maintain a circular buffer of length count and overwrite the oldest elements with newer ones.

Original (incorrect) - Implementation remarks

These overloads will have subtly different semantics from Skip and Take. Since for lazy enumerables we can't determine the count of the sequence in advance, we will have to capture it into an array during our first iteration (like Reverse does today), then skip/take the last N items from that array between yields.

    public static IEnumerable<TSource> SkipLast<TSource>(this IEnumerable<TSource> source, int count)
    {
        var array = source.ToArray();

        for (int i = count; i < array.Length; i++)
        {
            yield return array[i];
        }
    }

As a result, these will have different outputs:

int[] a = { 1, 2, 3, 4, 5 };

foreach (var item in a.Skip(3))
{
    a[4] = 10;
    Console.WriteLine(item); // 4 10
}

a[4] = 5;

foreach (var item in a.TakeLast(2))
{
    a[4] = 10;
    Console.WriteLine(item); // 4 5
}

Another consequence is that this will lead to more allocations for lists/arrays. Perhaps this is worth it for added convenience, though.

[jamesqo]

cc @JonHanna, @svick, @VSadov, @justinvp

[JonHanna]

The difference in outputs isn't too big a concern IMO, especially since variations in behaviour along similar lines already exist.

I've wanted a TakeLast() in the past, though it wasn't too arduous in the circumstances to just ToList() and then take the chunk I needed, it's always nice to have a convenient and (if possible) efficient form to hand. SkipLast() would at the very least be analogous (I'm of the opinion that things that seem like gaps in an API are as much in need of justification as additions are).

The most memory-efficient approach to TakeLast(4) would only buffer 4 elements. The most time-efficient would buffer the lot and hence not need to do any copying. Having two buffers of 4 elements seems a reasonable balance between the two, as large enough that no copying would be needed, but then immediately opting for that approach would mean that TakeLast(10000) could create a massive buffer even if there were only a small number of elements in the source, so two growing buffers would be needed to avoid that.

[jamesqo]

@JonHanna

The most memory-efficient approach to TakeLast(4) would only buffer 4 elements. The most time-efficient would buffer the lot and hence not need to do any copying.

I don't think we would need to do any copying with a buffer of 4 elements, actually. We could just implement it as a circular buffer of length N that automatically overwrote the most stale items, if I understand correctly. You're right though that we may need to do some dynamic growth in case count is really large.

[svick]

@JonHanna I don't understand, why would buffering only 4 elements require copying? The way I imagine the implementation, it would use List<T> in two phases. Phase one would be filling the list, phase two would treat it as circular buffer. At the end, you would yield items from the list (starting from the right position). Is there any copying in there that I'm missing?

SkipLast could be implemented almost the same way, except it would yield while iterating in phase two, not after iterating like TakeLast.

In code:

public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> source, int count)
{
	var buffer = new List<T>();
	int pos = 0;
	
	foreach (var item in source)
	{
		if (buffer.Count < count)
		{
			// phase 1
			buffer.Add(item);
		}
		else
		{
			// phase 2
			buffer[pos] = item;
			pos = (pos+1) % count;
		}
	}
	
	for (int i = 0; i < buffer.Count; i++)
	{
		yield return buffer[pos];
		pos = (pos+1) % count;
	}
}

public static IEnumerable<T> SkipLast<T>(this IEnumerable<T> source, int count)
{
	var buffer = new List<T>();
	int pos = 0;
	
	foreach (var item in source)
	{
		if (buffer.Count < count)
		{
			// phase 1
			buffer.Add(item);
		}
		else
		{
			// phase 2
			yield return buffer[pos];
			buffer[pos] = item;
			pos = (pos+1) % count;
		}
	}
}

[JonHanna]

Yes, we could be circular. You are both correct.

[jamesqo]

@karelz, maybe this could be marked ready-for-review? I don't think this would be a very controversial API; as I showed in the description there are many questions on StackOverflow this would solve.

[karelz]

@VSadov @OmarTawfik are in the middle of Linq area triage (see areas), they will get to it ...

[jamesqo]

@karelz 👍 Thanks for linking me to that document, I did not know about it.

[karelz]

Docs discoverability and cleanup is on my list for December :)

[terrajobst]

The API looks good as proposed. @stephentoub reminded us that Queue<T> already uses a circular buffer strategy, so implementing this wouldn't be too hard.

[JonHanna]

I'd like to see Queryable keep its parity with Enumerable on any method that doesn't materialise a collection (since that drags things into memory anyway) or attach elements like Append.

[JonHanna]

Right, I wasn't going to try typing this on my phone. I think we should have something like this:

public static IQueryable<T> SkipLast<T>(this IQueryable<T> source, int count)
{
    if (source == null)
    {
        throw Error.ArgumentNull(nameof(source));
    }

    EnumerableQuery eq = source as EnumerableQuery;
    if (eq != null)
    {
        // Call optimised version for in-memory collections.
        EnumerableQuery<T> eqt = source as EnumerableQuery<T>;
        if (eqt != null)
        {
            return new EnumerableQuery<T>(eqt.AsEnumerable().SkipLast(count));
        }

        // EnumerableQuery of a different element type, passed covariantly. Call for the actual type.
        return (IQueryable<T>) new Func<IQueryable<T>, int, IQueryable<T>>(SkipLast).GetMethodInfo()
            .GetGenericMethodDefinition()
            .MakeGenericMethod(source.ElementType)
            .Invoke(null, new object[] {source, count});
    }

    // Create a query that does a SkipLast. QueryProviders may recongise the sequence and optimise further
    return source.Reverse().Skip(count).Reverse();
}

public static IQueryable<T> TakeLast<T>(this IQueryable<T> source, int count)
{
    if (source == null)
    {
        throw Error.ArgumentNull(nameof(source));
    }

    EnumerableQuery eq = source as EnumerableQuery;
    if (eq != null)
    {
        // Call optimised version for in-memory collections.
        EnumerableQuery<T> eqt = source as EnumerableQuery<T>;
        if (eqt != null)
        {
            return new EnumerableQuery<T>(eqt.AsEnumerable().TakeLast(count));
        }

        // EnumerableQuery of a different element type, passed covariantly. Call for the actual type.
        return (IQueryable<T>)new Func<IQueryable<T>, int, IQueryable<T>>(SkipLast).GetMethodInfo()
            .GetGenericMethodDefinition()
            .MakeGenericMethod(source.ElementType)
            .Invoke(null, new object[] {source, count});
    }

    // Create a query that does a TakeLast. QueryProviders may recongise the sequence and optimise further
    return source.Reverse().Take(count).Reverse();
}

[karelz]

Do we have a static type for extension methods where we should put these?
Should we track these APIs as separate API proposal (given that we already have approval), or should we add it to it and go back to review next week?
Are there more APIs missing on IQueryable? Do we need to do broader sweep to bring it to parity with Enumerable?

Of course: I'd like to wait on @VSadov @OmarTawfik to agree with the approach.

[JonHanna]

@karelz they'd go into System.Linq.Queryable and currently there's parity bar those I mentioned above that only make sense to have for one or the other (e.g. to do ToList we have to enumerate elements anyway, and therefore it only belongs on Enumerable). There's also a test in the test project for each that'll fail if you add a public method to one and not either add it to the other, or add it to the list of exceptions in the test (which should of course be done because there's a good reason for the exception, rather than just to make the failure go away).

(Actually, we had one parity gap in that IOrderedQueryable<T> was covariant and IOrderedEnumerable<T> wasn't. So we've had parity as of 5 hours ago).