Skip to content

oneDAL Binary Classification LBFGS - Index was outside the bounds of the array. #6533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
luisquintanilla opened this issue Jan 4, 2023 · 1 comment
Labels
bug Something isn't working
Milestone

Comments

@luisquintanilla
Copy link
Contributor

luisquintanilla commented Jan 4, 2023

System Information (please complete the following information):

  • OS & Version: Windows 11
  • ML.NET Version: ML.NET 3.0 prerelease
  • .NET Version: .NET 6 & .NET 6

Describe the bug

Training a binary classification model using the Lbfgs trainer and setting the MLNET_BACKEND environment variable to ONEDAL produces the following error:

Unhandled exception. System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Microsoft.ML.Trainers.LbfgsTrainerBase`3.TrainCoreOneDal(IChannel ch, RoleMappedData data)
   at Microsoft.ML.Trainers.LbfgsTrainerBase`3.TrainModelCore(TrainContext context)
   at Microsoft.ML.Trainers.TrainerEstimatorBase`2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
   at Microsoft.ML.Trainers.TrainerEstimatorBase`2.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Program.<Main>$(String[] args) in C:\Dev\OneDalTest\OneDalTest\Program.cs:line 31

To Reproduce

  1. Create a C# console application
  2. Install the latest prerelease versions of Microsoft.ML and Microsoft.ML.OneDAL packages.
  3. Paste the following code into the Program.cs file.
// Initialize MLContext
var ctx = new MLContext();

// Define data
var trainingData = new [] 
{
    new {Arch="ARM", Trainer="LightGBM", oneDALSupport=false},
    new {Arch="x86", Trainer="FastTree", oneDALSupport=true},
    new {Arch="x86", Trainer="LbfgsLogisticRegression", oneDALSupport=true},
    new {Arch="ARM", Trainer="FastTree", oneDALSupport=false}
};

// Load data into IDataView
var trainingDv = ctx.Data.LoadFromEnumerable(trainingData);

// Define data processing pipeline & trainer
var pipeline = 
    ctx.Transforms.Categorical.OneHotEncoding(new [] {
            new InputOutputColumnPair("ArchEncoded", "Arch"),
            new InputOutputColumnPair("TrainerEncoded", "Trainer")})
        .Append(ctx.Transforms.Concatenate("Features", "ArchEncoded", "TrainerEncoded"))
        .Append(ctx.BinaryClassification.Trainers.LbfgsLogisticRegression(labelColumnName:"oneDALSupport"));

// Train model
var model = pipeline.Fit(trainingDv);
  1. Set the MLNET_BACKEND environment variable to ONEDAL
  2. Run the application.

Expected behavior
The model trains successfully.

Additional context

The same code using the FastTree trainer trains the model successfully.

@ghost ghost added the untriaged New issue has not been triaged label Jan 4, 2023
@luisquintanilla luisquintanilla added the bug Something isn't working label Jan 4, 2023
@michaelgsharp michaelgsharp added this to the ML.NET 3.0 milestone Feb 6, 2023
@ghost ghost removed the untriaged New issue has not been triaged label Feb 6, 2023
@michaelgsharp
Copy link
Contributor

@rgesteve for visability.

rgesteve added a commit to rgesteve/machinelearning that referenced this issue Sep 26, 2023
michaelgsharp pushed a commit that referenced this issue Sep 27, 2023
* Initial structure and started fleshing out some sections

* Some corrections and paragraph on DL usages

* Starting fleshing out DL on ML.NET section

* Addresses #6533
JakeRadMSFT added a commit that referenced this issue Oct 5, 2023
* Update build templates to handle feature branches (#6744)

* Update build templates

* Update build templates to include all releases/* and feature/*

* Update releases to release

* Update triggers for PR Validation Build

* Add triggers for Code Coverage

* Update version to 4.0 for feature branch (#6743)

* Add missing implementation for datetime relevant arrow type into dataframe (#6675)

* Add missing implementation for datetime relevant arrow type

* Return required usage

* Fix the behavior or column SetName method (#6676)

* Fix the behavior or column SetName method

* Fix stack overflow exception

* Fix merge issues

---------

Co-authored-by: Michael Sharp <[email protected]>

* Fix DataFrame to allow to store columns with size more than 2 Gb (#6710)

* Fix error with allocating more than MaxCapacity of Byte Memory Buffer

* Remove Unit test as it consumes too much memory

* Fix issue with increasing buffer capacity over limit when double it size

* avoid empty dataset (#6756)

* Fix dataframe arithmetics for columns having several value buffers (column size is more than 2 Gb) (#6724)

* Fix dataframe arithmetics

* Fix

* Run tests that requires more than 2 Gb of Memory only on 64-bit env (#6758)

* Reduce coupling of Data.Analysis.Tests project (#6759)

* Provide ability to filter dataframe column by null via ElementWise Methods (#6723)

* Provide ability to filter by null value

* Add comments

* Fix code review findings

* Fix incorrect DataFrame min max computation with NULL (#6734)

* Step 1

* Step 2

* Fixed code review findings

* Clean DataFrame meaningless code (#6761)

* Add NameEntityRecognition and Q&A deep learning tasks. (#6760)

* NER

* QA almost done, runtime error

* QA finished

* fixes from PR comments

* fixed build

* build fixes

* perf changes

* made disposable

* fixed not disposing model

* added some disposables to TensorFlow for memory

* build testing

* fixing build

* added missing dispose

* build fixes

* build fixes

* testing macos fix

* fix issue (#6768)

* fixed mac build and minor torch sharp changes (#6776)

* Improve DataFrame Arithmetics implementation (#6763)

* Change methods signature generation

* Change DataFrameColumn Arithmetics

* Change DataFrameColumn Operations

* Fix unit tests

* Fix spaces

* Fix code review findings

* Add QA sweepable estimator in AutoML  (#6781)

* Add QA sweepable

* clean

* Modernized some argument checks that still used string literals for parameter names (#6766)

Co-authored-by: John Doe <john@doe>

* removed deprecated yosemite brew (#6805)

* Add TargetType to Type_convert (#6785)

* Add target Type in convert  type

* Add custom type "DataKind"

* clean

* Add DataKind name space

* clean test

* File-scoped namespaces in files under `Environment` (`Microsoft.ML.Core`) (#6791)

Co-authored-by: Lehonti Ramos <john@doe>

* File-scoped namespaces in files under `EntryPoints` (`Microsoft.ML.Core`) (#6790)

Co-authored-by: Lehonti Ramos <john@doe>

* Fix issue with addIndexColumn in DataFrame.LoadCsv (#6769)

* Fix issue with addIndexColumn in DataFrame.LoadCsv

* Fix tests

* Fix DataFrame.LoadCsv can not load CSV with duplicate column names (#6772)

* File-scoped namespaces in files under `ComponentModel` (`Microsoft.ML.Core`) (#6788)

Co-authored-by: Lehonti Ramos <john@doe>

* File-scoped namespaces in files under `Data` (`Microsoft.ML.Core`) (#6789)

Co-authored-by: Lehonti Ramos <john@doe>

* Fix inconsistent null handling in DataFrame Arithmetics (#6770)

* Fix inconsistent null handling in DataFrame Arithmetics

* Fix Null Count and division by zero issues

* Minor changes to restart build and rerun flaky tests

* File-scoped namespaces in files under `Prediction` (`Microsoft.ML.Core`) (#6792)

Co-authored-by: Lehonti Ramos <john@doe>

* Allow to define CultureInfo for parsing values on reading DataFrame from csv (#6782)

* Use CultureInfo for parsing values in csv file

* Fix merge issues

* Append dataframe rows based on column names (#6808)

* Append dataframe rows based on column names

* Update DataFrame.cs

---------

Co-authored-by: Michael Sharp <[email protected]>

* removed codecov token (#6811)

* Fix wrong type conversion on PrimitiveDataFrameColumn (#6834)

* Fix wrong type conversion on PrimitiveDataFrameColumn

* Added tests for #6829

* Fix test

* Add file generated from tt template and fix unit tests

---------

Co-authored-by: Aleksei Smirnov <[email protected]>

* update interactive kernel version (#6836)

* update interactive kernel version

* update

* Update Microsoft.Data.Analysis.Interactive.Tests.csproj

* Add performance benchmarks for dataframe arithmetic operations  (#6827)

* Add performance tests

* Add extra tests

* Fix

* Fix typo

* Fix Divide_Int16 and Divide_Int32_Int16 benchmarks

* Fix

* Change csproj file

* Update BenchmarkDotNetVersion to 0.13.5

* Fix

* Change to 0.13.1 because that is what is latest version in our nuget feeds.

---------

Co-authored-by: Jake Radzikowski <[email protected]>

* Improve performance of column cloning inside DataFrame arithmetics (#6814)

* Optimize PrimitiveColumnContainer.Clone method

* Avoid unnecessary type conversion during binary operations

* Remove using

* Fix DataFrameBuffer constructor

* remove uncorrectly added using

* Make DataFrameBuffer Length field protected

* Fix typo

* Use RawSpan

* Simplify tt files for PrimitiveDataFrameColumnAritmetics (#6830)

* First step of tt refactoring

* Step 2

* Step 3

* Addresses #6533 (#6838)

* Initial structure and started fleshing out some sections

* Some corrections and paragraph on DL usages

* Starting fleshing out DL on ML.NET section

* Addresses #6533

* Update dependencies (#6837)

* Update dependencies

* Add reference to NuGet.Packaging.Core

* PrimitiveDataFrameColumn.Clone method crashes when is used with IEnumerable mapIndices argument (#6822)

* Split Test for AppendMany into 4 different tests

* Block init of null validity buffer instead of setting individual bits

* Add unit tests for PrimitiveDataFrameColumn.Clone

* Fixes #6821

* Fix

* Fix bug with AppendMany values to not empty column

* Restart unit tests

* Add more unit tests

* Fix failing unit test

* Fix code review findings

* 6847 incorrectly sets column value (#6849)

* Fix DataFrame incorrectly sets column value for index higher than Buffer.MaxCapacity

* Revert renaming

* Increase performance of arithmetic operations by enhancing calculations on nullable values  (#6846)

* Optimize PrimitiveColumnContainer.Clone method

* Avoid unnecessary type conversion during binary operations

* Remove using

* Fix DataFrameBuffer constructor

* remove uncorrectly added using

* Make DataFrameBuffer Length field protected

* Add performance tests

* Split Test for AppendMany into 4 different tests

* Block init of null validity buffer instead of setting individual bits

* Add unit tests for PrimitiveDataFrameColumn.Clone

* Fixes #6821

* Fix

* Add extra tests

* Fix

* Fix typo

* Fix Divide_Int16 and Divide_Int32_Int16 benchmarks

* Fix

* Avoid using constructor, that copies memory

* First step of tt refactoring

* Step 2

* Step 3

* Move iteration over buffers outside of the PrimitiveDataFrameColumnArithmetic

* Change PrimitiveDataFrameColumnArithmetic

* Fix typo

* Use RawSpan

* Fix bug with AppendMany values to not empty column

* Restart unit tests

* Add more unit tests

* Add GetBitCount method

* Fix failing unit test

* Implementation

* Change unit tests

* Update unit tests

* Refactoring BinaryOperation

* Intermediate changes

* Intermediate results

* Implement Binary Scalar Reverse Operarions

* Add implementation for BinaryIntOperations

* Implement Comparison Operations

* Implement actual calculations for Comparison operations

* Uncomment performance tests

* Remove unintentional code changes

* Add reference to Apache Arrow project license in THIRD-PARTY-NOTICES

* Fix license issues

* Fixes incorrect work of DataFrame with VBufferColumn when number of e… (#6851)

* Fixes incorrect work of DataFrame with VBufferColumn when number of elements is greater than Int.MaxValue

* Fix calculation of max capacity and amount of required buffers

* Fix unit test

* Run test allocating more than 2 Gb of memory on 64bit env only

* Fix StringDataFrameColumn same way as VBufferDataFrameColumn

* Fix wrong amount of buffers created in constructor of StringDataFrameColumn

* Fix code review findings

---------

Co-authored-by: Aleksei Smirnov <[email protected]>
Co-authored-by: Michael Sharp <[email protected]>
Co-authored-by: Xiaoyun Zhang <[email protected]>
Co-authored-by: zewditu Hailemariam <[email protected]>
Co-authored-by: Lehonti Ramos <[email protected]>
Co-authored-by: John Doe <john@doe>
Co-authored-by: Raffaello Fraboni <[email protected]>
Co-authored-by: R. G. Esteves <[email protected]>
Co-authored-by: Eric StJohn <[email protected]>
@github-actions github-actions bot locked and limited conversation to collaborators Dec 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants