Velox, GCS and cURL | CURL error [77]=Problem with the SSL CA cert (path? access rights?)

[A-Mongy]

I am trying to test Gluten with Velox backend with spark 3.4.1

I am using the jar from https://downloads.apache.org/incubator/gluten/1.3.0-incubating/ for spark34

GCS configs are correctly set

spark.hadoop.fs.gs.auth.type                         SERVICE_ACCOUNT_JSON_KEYFILE
spark.hadoop.fs.gs.auth.service.account.json.keyfile // path to the json file with the credentials.

curl seems to be working right, I tried to update the ca certificates also, so I am not sure what is the issue here.

I get the following error in the executor logs

25/06/11 19:54:40 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 84.5 KiB, free 1458.3 MiB)
25/06/11 19:54:40 INFO TorrentBroadcast: Reading broadcast variable 0 took 102 ms
25/06/11 19:54:40 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 232.1 KiB, free 1458.1 MiB)
E20250611 19:54:41.385499    92 Exceptions.h:66] Line: /root/src/weiting/gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp:62, Function:checkGcsStatus, Expression:  Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)', Source: RUNTIME, ErrorCode: INVALID_STATE
25/06/11 19:54:41 ERROR TaskResources: Task 0 failed by error: 
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)'
Retriable: False
Context: Split [Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245] Task Gluten_Stage_0_TID_0_VTID_0
Additional Context: Operator: TableScan[0] 0
Function: checkGcsStatus
File: /root/src/weiting/gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp
Line: 62
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox12_GLOBAL__N_114checkGcsStatusEN6google5cloud5v2_226StatusERKSt17basic_string_viewIcSt11char_traitsIcEERKSsSD_
# 4  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile13preadInternalEmmPc
# 5  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile6preadvEmRKSt6vectorIN5folly5RangeIPcEESaIS7_EE
# 6  _ZN8facebook5velox4dwio6common19ReadFileInputStream4readERKSt6vectorIN5folly5RangeIPcEESaIS8_EEmNS2_10MetricsLog11MetricsTypeE
# 7  _ZN8facebook5velox4dwio6common19DirectCoalescedLoad8loadDataEb
# 8  _ZN8facebook5velox5cache13CoalescedLoad12loadOrFutureEPN5folly10SemiFutureIbEEb
# 9  _ZN8facebook5velox4dwio6common17DirectInputStream12loadPositionEv
# 10 _ZN8facebook5velox4dwio6common17DirectInputStream4NextEPPKvPi
# 11 _ZN8facebook5velox7parquet10ReaderBase16loadFileMetaDataEv
# 12 _ZN8facebook5velox7parquet10ReaderBaseC1ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 13 _ZN8facebook5velox7parquet13ParquetReaderC2ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 14 _ZN8facebook5velox7parquet20ParquetReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 15 _ZN8facebook5velox9connector4hive11SplitReader12createReaderEv
# 16 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 17 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 18 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 19 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv
# 20 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 21 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 22 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 23 _ZN6gluten24WholeStageResultIterator4nextEv
# 24 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 25 0x00007e6f146b9930

        at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
        at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
        at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
        at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
        at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
        at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
        at scala.collection.Iterator.isEmpty(Iterator.scala:387)
        at scala.collection.Iterator.isEmpty$(Iterator.scala:387)
        at org.apache.spark.InterruptibleIterator.isEmpty(InterruptibleIterator.scala:28)
        at org.apache.gluten.execution.VeloxColumnarToRowExec$.toRowIterator(VeloxColumnarToRowExec.scala:122)
        at org.apache.gluten.execution.VeloxColumnarToRowExec.$anonfun$doExecuteInternal$1(VeloxColumnarToRowExec.scala:78)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:853)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:853)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
        at org.apache.spark.scheduler.Task.run(Task.scala:139)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError

[A-Mongy]

Any help is appreciated here
@zhouyuan, @holdenk, @raboof, @EpsilonPrime

[zhouyuan]

@A-Mongy AFAIK the gcs connector should be working fine. The issue you posted seems related with ca-certs on runtime. I find some docs here (https://bobcares.com/blog/curl-error-77-problem-with-the-ssl-ca-cert/). Since you have already updated ca-certs, would you please do a quick test by using the nightly package?
https://nightlies.apache.org/gluten/

thanks, -yuan

[A-Mongy]

Thanks @zhouyuan for getting back to me on this!
I have tried to change the permission for the etc/ssl/ folder

total 24
drwxr-xr-x 1 root root  4096 Apr 15 04:41 certs
-rwxr-xr-x 1 root root 12332 Oct 27  2024 openssl.cnf
drwxr-xr-x 1 root root  4096 Oct 27  2024 private

So shouldn't be a permission issue.
I have also tried the nightly jars, and I still see the same problem
Here is an updated log of the error, as I turned on debug mode:

20250618 00:30:14.379505    91 HiveConnectorUtil.cpp:286] No need to check type lowercase modeVARCHAR
I20250618 00:30:14.379514    91 HiveConnectorUtil.cpp:286] No need to check type lowercase modeVARCHAR
I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1
I20250618 00:30:14.384630    96 EventBase.h:883] expCoeff_ -5e-07 folly::EventBase::SmoothLoopTime::SmoothLoopTime(std::chrono::microseconds)
I20250618 00:30:14.384725    96 EventBase.cpp:543] EventBase(): Starting loop.
I20250618 00:30:14.384757    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551576 next 18446744073709551577
I20250618 00:30:14.384773    96 EventBase.cpp:1058] latest 18446744073709551576 next 18446744073709551577
I20250618 00:30:14.384784    96 EventBase.cpp:801] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) startWork_ 29679038962635
I20250618 00:30:14.384800    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384812    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384876    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384821    96 EventBase.cpp:655] EventBase 0x7ee85c00a760 did not timeout  loop time guess: 76 idle time: 40 busy time: 36 avgLoopTime: 0.001368 maxLatencyLoopTime: 0.001368 maxLatency_: 0us notificationQueueSize: 1 nothingHandledYet(): 0
I20250618 00:30:14.384897    96 EventBase.cpp:686] EventBase 0x7ee85c00a760 loop time: 0
I20250618 00:30:14.384910    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551577 next 18446744073709551578
I20250618 00:30:14.384919    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551578
I20250618 00:30:14.384930    96 EventBase.cpp:801] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) startWork_ 29679039108575
I20250618 00:30:14.384951    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384963    96 EventBase.cpp:1058] latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384982    96 EventBase.cpp:1058] latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384972    96 EventBase.cpp:655] EventBase 0x7ee85c00a760 did not timeout  loop time guess: 151 idle time: 109 busy time: 42 avgLoopTime: 0.008853 maxLatencyLoopTime: 0.008853 maxLatency_: 0us notificationQueueSize: 0 nothingHandledYet(): 0
I20250618 00:30:14.384994    96 EventBase.cpp:686] EventBase 0x7ee85c00a760 loop time: 0
E20250618 00:30:14.421082    91 Exceptions.h:66] Line: /work/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp:62, Function:checkGcsStatus, Expression:  Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)', Source: RUNTIME, ErrorCode: INVALID_STATE
I20250618 00:30:14.421433    91 Task.cpp:2402] Terminating task Gluten_Stage_0_TID_0_VTID_0 with state Failed after running for 43ms
I20250618 00:30:14.421552    91 Task.cpp:1343] All drivers (1) finished for task Gluten_Stage_0_TID_0_VTID_0 after running for 43ms
25/06/18 00:30:14 ERROR TaskResources: Task 0 failed by error: 
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)'
Retriable: False
Context: Split [Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245] Task Gluten_Stage_0_TID_0_VTID_0
Additional Context: Operator: TableScan[0] 0
Function: checkGcsStatus
File: /work/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp
Line: 62
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox12_GLOBAL__N_114checkGcsStatusEN6google5cloud5v2_226StatusERKSt17basic_string_viewIcSt11char_traitsIcEERKSsSD_
# 4  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile13preadInternalEmmPc
# 5  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile6preadvEmRKSt6vectorIN5folly5RangeIPcEESaIS7_EEPNS0_11filesystems4File7IoStatsE
# 6  _ZN8facebook5velox4dwio6common19ReadFileInputStream4readERKSt6vectorIN5folly5RangeIPcEESaIS8_EEmNS2_10MetricsLog11MetricsTypeE
# 7  _ZN8facebook5velox4dwio6common19DirectCoalescedLoad8loadDataEb
# 8  _ZN8facebook5velox5cache13CoalescedLoad12loadOrFutureEPN5folly10SemiFutureIbEEb
# 9  _ZN8facebook5velox4dwio6common17DirectInputStream12loadPositionEv
# 10 _ZN8facebook5velox4dwio6common17DirectInputStream4NextEPPKvPi
# 11 _ZN8facebook5velox7parquet10ReaderBase16loadFileMetaDataEv
# 12 _ZN8facebook5velox7parquet10ReaderBaseC1ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 13 _ZN8facebook5velox7parquet13ParquetReaderC2ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 14 _ZN8facebook5velox7parquet20ParquetReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 15 _ZN8facebook5velox9connector4hive11SplitReader12createReaderEv
# 16 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 17 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 18 _ZN8facebook5velox4exec9TableScan8getSplitEv
# 19 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 20 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv
# 21 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 22 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 23 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 24 _ZN6gluten24WholeStageResultIterator4nextEv
# 25 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 26 0x00007ee91c6b9930

You can notice the two lines where the gcs bucket is also mentioned are:

I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1

So I would say gluten is able to access the gcs bucket? what do you think?

[PHILO-HE]

You can notice the two lines where the gcs bucket is also mentioned are:

I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1

So I would say gluten is able to access the gcs bucket? what do you think?

@A-Mongy, I just reviewed the relevant code. This log cannot tell the bucket is accessible. It is logged before the reader is initialized.

[A-Mongy]

You can notice the two lines where the gcs bucket is also mentioned are:

I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1

So I would say gluten is able to access the gcs bucket? what do you think?

@A-Mongy, I just reviewed the relevant code. This log cannot tell the bucket is accessible. It is logged before the reader is initialized.

https://github.com/facebookincubator/velox/blob/main/velox/connectors/hive/FileHandle.cpp#L61-L65
In line 61, this looks like it's accessing the gcs bucket, no?
The logging happens in line 65 so this didn't raise an error.

    fileHandle->file = filesystems::getFileSystem(filename, properties_)
                           ->openFileForRead(filename, options);

[PHILO-HE]

@A-Mongy, I see. Then, it seems Velox can access the object metadata if no error is reported when calling this code.

https://github.com/facebookincubator/velox/blob/b4ee6722a5f446b92888c7a47e8bccb0118688ff/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp#L865

https://github.com/facebookincubator/velox/blob/b4ee6722a5f446b92888c7a47e8bccb0118688ff/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp#L108

[A-Mongy]

@PHILO-HE, I am confused, is the S3 reference has anything to do with GCS buckets?

Also, on the executor pod, this is the out ofcurl -v https://storage.googleapis.com, and I can see TLSv1.3 (IN), TLS handshake, CERT verify (15): and connection is successful

*   Trying 173.194.64.207:443...
* Connected to storage.googleapis.com (173.194.64.207) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=storage.googleapis.com
*  start date: Jun  2 08:39:34 2025 GMT
*  expire date: Aug 25 08:39:33 2025 GMT
*  issuer: C=US; O=Google Trust Services; CN=WE2
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: storage.googleapis.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5762cb18e780)
> GET / HTTP/2
> Host: storage.googleapis.com
> user-agent: curl/7.88.1
> accept: */*
> 
< HTTP/2 400 
< content-type: application/xml; charset=UTF-8
< x-guploader-uploadid: ABgVH8-mL3I67FxOb9izaMRMV4DNstOyH3lpX9pzsAYWM0fhHTdIFdE0iGAmiCaM4q6ow-0
< content-length: 181
< date: Thu, 19 Jun 2025 10:12:56 GMT
< expires: Thu, 19 Jun 2025 10:12:56 GMT
< cache-control: private, max-age=0
< server: UploadServer
< 
* Connection #0 to host storage.googleapis.com left intact

[PHILO-HE]

@A-Mongy, it's strange. Looks certificate is already available at executor side.

[A-Mongy]

@zhztheplayer any hints on this?

[A-Mongy]

I have to add to this, when triggering a spark submit to only read data from GCS, reading a CSV or parquet file. There is no issues and it reads and prints the data from the file with no issue. So definitely not a GCS access issue and not a curl issue.

However, if I try to write data I can see the issue.
Also, with a recent jar I can see the issue is now hidden by the following error instead

Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: Empty Optional cannot be unwrapped
Retriable: False
Function: operator()
File: /work/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 571
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
# 4  _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 5  _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 6  _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 7  _ZN6gluten24WholeStageResultIterator4nextEv
# 8  Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 9  0x00007879946b9930

        at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
        at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
        at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
        ... 28 more
25/06/23 15:04:58 INFO CoarseGrainedExecutorBackend: Got assigned task 1
25/06/23 15:04:58 INFO Executor: Running task 0.1 in stage 0.0 (TID 1)

[zhouyuan]

It looks like something is wrong in the GCS code path as Gluten dont have tests to guard this yet.
Cc: @surnaik to check his comments

[vanphuoc3012]

Hi, I also face this problem when download nightly release jar. I fix by rebuild gluten in docker using ubutun image instead of using centos

[zhouyuan]

I was debugging another issue but thought this maybe related. It seems on centos7 the curl lib has some different location of the certs than on ubuntu. On ubuntu this can be fixed by:

mkdir -p /etc/pki/tls/certs
ln -s /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt