DaviX errors from ROOT.File.Cp

[jcatmore]

Check duplicate issues.

• Checked for duplicates

Description

In both 6.32 and 6.34 we have started observing error messages from DaviX when trying to access files from WebEOS sites, specifically the following site:
https://atlas-groupdata.web.cern.ch/atlas-groupdata/
We see different behaviour when ROOT.TFile.Cp is used in a single step, versus when the TFile is opened first (see reproducer). This is an issue due to a piece of software used in ATLAS (PathResolver) that uses this single step.

Reproducer

CASE 1 (doesn't work):

import ROOT
ROOT.TFile.Cp("https://atlas-groupdata.web.cern.ch/ElectronEfficiencyCorrection/2015_2018/rel21.2/Precision_Summer2020_v1/trigger/efficiencySF.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_2018_e26_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.TightLLH_d0z0_v13_isolTight.root", "/tmp/test.root")

Error in <TDavixFile::DavixReadBuffer>: can not read data with davix: HTTP 403 : Permission refused  (17)
Error in <TFile::Cp>: cannot read from source file https://atlas-groupdata.web.cern.ch/ElectronEfficiencyCorrection/2015_2018/rel21.2/Precision_Summer2020_v1/trigger/efficiencySF.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_2018_e26_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.TightLLH_d0z0_v13_isolTight.root. readsize=81971 read=0 readop=1
False

CASE 2 (works):

import ROOT
f=ROOT.TFile.Open("https://atlas-groupdata.web.cern.ch/ElectronEfficiencyCorrection/2015_2018/rel21.2/Precision_Summer2020_v1/trigger/efficiencySF.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_2018_e26_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.TightLLH_d0z0_v13_isolTight.root")

f.Cp("/tmp/test.root")

[TFile::Cp] Total 0.08 MB	|====================| 100.00 % [12.9 MB/s]
True

ROOT version

6.34.04-x86_64-el9-gcc13-opt
6.32.08-x86_64-el9-gcc13-opt

Installation method

CVMFS via AtlasSetup

Operating system

AlmaLinux9 (LXPLUS)

Additional context

It is possible that these issues are due to some settings changing on the site in question, but we aren't sure what to report. Some specific statement from a DaviX expert of the form "X was probably changed to Y and this is why there is an error" would help us to revert whatever was changed, if this is indeed the cause of the issue.

[dpiparo]

From lxplus: some https work some do not. Digging further.

~$ root -l -b -q -e "TFile::Cp(\"https://root.cern/download/atlfast.root\", \"/tmp/foo.root\")"

[TFile::Cp] Total 1.74 MB	|====================| 100.00 % [29.3 MB/s]
(bool) true

~$ root -l -b -q -e "TFile::Cp(\"https://atlas-groupdata.web.cern.ch/ElectronEfficiencyCorrection/2015_2018/rel21.2/Precision_Summer2020_v1/trigger/efficiencySF.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_2018_e26_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.TightLLH_d0z0_v13_isolTight.root\", \"/tmp/foo.root\")"
Error in <TDavixFile::DavixReadBuffer>: can not read data with davix: HTTP 403 : Permission refused  (17)
Error in <TFile::Cp>: cannot read from source file https://atlas-groupdata.web.cern.ch/ElectronEfficiencyCorrection/2015_2018/rel21.2/Precision_Summer2020_v1/trigger/efficiencySF.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_2018_e26_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.TightLLH_d0z0_v13_isolTight.root. readsize=81971 read=0 readop=1
(bool) false

[dpiparo]

Adding @mpatrascoiu in the loop.

[jcatmore]

Possible that this is related:
https://cern.service-now.com/service-portal?id=ticket&table=incident&n=INC4444682

The error message is different though.

[mpatrascoiu]

@joaopblopes is investigating this

We believe it might have to do with redirection support.

If redirection support is disabled, a similar error can be produced.
Can someone check from the ROOT side if Davix redirection support is enabled/disabled?

Also, are you using the latest Davix v0.8.10 version?

Cheers,
Mihai

[ponyisi]

My strong belief is this is not a Davix issue, but is related to the fact that TFile::Cp adds a query string to the URL (in this case ?filetype=raw&cachesz=4000000&readaheadsz=2000000&rmpolicy=1) when attempting to retrieve a file. If I try to use wget on the URL that TFile::Cp attempts to retrieve, I get the same 403 error. I suspect that Apache (or the CERN configuration of it) used to be more lenient about ignoring this, and that something has changed in the way the servers deal with these requests.

[ponyisi]

See e.g.

root/io/io/src/TFile.cxx

Line 5199 in 81f0261

opt += TString::Format("&cachesz=%d&readaheadsz=%d&rmpolicy=1", 4*buffersize, 2*buffersize);

.