BUG: read_csv with lines starting whitespace #: engine="python" skips them, engine="c" doesn't

[denis-bz]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from io import StringIO
print( f"{pd.__version__ = }" )

csv = """
col1;col2;col3
1;4.4;99
    #2;4.5;200
3;4.7;65
""" 

for engine in ["c", "python"]:
    print( f"\n-- {engine = }" )
    df = pd.read_csv( StringIO( csv ), sep=";", comment="#", engine=engine,
            skip_blank_lines=True,  # default
            )
    print( df )

Issue Description

pd.version = '2.1.2'

-- engine = 'c'
col1 col2 col3
0 1 4.4 99
1 \t NaN NaN
2 3 4.7 65

-- engine = 'python'
col1 col2 col3
0 1 4.4 99
1 3 4.7 65

Expected Behavior

with skip_blank_lines=True, all engines should skip lines starting with whitespace commentchar

Installed Versions

INSTALLED VERSIONS ------------------ commit : a60ad39 python : 3.10.0.final.0 python-bits : 64 OS : Darwin OS-release : 19.6.0 Version : Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : None LOCALE : en_GB.UTF-8

pandas : 2.1.2
numpy : 1.26.1
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 57.4.0
pip : 23.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.18.0.dev
pandas_datareader : None
bs4 : 4.12.2
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.6.0
gcsfs : None
matplotlib : 3.8.1
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.3
sqlalchemy : None
tables : None
tabulate : None
xarray : 2023.10.1
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[denis-bz]

messier than I thought, try sep=r"\s+" too
related: #36069

[denis-bz]

Fwiw, a tiny testcase generator for this --

# 24 x 4 csv, permutations of A , \t #
# to test https://github.com/pandas-dev/pandas/issues/55916

from itertools import permutations
from io import StringIO 
import pandas as pd

print( f"{pd.__version__ = }" )  # 2.1.3

rows = ["A,B"] + ["".join( p ) for p in permutations( "A,\t#" )]
csvstr = "\n".join( rows )

for engine in ["c", "python"]:
  # for sep, for comment
    print( f"\n-- {engine = }" )
    df = pd.read_csv( StringIO( csvstr ), sep=",", comment="#", engine=engine,
            skip_blank_lines=True,  # default
            )
    print( df )