test_apply_pyproject_equivalent_to_setuptcfg is flaky

[jaraco]

Recently, I've noticed that test_apply_pyproject_equivalent_to_setupcfg will fail but then succeed on a retry. Here's an example run from a distutils test:

=================================== FAILURES ===================================
_ test_apply_pyproject_equivalent_to_setupcfg[https://github.com/sqlalchemy/sqlalchemy/raw/533f5718904b620be8d63f2474229945d6f8ba5d/setup.cfg] _
[gw0] linux -- Python 3.10.20 /home/runner/work/distutils/distutils/integration/setuptools/.tox/py/bin/python

url = 'https://github.com/sqlalchemy/sqlalchemy/raw/533f5718904b620be8d63f2474229945d6f8ba5d/setup.cfg'
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f6477d612d0>
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/popen-gw0/test_apply_pyproject_equivalen10')

    @pytest.mark.parametrize("url", urls_from_file(HERE / EXAMPLES_FILE))
    @pytest.mark.filterwarnings("ignore")
    @pytest.mark.uses_network
    def test_apply_pyproject_equivalent_to_setupcfg(url, monkeypatch, tmp_path):
        monkeypatch.setattr(expand, "read_attr", Mock(return_value="0.0.1"))
        monkeypatch.setattr(
            Distribution, "_expand_patterns", Mock(side_effect=_mock_expand_patterns)
        )
        setupcfg_example = retrieve_file(url)
        pyproject_example = Path(tmp_path, "pyproject.toml")
        setupcfg_text = setupcfg_example.read_text(encoding="utf-8")
        toml_config = LiteTranslator().translate(setupcfg_text, "setup.cfg")
        pyproject_example.write_text(toml_config, encoding="utf-8")
    
>       dist_toml = pyprojecttoml.apply_configuration(makedist(tmp_path), pyproject_example)

setuptools/tests/config/test_apply_pyprojecttoml.py:63: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
setuptools/config/pyprojecttoml.py:72: in apply_configuration
    config = read_configuration(filepath, True, ignore_option_errors, dist)
setuptools/config/pyprojecttoml.py:149: in read_configuration
    validate(subset, filepath)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

config = {'project': {'dynamic': ['version']}, 'tool': {'setuptools': {'include-package-data': False}}}
filepath = '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_apply_pyproject_equivalen10/pyproject.toml'

    def validate(config: dict, filepath: StrPath) -> bool:
        from . import _validate_pyproject as validator
    
        trove_classifier = validator.FORMAT_FUNCTIONS.get("trove-classifier")
        if hasattr(trove_classifier, "_disable_download"):
            # Improve reproducibility by default. See abravalheri/validate-pyproject#31
            trove_classifier._disable_download()  # type: ignore[union-attr]
    
        try:
            return validator.validate(config)
        except validator.ValidationError as ex:
            summary = f"configuration error: {ex.summary}"
            if ex.name.strip("`") != "project":
                # Probably it is just a field missing/misnamed, not worthy the verbosity...
                _logger.debug(summary)
                _logger.debug(ex.details)
    
            error = f"invalid pyproject.toml config: {ex.name}."
>           raise ValueError(f"{error}\n{summary}") from None
E           ValueError: invalid pyproject.toml config: `project`.
E           configuration error: `project` must contain ['name'] properties

setuptools/config/pyprojecttoml.py:61: ValueError
----------------------------- Captured stdout call -----------------------------
Downloading https://github.com/sqlalchemy/sqlalchemy/raw/533f5718904b620be8d63f2474229945d6f8ba5d/setup.cfg to /home/runner/work/distutils/distutils/integration/setuptools/setuptools/tests/config/downloads/sqlalchemy_sqlalchemy533f5718904b620be8d63f2474229945d6f8ba5d_setup.cfg

[jaraco]

Another manifestation:

=================================== FAILURES ===================================
_ test_apply_pyproject_equivalent_to_setupcfg[https://github.com/pandas-dev/pandas/raw/bc17343f934a33dc231c8c74be95d8365537c376/setup.cfg] _
[gw0] linux -- Python 3.10.20 /home/runner/work/distutils/distutils/integration/setuptools/.tox/py/bin/python

self = <urllib.request.HTTPSHandler object at 0x7fdca3e30760>
http_class = <class 'http.client.HTTPSConnection'>
req = <urllib.request.Request object at 0x7fdca3964b50>
http_conn_args = {'context': None, 'check_hostname': None}

    def do_open(self, http_class, req, **http_conn_args):
        """Return an HTTPResponse object for the request, using http_class.
    
        http_class must implement the HTTPConnection API from http.client.
        """
        host = req.host
        if not host:
            raise URLError('no host given')
    
        # will parse host:port
        h = http_class(host, timeout=req.timeout, **http_conn_args)
        h.set_debuglevel(self._debuglevel)
    
        headers = dict(req.unredirected_hdrs)
        headers.update({k: v for k, v in req.headers.items()
                        if k not in headers})
    
        # TODO(jhylton): Should this be redesigned to handle
        # persistent connections?
    
        # We want to make an HTTP/1.1 request, but the addinfourl
        # class isn't prepared to deal with a persistent connection.
        # It will try to read all remaining data from the socket,
        # which will block while the server waits for the next request.
        # So make sure the connection gets closed after the (only)
        # request.
        headers["Connection"] = "close"
        headers = {name.title(): val for name, val in headers.items()}
    
        if req._tunnel_host:
            tunnel_headers = {}
            proxy_auth_hdr = "Proxy-Authorization"
            if proxy_auth_hdr in headers:
                tunnel_headers[proxy_auth_hdr] = headers[proxy_auth_hdr]
                # Proxy-Authorization should not be sent to origin
                # server.
                del headers[proxy_auth_hdr]
            h.set_tunnel(req._tunnel_host, headers=tunnel_headers)
    
        try:
            try:
>               h.request(req.get_method(), req.selector, req.data, headers,
                          encode_chunked=req.has_header('Transfer-encoding'))

/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:1348: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1303: in request
    self._send_request(method, url, body, headers, encode_chunked)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1349: in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1298: in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1058: in _send_output
    self.send(msg)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:996: in send
    self.connect()
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1475: in connect
    self.sock = self._context.wrap_socket(self.sock,
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/ssl.py:513: in wrap_socket
    return self.sslsocket_class._create(
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/ssl.py:1104: in _create
    self.do_handshake()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ssl.SSLSocket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
block = False

    @_sslcopydoc
    def do_handshake(self, block=False):
        self._check_connected()
        timeout = self.gettimeout()
        try:
            if timeout == 0.0 and block:
                self.settimeout(None)
>           self._sslobj.do_handshake()
E           ConnectionResetError: [Errno 104] Connection reset by peer

/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/ssl.py:1375: ConnectionResetError

During handling of the above exception, another exception occurred:

url = 'https://github.com/pandas-dev/pandas/raw/bc17343f934a33dc231c8c74be95d8365537c376/setup.cfg'
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fdca397ea40>
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/popen-gw0/test_apply_pyproject_equivalen14')

    @pytest.mark.parametrize("url", urls_from_file(HERE / EXAMPLES_FILE))
    @pytest.mark.filterwarnings("ignore")
    @pytest.mark.uses_network
    def test_apply_pyproject_equivalent_to_setupcfg(url, monkeypatch, tmp_path):
        monkeypatch.setattr(expand, "read_attr", Mock(return_value="0.0.1"))
        monkeypatch.setattr(
            Distribution, "_expand_patterns", Mock(side_effect=_mock_expand_patterns)
        )
>       setupcfg_example = retrieve_file(url)

setuptools/tests/config/test_apply_pyprojecttoml.py:57: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
setuptools/tests/config/downloads/__init__.py:36: in retrieve_file
    download(url, path)
setuptools/tests/config/downloads/__init__.py:53: in download
    with urlopen(url) as f:
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:216: in urlopen
    return opener.open(url, data, timeout)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:525: in open
    response = meth(req, response)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:634: in http_response
    response = self.parent.error(
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:557: in error
    result = self._call_chain(*args)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:496: in _call_chain
    result = func(*args)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:749: in http_error_302
    return self.parent.open(new, timeout=req.timeout)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:519: in open
    response = self._open(req, data)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:536: in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:496: in _call_chain
    result = func(*args)
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:1391: in https_open
    return self.do_open(http.client.HTTPSConnection, req,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPSHandler object at 0x7fdca3e30760>
http_class = <class 'http.client.HTTPSConnection'>
req = <urllib.request.Request object at 0x7fdca3964b50>
http_conn_args = {'context': None, 'check_hostname': None}

    def do_open(self, http_class, req, **http_conn_args):
        """Return an HTTPResponse object for the request, using http_class.
    
        http_class must implement the HTTPConnection API from http.client.
        """
        host = req.host
        if not host:
            raise URLError('no host given')
    
        # will parse host:port
        h = http_class(host, timeout=req.timeout, **http_conn_args)
        h.set_debuglevel(self._debuglevel)
    
        headers = dict(req.unredirected_hdrs)
        headers.update({k: v for k, v in req.headers.items()
                        if k not in headers})
    
        # TODO(jhylton): Should this be redesigned to handle
        # persistent connections?
    
        # We want to make an HTTP/1.1 request, but the addinfourl
        # class isn't prepared to deal with a persistent connection.
        # It will try to read all remaining data from the socket,
        # which will block while the server waits for the next request.
        # So make sure the connection gets closed after the (only)
        # request.
        headers["Connection"] = "close"
        headers = {name.title(): val for name, val in headers.items()}
    
        if req._tunnel_host:
            tunnel_headers = {}
            proxy_auth_hdr = "Proxy-Authorization"
            if proxy_auth_hdr in headers:
                tunnel_headers[proxy_auth_hdr] = headers[proxy_auth_hdr]
                # Proxy-Authorization should not be sent to origin
                # server.
                del headers[proxy_auth_hdr]
            h.set_tunnel(req._tunnel_host, headers=tunnel_headers)
    
        try:
            try:
                h.request(req.get_method(), req.selector, req.data, headers,
                          encode_chunked=req.has_header('Transfer-encoding'))
            except OSError as err: # timeout error
>               raise URLError(err)
E               urllib.error.URLError: <urlopen error [Errno 104] Connection reset by peer>

/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:1351: URLError
----------------------------- Captured stdout call -----------------------------
Downloading https://github.com/pandas-dev/pandas/raw/bc17343f934a33dc231c8c74be95d8365537c376/setup.cfg to /home/runner/work/distutils/distutils/integration/setuptools/setuptools/tests/config/downloads/pandas-dev_pandasbc17343f934a33dc231c8c74be95d8365537c376_setup.cfg

[abravalheri]

Hi @jaraco, thank you very much for reporting this.
The second instance looks like a network problem, right?

In the setuptools repo we do some caching which may stabilise the network problems:

setuptools/.github/workflows/main.yml

Lines 104 to 114 in cb3f8d2

	- uses: actions/cache@v4
	id: cache
	with:
	path: setuptools/tests/config/downloads/*.cfg
	key: >-
	${{ hashFiles('setuptools/tests/config/setupcfg_examples.txt') }}-
	${{ hashFiles('setuptools/tests/config/downloads/*.py') }}
	- name: Populate download cache
	if: steps.cache.outputs.cache-hit != 'true'
	working-directory: setuptools/tests/config
	run: python -m downloads.preload setupcfg_examples.txt

---

Now I believe this test was more important in the beginning when setuptools adopted pyproject.toml, but now that is used in the wild, its utility has decreased a bit. If there is no drop in coverage, I think we could remove it (but I am not sure removing it will not result in loss of coverage).