Commit esp8266/Arduino@9790e1c break git installation #5789

luc-github · 2019-02-19T17:51:04Z

-## Basic Infos

[ x] This issue complies with the issue POLICY doc.
[ x] I have read the documentation at readthedocs and the issue is not addressed there.
[ x] I have tested that the issue is present in current master branch (aka latest git).
[x ] I have searched the issue tracker for a similar issue.
[N/A ] If there is a stack dump, I have decoded it.
[ x] I have filled out all fields below.

Platform

Hardware: [ESP-12]
Core Version: [latest git ]
Development Env: [Arduino IDE]
Operating System: [Windows 10 python 2.7.13 and 2.7.15]

Settings in IDE N/A

Problem Description

Commit 9790e1c cause git install to fail
update python 2.7.13 to 2.7.15 does not solve the issue

mcve

C:\Users\User\Documents\Arduino\hardware\esp8266com>git clone https://github.com/esp8266/Arduino.git esp8266
Cloning into 'esp8266'...
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 21186 (delta 6), reused 14 (delta 0), pack-reused 21158R
Receiving objects: 100% (21186/21186), 71.20 MiB | 1.49 MiB/s, done.

Resolving deltas: 100% (12571/12571), done.

C:\Users\User\Documents\Arduino\hardware\esp8266com>cd esp8266

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>git submodule update --init
Submodule 'libraries/SoftwareSerial' (https://github.com/plerup/espsoftwareserial.git) registered for path 'libraries/SoftwareSerial'
Submodule 'tools/esptool' (https://github.com/espressif/esptool.git) registered for path 'tools/esptool'
Submodule 'tools/pyserial' (https://github.com/pyserial/pyserial.git) registered for path 'tools/pyserial'
Submodule 'lwip2' (https://github.com/d-a-v/esp82xx-nonos-linklayer.git) registered for path 'tools/sdk/lwip2/builder'
Submodule 'tools/sdk/ssl/bearssl' (https://github.com/earlephilhower/bearssl-esp8266) registered for path 'tools/sdk/ssl/bearssl'
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/libraries/SoftwareSerial'...
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/tools/esptool'...
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/tools/pyserial'...
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/tools/sdk/lwip2/builder'...
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl'...
Submodule path 'libraries/SoftwareSerial': checked out '23ae000cb2cf4d5823a2744f6b8ae831575ff135'
Submodule path 'tools/esptool': checked out '9ad444a6e06e58833d5e6044c1d5f3eb3dd56023'
Submodule path 'tools/pyserial': checked out 'c54c81d933b847458d465cd77e96cd702ff2e7be'
Submodule path 'tools/sdk/lwip2/builder': checked out 'a501b5722bd050f41a9351e69ca75937150ec138'
Submodule path 'tools/sdk/ssl/bearssl': checked out '67786877341aac98c62e3b765fc64f4c49d81370'

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>cd tools

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading python-3.7.2.post1-embed-win32.zip
Done
Extracting dist/python-3.7.2.post1-embed-win32.zip
Traceback (most recent call last):
  File "get.py", line 121, in <module>
    main()
  File "get.py", line 118, in main
    get_tool(tool)
  File "get.py", line 84, in get_tool
    unpack(local_path, '.')
  File "get.py", line 61, in unpack
    rename_to = re.match(r'^([a-z][^\-]*\-*)+', dirname).group(0).strip('-')
AttributeError: 'NoneType' object has no attribute 'group'

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>

doing installation prior to commit 9790e1c works as expected

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>git checkout e7e7a4da177c8d511ba363bb09ccb50674cf2374
....
HEAD is now at e7e7a4da... prepare allman style (#5774)

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>cd tools

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading i686-w64-mingw32.esptool-f80ae31.zip
Done
Extracting dist/i686-w64-mingw32.esptool-f80ae31.zip
Downloading i686-w64-mingw32.xtensa-lx106-elf-20ed2b9c.zip
Done
Extracting dist/i686-w64-mingw32.xtensa-lx106-elf-20ed2b9c.zip
Downloading i686-w64-mingw32.mkspiffs-7fefeac.zip
Done
Extracting dist/i686-w64-mingw32.mkspiffs-7fefeac.zip

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>

The text was updated successfully, but these errors were encountered:

reaper7 · 2019-02-19T21:06:59Z

I have same problem:
my comment: 9790e1c#commitcomment-32366937

d-a-v · 2019-02-19T22:33:49Z

@reaper7 It's easier to follow issues when they are "issues".
If it persists after #5791, can you please open a new issue ?

reaper7 · 2019-02-20T06:49:12Z

#5791 fixed problem with downloading necessary files via get.py
but some more problems still exists #5792

luc-github · 2019-02-20T07:56:12Z

I can confirm download is Ok :

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>cd tools

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading python-3.7.2.post1-embed-win32.zip
Done
Extracting dist/python-3.7.2.post1-embed-win32.zip
Downloading i686-w64-mingw32.xtensa-lx106-elf-20ed2b9c.zip
Done
Extracting dist/i686-w64-mingw32.xtensa-lx106-elf-20ed2b9c.zip
Downloading i686-w64-mingw32.mkspiffs-7fefeac.zip
Done
Extracting dist/i686-w64-mingw32.mkspiffs-7fefeac.zip

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>

But I got issue when compiling :

Compiling core...
"C:\\Users\\User\\Documents\\Arduino\\hardware\\esp8266com\\esp8266/tools/python/python.exe" "C:\\Users\\User\\Documents\\Arduino\\hardware\\esp8266com\\esp8266/tools/signing.py" --mode header --publickey "C:\\Users\\User\\Documents\\gitlab\\ESP3D\\esp3d/public.key" --out "C:\\Users\\User\\AppData\\Local\\Temp\\arduino_build_368303/core/Updater_Signing.h"
Using library ESP8266WiFi at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi 
Using library ESP8266WebServer at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WebServer 
Using library EEPROM at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\EEPROM 
Using library ESP8266SSDP at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266SSDP 
Using library ESP8266mDNS at version 1.2 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266mDNS 
Using library ArduinoOTA at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ArduinoOTA 
Using library DNSServer at version 1.1.1 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\DNSServer 
Using library arduinoWebSockets at version 2.1.0 in folder: C:\Users\User\Documents\Arduino\libraries\arduinoWebSockets 
Using library Hash at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\Hash 
exec: "C:\\Users\\User\\Documents\\Arduino\\hardware\\esp8266com\\esp8266/tools/python/python.exe": file does not exist
Error compiling for board Generic ESP8266 Module.

which is true as python.exe is in tools directory:

MarkusAD · 2019-02-20T11:23:59Z

exec: "C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/python/python.exe": file does not exist

path refers to tools/python/python.exe but python.exe is in tools (with no python subdir).

luc-github · 2019-02-20T23:13:35Z

Yes this is what I describe in screen shots : error is correct not python.exe in python directory as it is directly in tools

luc-github · 2019-02-20T23:17:13Z

Looks like @d-a-v @earlephilhower there was some update upstream but it break again sorry
I do a full delete and full git download and

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>cd tools

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading python-3.7.2.post1-embed-win32.zip
Done200%
Hash mismatch for dist/python-3.7.2.post1-embed-win32.zip, delete the file and try again
Traceback (most recent call last):
  File "get.py", line 121, in <module>
    main()
  File "get.py", line 118, in main
    get_tool(tool)
  File "get.py", line 83, in get_tool
    raise RuntimeError()
RuntimeError

Delete the file in dir and redo


C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading python-3.7.2.post1-embed-win32.zip
Done200%
Hash mismatch for dist/python-3.7.2.post1-embed-win32.zip, delete the file and try again
Traceback (most recent call last):
  File "get.py", line 121, in <module>
    main()
  File "get.py", line 118, in main
    get_tool(tool)
  File "get.py", line 83, in get_tool
    raise RuntimeError()
RuntimeError

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>

earlephilhower · 2019-02-20T23:23:13Z

You need to apply the latest PR, it adjusts the SHA256 sig and the zip file to "v2". The old one doesn't work anyway, so the old file was removed from the repo to avoid other people ending up with messy tools dirs.

mcspr · 2019-02-20T23:25:20Z

regarding #5792 (comment) os.fork use, not to spread discussion across issues. this happened to be 🔥 hot topic while I was trying this... sigh

Trying out with subprocess.call([sys.executable, esptool_path, ...]) it does work, but with one quirk.

Because python is embedded, it does not accept PYTHONPATH environment var (i.e. sys.path). There is python/python37._pth that is used to load modules. By default it is like this:

python37.zip
.

So one can either add pyserial.zip line into this file and replace submodule with pyserial archive, or just symlink serial directory from tools/pyserial. Both should work on Windows.

luc-github · 2019-02-20T23:27:44Z

@earlephilhower I did not saw PR was not merged - will apply and report

earlephilhower · 2019-02-20T23:32:23Z

@mcspr that's a good idea. I think a cleaner one would be, since I have to build a new ZIP file for python because the -embed one does not have a subdir: Just add a pyserial installation to the python-embed.zip itself, dropping pyserial as a tool installation/git repo completely.

I had hoped to just take the python ZIP as-is, but that's obviously not possible now that I see it doesn't actually include a subdir and that makes Arduino go sideways.

luc-github · 2019-02-20T23:41:45Z

@earlephilhower I confirm :

PR fix installation issue on my side / Zip modification
Zip modification upstream fix also compilation issue with python path
problem of upload as described above is present

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/python/python.exe C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/upload.py C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/pyserial C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/esptool/esptool.py --chip esp8266 --port COM7 --baud 115200 --trace version --end --chip esp8266 --port COM7 --baud 115200 --trace write_flash 0x0 C:\Users\User\AppData\Local\Temp\arduino_build_189745/esp3d.ino.bin --end 
Traceback (most recent call last):
  File "C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/upload.py", line 24, in <module>
    pid  = os.fork()
AttributeError: module 'os' has no attribute 'fork'
AttributeError: module 'os' has no attribute 'fork'

luc-github · 2019-02-20T23:44:11Z

Thank you guys - Windows can be a pain ^_^

mcspr · 2019-02-21T02:32:40Z

subprocess support is easy enough:
master...mcspr:upload-nofork

Assuming that pyserial will be included in the python zip + submodule changes in the same PR. Prerequisite for now (using system pip):

esp8266/tools/python $ pip download pyserial
... Saved pyserial-3.4-py2.py3-none-any.whl ...
esp8266/tools/python $ echo pyserial-3.4-py2.py3-none-any.whl >> python37._pth

luc-github · 2019-02-23T14:14:35Z

I did not followed what fix it
Because seems there are several threads but I have tried today and seems ok
so I close issue
Thank you

luc-github changed the title ~~Commit https://github.com/esp8266/Arduino/commit/9790e1cb7cfa01ac74056fb835e6282af123aba0 break git installation~~ Commit esp8266/Arduino@9790e1c break git installation Feb 19, 2019

d-a-v mentioned this issue Feb 19, 2019

get.py: fix python regexp ('_' was missing) #5791

Merged

reaper7 mentioned this issue Feb 20, 2019

Commit esp8266/Arduino@9790e1c python issue #5792

Closed

6 tasks

luc-github closed this as completed Feb 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commit esp8266/Arduino@9790e1c break git installation #5789

Commit esp8266/Arduino@9790e1c break git installation #5789

luc-github commented Feb 19, 2019 •

edited

Loading

reaper7 commented Feb 19, 2019

d-a-v commented Feb 19, 2019

reaper7 commented Feb 20, 2019

luc-github commented Feb 20, 2019

MarkusAD commented Feb 20, 2019

luc-github commented Feb 20, 2019

luc-github commented Feb 20, 2019

earlephilhower commented Feb 20, 2019

mcspr commented Feb 20, 2019 •

edited

Loading

luc-github commented Feb 20, 2019

earlephilhower commented Feb 20, 2019

luc-github commented Feb 20, 2019 •

edited

Loading

luc-github commented Feb 20, 2019

mcspr commented Feb 21, 2019

luc-github commented Feb 23, 2019

Commit esp8266/Arduino@9790e1c break git installation #5789

Commit esp8266/Arduino@9790e1c break git installation #5789

Comments

luc-github commented Feb 19, 2019 • edited Loading

Platform

Settings in IDE N/A

Problem Description

mcve

reaper7 commented Feb 19, 2019

d-a-v commented Feb 19, 2019

reaper7 commented Feb 20, 2019

luc-github commented Feb 20, 2019

MarkusAD commented Feb 20, 2019

luc-github commented Feb 20, 2019

luc-github commented Feb 20, 2019

earlephilhower commented Feb 20, 2019

mcspr commented Feb 20, 2019 • edited Loading

luc-github commented Feb 20, 2019

earlephilhower commented Feb 20, 2019

luc-github commented Feb 20, 2019 • edited Loading

luc-github commented Feb 20, 2019

mcspr commented Feb 21, 2019

luc-github commented Feb 23, 2019

luc-github commented Feb 19, 2019 •

edited

Loading

mcspr commented Feb 20, 2019 •

edited

Loading

luc-github commented Feb 20, 2019 •

edited

Loading