Add Support for Library Auto-detection

[ladislas]

Note

This a work in progress, made available for testing purposes and collaboration.

Here is the first pull request regarding library auto-detection when one library uses another library, as discussed in #93

Goal

Right now we have to #include all the libraries used by the sketch and by any other library by hand. This can become quickly cumbersome when you use a lot a libraries...

With library auto-detection, no need to do that anymore.

Current status

It works. I've added an AUTO_LIB variable that you can set in your Makefile to:

• 1 - to activate auto-detection or use the Makefile as usual (with all libraries in the same sketch)
• 0 - to use the Makefile as usual

It works by listing every header file in your USER_LIB_PATH folder and including them while compiling.

Even though it's working, the solution is not very elegant.

[sej7278]

we have to #include all the libraries used by the sketch and by any other library by hand

not quite right, you #include the libraries you need in your sketch as you always would, you then just set ARDUINO_LIBS in your makefile. you don't have to #include all your libraries in every other library.

For example, probably my most complicated sketch has this in the makefile:

ARDUINO_LIBS = RF24 RF24Network Adafruit_Sensor Adafruit_BMP085_Unified \
   Wire DHT11 OneWire DallasTemperature SPI jeelib

and this in the sketch:

#include <SPI.h>
#include <RF24Network.h>
#include <RF24.h>
#include <OneWire.h>
#include <DallasTemperature.h>
#include <Ports.h>
#include <dht11.h>
#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BMP085_U.h>

none of those header files #includes any of the others.

[ladislas]

not quite right, you #include the libraries you need in your sketch as you always would, you then just set ARDUINO_LIBS in your makefile. you don't have to #include all your libraries in every other library.

@sej7278 you're right, I expressed myself poorly.

I do the same as you, but I don't have ARDUINO_LIBS in my Makefile. I just set USER_LIB_PATH, add the libs in my main sketch like your do, and voilà! In the main sketch, I have to reference all the libs used by all the libs. For example: I have a DriveSystem lib, that includes the Motor lib. In the main sketch, I have to include DriveSystem and Motor.

In the PR, with the AUTO_LIB set to 1, I can just include DriveSystem and it will include Motor during the compilation automatically.

lines 771-786 seem like a lot of duplicated code

yep I'll clean that up.

what are ARDUINO_LIBS_SRCS and AUTO_LIB for?

some tests.

[ladislas]

Hi guys! :)

So I spent the weekend working on it and I think I've got something working. :)

I spent all my Saturday trying to do some magic with the Arduino.mk but it got me nowhere near a working solution...

So I decided to write a little python script. It's really simple:

• it parses the sketch source(s) and extract all the #include "xxx.h" as MAIN_LIBS
• it then parses the MAIN_LIBS for other includes as LIBS_DEPS. It does that recursively, so if your library includes another library which in turn includes another library, it will work, as long as they are all in the USER_LIB_PATH

In Arduino.mk, I then sed the output to get the libraries names.

The main drawback is time: the python script takes a little less than a second to execute. But in return, if you use a lot of libraries, you only need to include the main libraries in your sketch, which is much better in term of style and time spent thinking of all the libraries dependencies. Of course you can include everything in your sketch as usual.

Maybe all of this can be done with pure Makefile style, but I was not able to figure out how... especially the recursive part.

I also took the liberty to enhance the compilation output a little. Two examples:

With everything included:

With just the main libraries included:

Hope you like it! Feedbacks and critics are more than welcome! :)

@sudar: I think you can merge it now so people can do some testing.

[sudar]

@ladislas Thanks for doing this and I have merged it into the auto-lib branch.

I am little concerned about the extra python script to do this. But let's see if enough people are interested in this.

[ladislas]

thanks @sudar ! :)

Are you concerned about using an external script? a python script? or my script in particular?

As I said, I'm not an expert and it might be possible to do everything in Makefile :)

[sej7278]

well we've already got a python script for ard-reset-arduino, but it does mean you should write a manpage for it if we're to add a "binary". i assume it will work standalone and doesn't have to be called from the makefile? it'll have to go in the bin/ directory in git and be runable from /usr/bin/ on linux.

on a related note i may have to dig out the old ard-parse-boards again to parse the 1.5-style boards.txt/platforms.txt/programmers.txt as make is going to be able to loop through arrays and such, sed's a bit too limiting. i'll do it in python not perl so just one dependency.

[sudar]

Are you concerned about using an external script? a python script? or my script in particular?

I was actually concerned about the time it takes for the python script to complete. But on second thought, I realized that it would be invoked only when auto-lib is explicitly enabled. So I guess it should be fine.

[ladislas]

@sudar: I removed the AUTO_LIB variable in my last PR. But I can add it back!

@sej7278: sure you can remove the .py extension.

And yes your right, for the moment it only works if the library has the same name as its folder.

I haven't had the time to make it work if the name differs, but I already see a problem: if two .h files have the same name in different folders, it will not work...

[ladislas]

Hi there! I think you can delete the auto-lib branch. I'm keeping my fork but won't have time to merge it back here for the moment :)

[lluiscampos]

Hi @sudar and @ladislas,

Extending the library auto-detection for library files is a very interesting feature. What is the remaining work to bring auto-lib branch to master? I am willing to work on it but I need some guidance.

I haven't had the time to make it work if the name differs,

I had a look at the script in auto-lib branch and I could take care of this.

but I already see a problem: if two .h files have the same name in different folders, it will not work...

Can you give some more detail on this?

Why not?

I can see that having two header files with the same name in different libraries can lead to problems, as we'll rely on ordering for the compiler choosing the right one. But as long as we keep this feature optional (iow, following Arduino IDE features) is up to the users to risk such problems. We should document this limitation and risk.

Generally speaking it is not a good practice to override header files, so your libraries shouldn't share filenames.

Thanks!

[ladislas]

@lluiscampos you should open a new issue to discuss this, it's not ideal to use a PR closed a long time ago.