diff options
Diffstat (limited to 'poky/bitbake')
19 files changed, 279 insertions, 164 deletions
diff --git a/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst b/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst index 722dc5a2cc..c8251d2a04 100644 --- a/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst +++ b/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst @@ -18,28 +18,32 @@ it. Obtaining BitBake ================= -See the :ref:`bitbake-user-manual/bitbake-user-manual-hello:obtaining bitbake` section for +See the :ref:`bitbake-user-manual/bitbake-user-manual-intro:obtaining bitbake` section for information on how to obtain BitBake. Once you have the source code on your machine, the BitBake directory appears as follows:: $ ls -al - total 100 - drwxrwxr-x. 9 wmat wmat 4096 Jan 31 13:44 . - drwxrwxr-x. 3 wmat wmat 4096 Feb 4 10:45 .. - -rw-rw-r--. 1 wmat wmat 365 Nov 26 04:55 AUTHORS - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 bin - drwxrwxr-x. 4 wmat wmat 4096 Jan 31 13:44 build - -rw-rw-r--. 1 wmat wmat 16501 Nov 26 04:55 ChangeLog - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 classes - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 conf - drwxrwxr-x. 3 wmat wmat 4096 Nov 26 04:55 contrib - -rw-rw-r--. 1 wmat wmat 17987 Nov 26 04:55 COPYING - drwxrwxr-x. 3 wmat wmat 4096 Nov 26 04:55 doc - -rw-rw-r--. 1 wmat wmat 69 Nov 26 04:55 .gitignore - -rw-rw-r--. 1 wmat wmat 849 Nov 26 04:55 HEADER - drwxrwxr-x. 5 wmat wmat 4096 Jan 31 13:44 lib - -rw-rw-r--. 1 wmat wmat 195 Nov 26 04:55 MANIFEST.in - -rw-rw-r--. 1 wmat wmat 2887 Nov 26 04:55 TODO + total 108 + drwxr-xr-x 9 fawkh 10000 4096 feb 24 12:10 . + drwx------ 36 fawkh 10000 4096 mar 2 17:00 .. + -rw-r--r-- 1 fawkh 10000 365 feb 24 12:10 AUTHORS + drwxr-xr-x 2 fawkh 10000 4096 feb 24 12:10 bin + -rw-r--r-- 1 fawkh 10000 16501 feb 24 12:10 ChangeLog + drwxr-xr-x 2 fawkh 10000 4096 feb 24 12:10 classes + drwxr-xr-x 2 fawkh 10000 4096 feb 24 12:10 conf + drwxr-xr-x 5 fawkh 10000 4096 feb 24 12:10 contrib + drwxr-xr-x 6 fawkh 10000 4096 feb 24 12:10 doc + drwxr-xr-x 8 fawkh 10000 4096 mar 2 16:26 .git + -rw-r--r-- 1 fawkh 10000 31 feb 24 12:10 .gitattributes + -rw-r--r-- 1 fawkh 10000 392 feb 24 12:10 .gitignore + drwxr-xr-x 13 fawkh 10000 4096 feb 24 12:11 lib + -rw-r--r-- 1 fawkh 10000 1224 feb 24 12:10 LICENSE + -rw-r--r-- 1 fawkh 10000 15394 feb 24 12:10 LICENSE.GPL-2.0-only + -rw-r--r-- 1 fawkh 10000 1286 feb 24 12:10 LICENSE.MIT + -rw-r--r-- 1 fawkh 10000 229 feb 24 12:10 MANIFEST.in + -rw-r--r-- 1 fawkh 10000 2413 feb 24 12:10 README + -rw-r--r-- 1 fawkh 10000 43 feb 24 12:10 toaster-requirements.txt + -rw-r--r-- 1 fawkh 10000 2887 feb 24 12:10 TODO At this point, you should have BitBake cloned to a directory that matches the previous listing except for dates and user names. @@ -52,7 +56,7 @@ directory to where your local BitBake files are and run the following command:: $ ./bin/bitbake --version - BitBake Build Tool Core version 1.23.0, bitbake version 1.23.0 + BitBake Build Tool Core version 2.3.1 The console output tells you what version you are running. @@ -130,23 +134,8 @@ Following is the complete "Hello World" example. directory. Run the ``bitbake`` command and see what it does:: $ bitbake - The BBPATH variable is not set and bitbake did not - find a conf/bblayers.conf file in the expected location. + ERROR: The BBPATH variable is not set and bitbake did not find a conf/bblayers.conf file in the expected location. Maybe you accidentally invoked bitbake from the wrong directory? - DEBUG: Removed the following variables from the environment: - GNOME_DESKTOP_SESSION_ID, XDG_CURRENT_DESKTOP, - GNOME_KEYRING_CONTROL, DISPLAY, SSH_AGENT_PID, LANG, no_proxy, - XDG_SESSION_PATH, XAUTHORITY, SESSION_MANAGER, SHLVL, - MANDATORY_PATH, COMPIZ_CONFIG_PROFILE, WINDOWID, EDITOR, - GPG_AGENT_INFO, SSH_AUTH_SOCK, GDMSESSION, GNOME_KEYRING_PID, - XDG_SEAT_PATH, XDG_CONFIG_DIRS, LESSOPEN, DBUS_SESSION_BUS_ADDRESS, - _, XDG_SESSION_COOKIE, DESKTOP_SESSION, LESSCLOSE, DEFAULTS_PATH, - UBUNTU_MENUPROXY, OLDPWD, XDG_DATA_DIRS, COLORTERM, LS_COLORS - - The majority of this output is specific to environment variables that - are not directly relevant to BitBake. However, the very first - message regarding the :term:`BBPATH` variable and the - ``conf/bblayers.conf`` file is relevant. When you run BitBake, it begins looking for metadata files. The :term:`BBPATH` variable is what tells BitBake where @@ -179,20 +168,14 @@ Following is the complete "Hello World" example. ``bitbake`` command again:: $ bitbake - ERROR: Traceback (most recent call last): - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped - return func(fn, *args) - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 173, in parse_config_file - return bb.parse.handle(fn, data, include) - File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 99, in handle - return h['handle'](fn, data, include) - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 120, in handle - abs_fn = resolve_file(fn, data) - File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 117, in resolve_file - raise IOError("file %s not found in %s" % (fn, bbpath)) - IOError: file conf/bitbake.conf not found in /home/scott-lenovo/hello - - ERROR: Unable to parse conf/bitbake.conf: file conf/bitbake.conf not found in /home/scott-lenovo/hello + ERROR: Unable to parse /home/scott-lenovo/bitbake/lib/bb/parse/__init__.py + Traceback (most recent call last): + File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 127, in resolve_file(fn='conf/bitbake.conf', d=<bb.data_smart.DataSmart object at 0x7f22919a3df0>): + if not newfn: + > raise IOError(errno.ENOENT, "file %s not found in %s" % (fn, bbpath)) + fn = newfn + FileNotFoundError: [Errno 2] file conf/bitbake.conf not found in <projectdirectory> + This sample output shows that BitBake could not find the ``conf/bitbake.conf`` file in the project directory. This file is @@ -254,18 +237,14 @@ Following is the complete "Hello World" example. exists, you can run the ``bitbake`` command again:: $ bitbake - ERROR: Traceback (most recent call last): - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped - return func(fn, *args) - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 177, in _inherit - bb.parse.BBHandler.inherit(bbclass, "configuration INHERITs", 0, data) - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 92, in inherit - include(fn, file, lineno, d, "inherit") - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 100, in include - raise ParseError("Could not %(error_out)s file %(fn)s" % vars(), oldfn, lineno) - ParseError: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass - - ERROR: Unable to parse base: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass + ERROR: Unable to parse /home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py + Traceback (most recent call last): + File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 67, in inherit(files=['base'], fn='configuration INHERITs', lineno=0, d=<bb.data_smart.DataSmart object at 0x7fab6815edf0>): + if not os.path.exists(file): + > raise ParseError("Could not inherit file %s" % (file), fn, lineno) + + bb.parse.ParseError: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass + In the sample output, BitBake could not find the ``classes/base.bbclass`` file. You need @@ -284,7 +263,10 @@ Following is the complete "Hello World" example. $ mkdir classes Move to the ``classes`` directory and then create the - ``base.bbclass`` file by inserting this single line: addtask build + ``base.bbclass`` file by inserting this single line:: + + addtask build + The minimal task that BitBake runs is the ``do_build`` task. This is all the example needs in order to build the project. Of course, the ``base.bbclass`` can have much more depending on which build @@ -328,10 +310,19 @@ Following is the complete "Hello World" example. BBFILES += "${LAYERDIR}/*.bb" BBFILE_COLLECTIONS += "mylayer" BBFILE_PATTERN_mylayer := "^${LAYERDIR_RE}/" + LAYERSERIES_CORENAMES = "hello_world_example" + LAYERSERIES_COMPAT_mylayer = "hello_world_example" For information on these variables, click on :term:`BBFILES`, - :term:`LAYERDIR`, :term:`BBFILE_COLLECTIONS` or :term:`BBFILE_PATTERN_mylayer <BBFILE_PATTERN>` - to go to the definitions in the glossary. + :term:`LAYERDIR`, :term:`BBFILE_COLLECTIONS`, :term:`BBFILE_PATTERN_mylayer <BBFILE_PATTERN>` + or :term:`LAYERSERIES_COMPAT` to go to the definitions in the glossary. + + .. note:: + + We are setting both LAYERSERIES_CORENAMES and LAYERSERIES_COMPAT in this particular case, because we + are using bitbake without OpenEmbedded. + You should usually just use LAYERSERIES_COMPAT to specify the OE-Core versions for which your layer + is compatible, and add the meta-openembedded layer to your project. You need to create the recipe file next. Inside your layer at the top-level, use an editor and create a recipe file named @@ -389,12 +380,14 @@ Following is the complete "Hello World" example. target:: $ bitbake printhello + Loading cache: 100% | + Loaded 0 entries from dependency cache. Parsing recipes: 100% |##################################################################################| - Time: 00:00:00 Parsing of 1 .bb files complete (0 cached, 1 parsed). 1 targets, 0 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies - NOTE: Preparing RunQueue - NOTE: Executing RunQueue Tasks + Initialising tasks: 100% |###############################################################################| + NOTE: No setscene tasks + NOTE: Executing Tasks ******************** * * * Hello, World! * diff --git a/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst index 09d09a8d9e..25620fdda1 100644 --- a/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -990,7 +990,7 @@ overview of their function and contents. ``bblayers.conf`` configuration file. To exclude a recipe from a world build using this variable, set the - variable to "1" in the recipe. + variable to "1" in the recipe. Set it to "0" to add it back to world build. .. note:: @@ -1096,6 +1096,29 @@ overview of their function and contents. variable is not available outside of ``layer.conf`` and references are expanded immediately when parsing of the file completes. + :term:`LAYERSERIES_COMPAT` + Lists the versions of the OpenEmbedded-Core (OE-Core) for which + a layer is compatible. Using the :term:`LAYERSERIES_COMPAT` variable + allows the layer maintainer to indicate which combinations of the + layer and OE-Core can be expected to work. The variable gives the + system a way to detect when a layer has not been tested with new + releases of OE-Core (e.g. the layer is not maintained). + + To specify the OE-Core versions for which a layer is compatible, use + this variable in your layer's ``conf/layer.conf`` configuration file. + For the list, use the Yocto Project release name (e.g. "kirkstone", + "mickledore"). To specify multiple OE-Core versions for the layer, use + a space-separated list:: + + LAYERSERIES_COMPAT_layer_root_name = "kirkstone mickledore" + + .. note:: + + Setting :term:`LAYERSERIES_COMPAT` is required by the Yocto Project + Compatible version 2 standard. + The OpenEmbedded build system produces a warning if the variable + is not set for any given layer. + :term:`LAYERVERSION` Optionally specifies the version of a layer as a single number. You can use this variable within diff --git a/poky/bitbake/doc/releases.rst b/poky/bitbake/doc/releases.rst index 6635032c01..6a9774d4e7 100644 --- a/poky/bitbake/doc/releases.rst +++ b/poky/bitbake/doc/releases.rst @@ -1,61 +1,57 @@ .. SPDX-License-Identifier: CC-BY-2.5 -=========================== - Supported Release Manuals -=========================== +================================= +BitBake Supported Release Manuals +================================= + +***************************** +Release Series 4.1 (langdale) +***************************** + +- :yocto_docs:`BitBake 2.2 User Manual </bitbake/2.2/>` + +***************************** +Release Series 4.0 (kirstone) +***************************** + +- :yocto_docs:`BitBake 2.0 User Manual </bitbake/2.0/>` + +**************************** +Release Series 3.1 (dunfell) +**************************** + +- :yocto_docs:`BitBake 1.46 User Manual </bitbake/1.46/>` + +================================ +BitBake Outdated Release Manuals +================================ ****************************** Release Series 3.4 (honister) ****************************** -- :yocto_docs:`3.4 BitBake User Manual </3.4/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.4.1 BitBake User Manual </3.4.1/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.4.2 BitBake User Manual </3.4.2/bitbake-user-manual/bitbake-user-manual.html>` +- :yocto_docs:`BitBake 1.52 User Manual </bitbake/1.52/>` ****************************** Release Series 3.3 (hardknott) ****************************** -- :yocto_docs:`3.3 BitBake User Manual </3.3/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.1 BitBake User Manual </3.3.1/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.2 BitBake User Manual </3.3.2/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.3 BitBake User Manual </3.3.3/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.4 BitBake User Manual </3.3.4/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.5 BitBake User Manual </3.3.5/bitbake-user-manual/bitbake-user-manual.html>` +- :yocto_docs:`BitBake 1.50 User Manual </bitbake/1.50/>` -**************************** -Release Series 3.1 (dunfell) -**************************** +******************************* +Release Series 3.2 (gatesgarth) +******************************* + +- :yocto_docs:`BitBake 1.48 User Manual </bitbake/1.48/>` + +******************************************* +Release Series 3.1 (dunfell first versions) +******************************************* - :yocto_docs:`3.1 BitBake User Manual </3.1/bitbake-user-manual/bitbake-user-manual.html>` - :yocto_docs:`3.1.1 BitBake User Manual </3.1.1/bitbake-user-manual/bitbake-user-manual.html>` - :yocto_docs:`3.1.2 BitBake User Manual </3.1.2/bitbake-user-manual/bitbake-user-manual.html>` - :yocto_docs:`3.1.3 BitBake User Manual </3.1.3/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.4 BitBake User Manual </3.1.4/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.5 BitBake User Manual </3.1.5/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.6 BitBake User Manual </3.1.6/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.7 BitBake User Manual </3.1.7/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.8 BitBake User Manual </3.1.8/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.9 BitBake User Manual </3.1.9/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.10 BitBake User Manual </3.1.10/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.11 BitBake User Manual </3.1.11/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.12 BitBake User Manual </3.1.12/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.13 BitBake User Manual </3.1.13/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.14 BitBake User Manual </3.1.14/bitbake-user-manual/bitbake-user-manual.html>` - -========================== - Outdated Release Manuals -========================== - -******************************* -Release Series 3.2 (gatesgarth) -******************************* - -- :yocto_docs:`3.2 BitBake User Manual </3.2/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.2.1 BitBake User Manual </3.2.1/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.2.2 BitBake User Manual </3.2.2/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.2.3 BitBake User Manual </3.2.3/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.2.4 BitBake User Manual </3.2.4/bitbake-user-manual/bitbake-user-manual.html>` ************************* Release Series 3.0 (zeus) diff --git a/poky/bitbake/lib/bb/build.py b/poky/bitbake/lib/bb/build.py index 5a1727116a..44d08f5c55 100644 --- a/poky/bitbake/lib/bb/build.py +++ b/poky/bitbake/lib/bb/build.py @@ -25,6 +25,7 @@ import bb import bb.msg import bb.process import bb.progress +from io import StringIO from bb import data, event, utils bblogger = logging.getLogger('BitBake') @@ -177,7 +178,9 @@ class StdoutNoopContextManager: @property def name(self): - return sys.stdout.name + if "name" in dir(sys.stdout): + return sys.stdout.name + return "<mem>" def exec_func(func, d, dirs = None): @@ -296,9 +299,21 @@ def exec_func_python(func, d, runfile, cwd=None): lineno = int(d.getVarFlag(func, "lineno", False)) bb.methodpool.insert_method(func, text, fn, lineno - 1) + if verboseStdoutLogging: + sys.stdout.flush() + sys.stderr.flush() + currout = sys.stdout + currerr = sys.stderr + sys.stderr = sys.stdout = execio = StringIO() comp = utils.better_compile(code, func, "exec_func_python() autogenerated") utils.better_exec(comp, {"d": d}, code, "exec_func_python() autogenerated") finally: + if verboseStdoutLogging: + execio.flush() + logger.plain("%s" % execio.getvalue()) + sys.stdout = currout + sys.stderr = currerr + execio.close() # We want any stdout/stderr to be printed before any other log messages to make debugging # more accurate. In some cases we seem to lose stdout/stderr entirely in logging tests without this. sys.stdout.flush() @@ -441,7 +456,11 @@ exit $ret if fakerootcmd: cmd = [fakerootcmd, runfile] - if verboseStdoutLogging: + # We only want to output to logger via LogTee if stdout is sys.__stdout__ (which will either + # be real stdout or subprocess PIPE or similar). In other cases we are being run "recursively", + # ie. inside another function, in which case stdout is already being captured so we don't + # want to Tee here as output would be printed twice, and out of order. + if verboseStdoutLogging and sys.stdout == sys.__stdout__: logfile = LogTee(logger, StdoutNoopContextManager()) else: logfile = StdoutNoopContextManager() diff --git a/poky/bitbake/lib/bb/cache.py b/poky/bitbake/lib/bb/cache.py index b309775bb9..10910a6809 100644 --- a/poky/bitbake/lib/bb/cache.py +++ b/poky/bitbake/lib/bb/cache.py @@ -216,7 +216,7 @@ class CoreRecipeInfo(RecipeInfoCommon): # Collect files we may need for possible world-dep # calculations - if not self.not_world: + if not bb.utils.to_boolean(self.not_world): cachedata.possible_world.append(fn) #else: # logger.debug2("EXCLUDE FROM WORLD: %s", fn) diff --git a/poky/bitbake/lib/bb/fetch2/__init__.py b/poky/bitbake/lib/bb/fetch2/__init__.py index cf65727a20..31729885ab 100644 --- a/poky/bitbake/lib/bb/fetch2/__init__.py +++ b/poky/bitbake/lib/bb/fetch2/__init__.py @@ -749,10 +749,13 @@ def subprocess_setup(): # SIGPIPE errors are known issues with gzip/bash signal.signal(signal.SIGPIPE, signal.SIG_DFL) -def get_autorev(d): - # only not cache src rev in autorev case +def mark_recipe_nocache(d): if d.getVar('BB_SRCREV_POLICY') != "cache": d.setVar('BB_DONT_CACHE', '1') + +def get_autorev(d): + mark_recipe_nocache(d) + d.setVar("__BBAUTOREV_SEEN", True) return "AUTOINC" def get_srcrev(d, method_name='sortable_revision'): @@ -769,7 +772,7 @@ def get_srcrev(d, method_name='sortable_revision'): that fetcher provides a method with the given name and the same signature as sortable_revision. """ - d.setVar("__BBSEENSRCREV", "1") + d.setVar("__BBSRCREV_SEEN", "1") recursion = d.getVar("__BBINSRCREV") if recursion: raise FetchError("There are recursive references in fetcher variables, likely through SRC_URI") @@ -1219,6 +1222,7 @@ def srcrev_internal_helper(ud, d, name): if srcrev == "INVALID" or not srcrev: raise FetchError("Please set a valid SRCREV for url %s (possible key names are %s, or use a ;rev=X URL parameter)" % (str(attempts), ud.url), ud.url) if srcrev == "AUTOINC": + d.setVar("__BBAUTOREV_ACTED_UPON", True) srcrev = ud.method.latest_revision(ud, d, name) return srcrev @@ -1291,18 +1295,13 @@ class FetchData(object): if checksum_name in self.parm: checksum_expected = self.parm[checksum_name] - elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az"]: + elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate"]: checksum_expected = None else: checksum_expected = d.getVarFlag("SRC_URI", checksum_name) setattr(self, "%s_expected" % checksum_id, checksum_expected) - for checksum_id in CHECKSUM_LIST: - configure_checksum(checksum_id) - - self.ignore_checksums = False - self.names = self.parm.get("name",'default').split(',') self.method = None @@ -1324,6 +1323,11 @@ class FetchData(object): if hasattr(self.method, "urldata_init"): self.method.urldata_init(self, d) + for checksum_id in CHECKSUM_LIST: + configure_checksum(checksum_id) + + self.ignore_checksums = False + if "localpath" in self.parm: # if user sets localpath for file, use it instead. self.localpath = self.parm["localpath"] diff --git a/poky/bitbake/lib/bb/fetch2/crate.py b/poky/bitbake/lib/bb/fetch2/crate.py index f091200dd9..590dc9c126 100644 --- a/poky/bitbake/lib/bb/fetch2/crate.py +++ b/poky/bitbake/lib/bb/fetch2/crate.py @@ -33,7 +33,7 @@ class Crate(Wget): return ud.type in ['crate'] def recommends_checksum(self, urldata): - return False + return True def urldata_init(self, ud, d): """ @@ -56,8 +56,10 @@ class Crate(Wget): if len(parts) < 5: raise bb.fetch2.ParameterError("Invalid URL: Must be crate://HOST/NAME/VERSION", ud.url) - # last field is version - version = parts[len(parts) - 1] + # version is expected to be the last token + # but ignore possible url parameters which will be used + # by the top fetcher class + version, _, _ = parts[len(parts) -1].partition(";") # second to last field is name name = parts[len(parts) - 2] # host (this is to allow custom crate registries to be specified @@ -69,7 +71,8 @@ class Crate(Wget): ud.url = "https://%s/%s/%s/download" % (host, name, version) ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version) - ud.parm['name'] = name + if 'name' not in ud.parm: + ud.parm['name'] = name logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename'])) diff --git a/poky/bitbake/lib/bb/fetch2/git.py b/poky/bitbake/lib/bb/fetch2/git.py index 5bb8393133..2a3c06fe4e 100644 --- a/poky/bitbake/lib/bb/fetch2/git.py +++ b/poky/bitbake/lib/bb/fetch2/git.py @@ -367,9 +367,13 @@ class Git(FetchMethod): # If the repo still doesn't exist, fallback to cloning it if not os.path.exists(ud.clonedir): - # We do this since git will use a "-l" option automatically for local urls where possible + # We do this since git will use a "-l" option automatically for local urls where possible, + # but it doesn't work when git/objects is a symlink, only works when it is a directory. if repourl.startswith("file://"): - repourl = repourl[7:] + repourl_path = repourl[7:] + objects = os.path.join(repourl_path, 'objects') + if os.path.isdir(objects) and not os.path.islink(objects): + repourl = repourl_path clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir) if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) @@ -733,11 +737,11 @@ class Git(FetchMethod): """ Compute the HEAD revision for the url """ - if not d.getVar("__BBSEENSRCREV"): + if not d.getVar("__BBSRCREV_SEEN"): raise bb.fetch2.FetchError("Recipe uses a floating tag/branch '%s' for repo '%s' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE)." % (ud.unresolvedrev[name], ud.host+ud.path)) # Ensure we mark as not cached - bb.fetch2.get_autorev(d) + bb.fetch2.mark_recipe_nocache(d) output = self._lsremote(ud, d, "") # Tags of the form ^{} may not work, need to fallback to other form diff --git a/poky/bitbake/lib/bb/fetch2/local.py b/poky/bitbake/lib/bb/fetch2/local.py index 0bb987c644..5332faafe2 100644 --- a/poky/bitbake/lib/bb/fetch2/local.py +++ b/poky/bitbake/lib/bb/fetch2/local.py @@ -72,7 +72,7 @@ class Local(FetchMethod): filespath = d.getVar('FILESPATH') if filespath: locations = filespath.split(":") - msg = "Unable to find file " + urldata.url + " anywhere. The paths that were searched were:\n " + "\n ".join(locations) + msg = "Unable to find file " + urldata.url + " anywhere to download to " + urldata.localpath + ". The paths that were searched were:\n " + "\n ".join(locations) raise FetchError(msg) return True diff --git a/poky/bitbake/lib/bb/fetch2/sftp.py b/poky/bitbake/lib/bb/fetch2/sftp.py index f87f292e5d..7884cce949 100644 --- a/poky/bitbake/lib/bb/fetch2/sftp.py +++ b/poky/bitbake/lib/bb/fetch2/sftp.py @@ -103,7 +103,7 @@ class SFTP(FetchMethod): if path[:3] == '/~/': path = path[3:] - remote = '%s%s:%s' % (user, urlo.hostname, path) + remote = '"%s%s:%s"' % (user, urlo.hostname, path) cmd = '%s %s %s %s' % (basecmd, port, remote, lpath) diff --git a/poky/bitbake/lib/bb/fetch2/wget.py b/poky/bitbake/lib/bb/fetch2/wget.py index 859b4f99e2..dc88317610 100644 --- a/poky/bitbake/lib/bb/fetch2/wget.py +++ b/poky/bitbake/lib/bb/fetch2/wget.py @@ -369,15 +369,7 @@ class Wget(FetchMethod): with opener.open(r, timeout=30) as response: pass - except urllib.error.URLError as e: - if try_again: - logger.debug2("checkstatus: trying again") - return self.checkstatus(fetch, ud, d, False) - else: - # debug for now to avoid spamming the logs in e.g. remote sstate searches - logger.debug2("checkstatus() urlopen failed: %s" % e) - return False - except ConnectionResetError as e: + except (urllib.error.URLError, ConnectionResetError, TimeoutError) as e: if try_again: logger.debug2("checkstatus: trying again") return self.checkstatus(fetch, ud, d, False) diff --git a/poky/bitbake/lib/bb/parse/ast.py b/poky/bitbake/lib/bb/parse/ast.py index 375ba3cb79..6441c5cf7c 100644 --- a/poky/bitbake/lib/bb/parse/ast.py +++ b/poky/bitbake/lib/bb/parse/ast.py @@ -400,6 +400,9 @@ def finalize(fn, d, variant = None): d.setVar('BBINCLUDED', bb.parse.get_file_depends(d)) + if d.getVar('__BBAUTOREV_SEEN') and d.getVar('__BBSRCREV_SEEN') and not d.getVar("__BBAUTOREV_ACTED_UPON"): + bb.fatal("AUTOREV/SRCPV set too late for the fetcher to work properly, please set the variables earlier in parsing. Erroring instead of later obtuse build failures.") + bb.event.fire(bb.event.RecipeParsed(fn), d) finally: bb.event.set_handlers(saved_handlers) diff --git a/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 3076067287..05c627ec8b 100644 --- a/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py +++ b/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py @@ -21,7 +21,7 @@ __config_regexp__ = re.compile( r""" ^ (?P<exp>export\s+)? (?P<var>[a-zA-Z0-9\-_+.${}/~:]+?) - (\[(?P<flag>[a-zA-Z0-9\-_+.]+)\])? + (\[(?P<flag>[a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@]+)\])? \s* ( (?P<colon>:=) | @@ -45,7 +45,7 @@ __include_regexp__ = re.compile( r"include\s+(.+)" ) __require_regexp__ = re.compile( r"require\s+(.+)" ) __export_regexp__ = re.compile( r"export\s+([a-zA-Z0-9\-_+.${}/~]+)$" ) __unset_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)$" ) -__unset_flag_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)\[([a-zA-Z0-9\-_+.]+)\]$" ) +__unset_flag_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)\[([a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@]+)\]$" ) __addpylib_regexp__ = re.compile(r"addpylib\s+(.+)\s+(.+)" ) def init(data): diff --git a/poky/bitbake/lib/bb/server/xmlrpcserver.py b/poky/bitbake/lib/bb/server/xmlrpcserver.py index 2e65dc34a9..04b0b17db1 100644 --- a/poky/bitbake/lib/bb/server/xmlrpcserver.py +++ b/poky/bitbake/lib/bb/server/xmlrpcserver.py @@ -118,7 +118,7 @@ class BitBakeXMLRPCServerCommands(): """ Run a cooker command on the server """ - return self.server.cooker.command.runCommand(command, self.server, self.server.readonly) + return self.server.cooker.command.runCommand(command, self.server.parent, self.server.readonly) def getEventHandle(self): return self.event_handle diff --git a/poky/bitbake/lib/bb/tests/fetch.py b/poky/bitbake/lib/bb/tests/fetch.py index 73eefc5938..da67168900 100644 --- a/poky/bitbake/lib/bb/tests/fetch.py +++ b/poky/bitbake/lib/bb/tests/fetch.py @@ -785,7 +785,7 @@ class FetcherLocalTest(FetcherTest): # Fetch and check revision self.d.setVar("SRCREV", "AUTOINC") - self.d.setVar("__BBSEENSRCREV", "1") + self.d.setVar("__BBSRCREV_SEEN", "1") url = "git://" + self.gitdir + ";branch=master;protocol=file;" + suffix fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() @@ -1654,7 +1654,7 @@ class GitShallowTest(FetcherTest): self.d.setVar('BB_GIT_SHALLOW', '1') self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '0') self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1') - self.d.setVar("__BBSEENSRCREV", "1") + self.d.setVar("__BBSRCREV_SEEN", "1") def assertRefs(self, expected_refs, cwd=None): if cwd is None: @@ -2218,7 +2218,7 @@ class GitLfsTest(FetcherTest): self.d.setVar('SRCREV', '${AUTOREV}') self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}') - self.d.setVar("__BBSEENSRCREV", "1") + self.d.setVar("__BBSRCREV_SEEN", "1") bb.utils.mkdirhier(self.srcdir) self.git_init(cwd=self.srcdir) @@ -2377,6 +2377,13 @@ class CrateTest(FetcherTest): d = self.d fetcher = bb.fetch2.Fetch(uris, self.d) + ud = fetcher.ud[fetcher.urls[0]] + + self.assertIn("name", ud.parm) + self.assertEqual(ud.parm["name"], "glob") + self.assertIn("downloadfilename", ud.parm) + self.assertEqual(ud.parm["downloadfilename"], "glob-0.2.11.crate") + fetcher.download() fetcher.unpack(self.tempdir) self.assertEqual(sorted(os.listdir(self.tempdir)), ['cargo_home', 'download' , 'unpacked']) @@ -2385,6 +2392,30 @@ class CrateTest(FetcherTest): self.assertTrue(os.path.exists(self.tempdir + "/cargo_home/bitbake/glob-0.2.11/src/lib.rs")) @skipIfNoNetwork() + def test_crate_url_params(self): + + uri = "crate://crates.io/aho-corasick/0.7.20;name=aho-corasick-renamed" + self.d.setVar('SRC_URI', uri) + + uris = self.d.getVar('SRC_URI').split() + d = self.d + + fetcher = bb.fetch2.Fetch(uris, self.d) + ud = fetcher.ud[fetcher.urls[0]] + + self.assertIn("name", ud.parm) + self.assertEqual(ud.parm["name"], "aho-corasick-renamed") + self.assertIn("downloadfilename", ud.parm) + self.assertEqual(ud.parm["downloadfilename"], "aho-corasick-0.7.20.crate") + + fetcher.download() + fetcher.unpack(self.tempdir) + self.assertEqual(sorted(os.listdir(self.tempdir)), ['cargo_home', 'download' , 'unpacked']) + self.assertEqual(sorted(os.listdir(self.tempdir + "/download")), ['aho-corasick-0.7.20.crate', 'aho-corasick-0.7.20.crate.done']) + self.assertTrue(os.path.exists(self.tempdir + "/cargo_home/bitbake/aho-corasick-0.7.20/.cargo-checksum.json")) + self.assertTrue(os.path.exists(self.tempdir + "/cargo_home/bitbake/aho-corasick-0.7.20/src/lib.rs")) + + @skipIfNoNetwork() def test_crate_url_multi(self): uri = "crate://crates.io/glob/0.2.11 crate://crates.io/time/0.1.35" @@ -2394,6 +2425,19 @@ class CrateTest(FetcherTest): d = self.d fetcher = bb.fetch2.Fetch(uris, self.d) + ud = fetcher.ud[fetcher.urls[0]] + + self.assertIn("name", ud.parm) + self.assertEqual(ud.parm["name"], "glob") + self.assertIn("downloadfilename", ud.parm) + self.assertEqual(ud.parm["downloadfilename"], "glob-0.2.11.crate") + + ud = fetcher.ud[fetcher.urls[1]] + self.assertIn("name", ud.parm) + self.assertEqual(ud.parm["name"], "time") + self.assertIn("downloadfilename", ud.parm) + self.assertEqual(ud.parm["downloadfilename"], "time-0.1.35.crate") + fetcher.download() fetcher.unpack(self.tempdir) self.assertEqual(sorted(os.listdir(self.tempdir)), ['cargo_home', 'download' , 'unpacked']) @@ -2403,6 +2447,18 @@ class CrateTest(FetcherTest): self.assertTrue(os.path.exists(self.tempdir + "/cargo_home/bitbake/time-0.1.35/.cargo-checksum.json")) self.assertTrue(os.path.exists(self.tempdir + "/cargo_home/bitbake/time-0.1.35/src/lib.rs")) + @skipIfNoNetwork() + def test_crate_incorrect_cksum(self): + uri = "crate://crates.io/aho-corasick/0.7.20" + self.d.setVar('SRC_URI', uri) + self.d.setVarFlag("SRC_URI", "aho-corasick.sha256sum", hashlib.sha256("Invalid".encode("utf-8")).hexdigest()) + + uris = self.d.getVar('SRC_URI').split() + + fetcher = bb.fetch2.Fetch(uris, self.d) + with self.assertRaisesRegexp(bb.fetch2.FetchError, "Fetcher failure for URL"): + fetcher.download() + class NPMTest(FetcherTest): def skipIfNoNpm(): import shutil @@ -2912,7 +2968,7 @@ class GitSharedTest(FetcherTest): super(GitSharedTest, self).setUp() self.recipe_url = "git://git.openembedded.org/bitbake;branch=master" self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40') - self.d.setVar("__BBSEENSRCREV", "1") + self.d.setVar("__BBSRCREV_SEEN", "1") @skipIfNoNetwork() def test_shared_unpack(self): diff --git a/poky/bitbake/lib/bb/tests/parse.py b/poky/bitbake/lib/bb/tests/parse.py index ee7f2534f1..d27c7c6f15 100644 --- a/poky/bitbake/lib/bb/tests/parse.py +++ b/poky/bitbake/lib/bb/tests/parse.py @@ -218,3 +218,24 @@ VAR = " \\ with self.assertRaises(bb.BBHandledException): d = bb.parse.handle(f.name, self.d)[''] + + at_sign_in_var_flag = """ +A[flag@.service] = "nonet" +B[flag@.target] = "ntb" + +unset A[flag@.service] +""" + def test_parse_at_sign_in_var_flag(self): + f = self.parsehelper(self.at_sign_in_var_flag) + d = bb.parse.handle(f.name, self.d)[''] + self.assertEqual(d.getVar("A"), None) + self.assertEqual(d.getVar("B"), None) + self.assertEqual(d.getVarFlag("A","flag@.service"), None) + self.assertEqual(d.getVarFlag("B","flag@.target"), "ntb") + + def test_parse_invalid_at_sign_in_var_flag(self): + invalid_at_sign = self.at_sign_in_var_flag.replace("B[f", "B[@f") + f = self.parsehelper(invalid_at_sign) + with self.assertRaises(bb.parse.ParseError): + d = bb.parse.handle(f.name, self.d)[''] + diff --git a/poky/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py b/poky/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py index 69f557657a..71afe3914e 100755 --- a/poky/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py +++ b/poky/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py @@ -35,18 +35,19 @@ verbose = False # [Codename, Yocto Project Version, Release Date, Current Version, Support Level, Poky Version, BitBake branch] current_releases = [ # Release slot #1 - ['Kirkstone','4.0','April 2022','4.0.6 (December 2022)','Stable - Long Term Support (until Apr. 2024)','','2.0'], + ['Kirkstone','4.0','April 2022','4.0.8 (March 2023)','Stable - Long Term Support (until Apr. 2024)','','2.0'], # Release slot #2 'local' ['HEAD','HEAD','','Local Yocto Project','HEAD','','HEAD'], # Release slot #3 'master' ['Master','master','','Yocto Project master','master','','master'], # Release slot #4 - ['Langdale','4.1','October 2022','4.1.2 (January 2023)','Support for 7 months (until May 2023)','','2.2'], + ['Mickledore','4.2','April 2023','4.2.0 (April 2023)','Support for 7 months (until October 2023)','','2.4'], +# ['Langdale','4.1','October 2022','4.1.2 (January 2023)','Support for 7 months (until May 2023)','','2.2'], # ['Honister','3.4','October 2021','3.4.2 (February 2022)','Support for 7 months (until May 2022)','26.0','1.52'], # ['Hardknott','3.3','April 2021','3.3.5 (March 2022)','Stable - Support for 13 months (until Apr. 2022)','25.0','1.50'], # ['Gatesgarth','3.2','Oct 2020','3.2.4 (May 2021)','EOL','24.0','1.48'], # Optional Release slot #5 - ['Dunfell','3.1','April 2020','3.1.22 (January 2023)','Stable - Long Term Support (until Apr. 2024)','23.0','1.46'], + ['Dunfell','3.1','April 2020','3.1.23 (February 2023)','Stable - Long Term Support (until Apr. 2024)','23.0','1.46'], ] default_poky_layers = [ diff --git a/poky/bitbake/lib/toaster/orm/fixtures/oe-core.xml b/poky/bitbake/lib/toaster/orm/fixtures/oe-core.xml index 615e88aba1..950f2a98af 100644 --- a/poky/bitbake/lib/toaster/orm/fixtures/oe-core.xml +++ b/poky/bitbake/lib/toaster/orm/fixtures/oe-core.xml @@ -23,9 +23,9 @@ <field type="CharField" name="branch">master</field> </object> <object model="orm.bitbakeversion" pk="4"> - <field type="CharField" name="name">langdale</field> + <field type="CharField" name="name">mickledore</field> <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field> - <field type="CharField" name="branch">2.2</field> + <field type="CharField" name="branch">2.4</field> </object> <object model="orm.bitbakeversion" pk="5"> <field type="CharField" name="name">dunfell</field> @@ -56,11 +56,11 @@ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/\">OpenEmbedded master</a> branch.</field> </object> <object model="orm.release" pk="4"> - <field type="CharField" name="name">langdale</field> - <field type="CharField" name="description">Openembedded Langdale</field> + <field type="CharField" name="name">mickledore</field> + <field type="CharField" name="description">Openembedded Mickledore</field> <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">4</field> - <field type="CharField" name="branch_name">langdale</field> - <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/?h=langdale\">OpenEmbedded Langdale</a> branch.</field> + <field type="CharField" name="branch_name">mickledore</field> + <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/?h=mickledore\">OpenEmbedded Mickledore</a> branch.</field> </object> <object model="orm.release" pk="5"> <field type="CharField" name="name">dunfell</field> diff --git a/poky/bitbake/lib/toaster/orm/fixtures/poky.xml b/poky/bitbake/lib/toaster/orm/fixtures/poky.xml index 04e12f96fd..121e52fd45 100644 --- a/poky/bitbake/lib/toaster/orm/fixtures/poky.xml +++ b/poky/bitbake/lib/toaster/orm/fixtures/poky.xml @@ -26,9 +26,9 @@ <field type="CharField" name="dirpath">bitbake</field> </object> <object model="orm.bitbakeversion" pk="4"> - <field type="CharField" name="name">langdale</field> + <field type="CharField" name="name">mickledore</field> <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field> - <field type="CharField" name="branch">langdale</field> + <field type="CharField" name="branch">mickledore</field> <field type="CharField" name="dirpath">bitbake</field> </object> <object model="orm.bitbakeversion" pk="5"> @@ -62,11 +62,11 @@ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/">Yocto Project Master branch</a>.</field> </object> <object model="orm.release" pk="4"> - <field type="CharField" name="name">langdale</field> - <field type="CharField" name="description">Yocto Project 4.1 "Langdale"</field> + <field type="CharField" name="name">mickledore</field> + <field type="CharField" name="description">Yocto Project 4.2 "Mickledore"</field> <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">4</field> - <field type="CharField" name="branch_name">langdale</field> - <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=langdale">Yocto Project Langdale branch</a>.</field> + <field type="CharField" name="branch_name">mickledore</field> + <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=mickledore">Yocto Project Mickledore branch</a>.</field> </object> <object model="orm.release" pk="5"> <field type="CharField" name="name">dunfell</field> @@ -177,7 +177,7 @@ <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field> <field type="IntegerField" name="layer_source">0</field> <field rel="ManyToOneRel" to="orm.release" name="release">4</field> - <field type="CharField" name="branch">langdale</field> + <field type="CharField" name="branch">mickledore</field> <field type="CharField" name="dirpath">meta</field> </object> <object model="orm.layer_version" pk="5"> @@ -222,7 +222,7 @@ <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field> <field type="IntegerField" name="layer_source">0</field> <field rel="ManyToOneRel" to="orm.release" name="release">4</field> - <field type="CharField" name="branch">langdale</field> + <field type="CharField" name="branch">mickledore</field> <field type="CharField" name="dirpath">meta-poky</field> </object> <object model="orm.layer_version" pk="10"> @@ -267,7 +267,7 @@ <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field> <field type="IntegerField" name="layer_source">0</field> <field rel="ManyToOneRel" to="orm.release" name="release">4</field> - <field type="CharField" name="branch">langdale</field> + <field type="CharField" name="branch">mickledore</field> <field type="CharField" name="dirpath">meta-yocto-bsp</field> </object> <object model="orm.layer_version" pk="15"> |