diff options
author | Andrew Geissler <geissonator@yahoo.com> | 2020-06-27 08:28:28 +0300 |
---|---|---|
committer | Andrew Geissler <geissonator@yahoo.com> | 2020-06-27 08:32:13 +0300 |
commit | d25ed3241ddffad58c7a52e45e388e6c48d5123a (patch) | |
tree | b097477c5b9204689d35c06f5761b1767093b338 /poky/bitbake/lib/bb/fetch2 | |
parent | c87764fefff10735006a31fab72d76c243a3eb40 (diff) | |
download | openbmc-d25ed3241ddffad58c7a52e45e388e6c48d5123a.tar.xz |
poky: subtree update:26ae42ded7..5951cbcabe
Alex Kiernan (1):
recipetool: Fix list concatenation when using edit
Alexander Kanavin (4):
apr-util: make gdbm optional
gobject-introspection: add a patch to fix a build race
icu: merge .inc into main recipe
icu: make filtered data generation optional, serial and off by default
Alexandru N. Onea (3):
bitbake: perforce: add basic progress handler for perforce
bitbake: perforce: add local path handling SRC_URI options
bitbake: bitbake-user-manual: update perforce fetcher docs
Andreas M?ller (1):
meson.bbclass: avoid unexpected operating-system names
Andreas Müller (6):
boost: Add upstream patch to fix build on depending projects
libinput: upgrade 1.15.5 -> 1.15.6
sqlite3: upgrade 3.32.2 -> 3.32.3
desktop-file-utils: upgrade 0.24 -> 0.26
file: upgrade 5.38 -> 5.39
ffmpeg: upgrade 4.2.3 -> 4.3
Andrej Valek (1):
oeqa/runtime/cases/ptest: Make output content path absolute
Andrew Geissler (1):
meson: backport library ordering fix
Armin Kuster (1):
libuv: move from meta-oe to core for bind update
Arthur She (1):
igt-gpu-tools: add new package
Changqing Li (1):
mime.bbclass: fix post install scriptlet error
Chen Qi (1):
systemd-serialgetty: do not use BindsTo
Daniel McGregor (3):
sign_rpm.bbclass: ignore thread count
systemd-conf: Accept MTU from DHCP
buildhistory-collect-srcrevs: sort directories
He Zhe (1):
ltp: Fix copy_file_rang02 for 32-bit arches
Hongxu Jia (1):
libmodulemd: switch branch master -> main
Jacob Kroon (5):
bitbake: lib/bb/utils.py: Do not preserve TERM in the environment
bitbake: bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example
bitbake.conf: Remove TERM from default BB_HASHBASE_WHITELIST
grub: Remove native version of grub-efi
distro_alias: Remove unused grub-efi distro aliases
Jens Rehsack (1):
u-boot: avoid blind merging all *.cfg
Joe Slater (1):
systemd: fix CVE-2020-13776
Joshua Watt (5):
sstatesig: Account for all dataCaches being passed
bitbake: bitbake: cache: Fix error message with bad multiconfig
wic: Fix error message when reporting invalid offset
classes/archiver: Create patched archive before configuring
bitbake: cache: Bump cache version
Konrad Weihmann (3):
oeqa/runtime: Add OERequirePackage decorator
bitbake: cookerdata: Add BBFILES_DYNAMIC inverse mode
bitbake: bitbake-user-manual: Add BBFILES_DYNAMIC
Mark Morton (2):
New source files and Makefile update for Test Manual
test-manual: Fixed codeblock formatting
Martin Jansa (1):
net-tools: backport a patch from upstream to use the same ifconfig format as debian/ubuntu
Mingli Yu (3):
python3: add the rdepends for python3-misc
python3: add rdepends for python3-idle
python3-dbusmock: add the missing rdepends
Otavio Salvador (2):
systemd: Sync systemd-serialgetty@.service with upstream
mtd-utils: Fix return value of ubiformat
Ovidiu Panait (2):
dbus-test: Remove EXTRA_OECONF_X configs
dbus,dbus-test: Move common parts to dbus.inc
Paul Barker (2):
bitbake: fetch2/gitsm: Mark srcrev as fetched once all submodules are processed
bitbake: fetch2/gitsm: Make need_update() process submodules
Paul Eggleton (5):
graph-tool: switch to argparse
graph-tool: add filter subcommand
dpkg-native: rebase and reinstate fix for "tar: file changed as we read it"
shadow-sysroot: drop unused SRC_URI checksums
devtool: fix typo
Peter Kjellerstedt (1):
relocatable.bbclass: Avoid an exception if an empty pkgconfig dir exist
Pierre-Jean Texier (3):
diffoscope: upgrade 146 -> 147
ell: upgrade 0.31 -> 0.32
curl: upgrade 7.70.0 -> 7.71.0
Rasmus Villemoes (1):
curl: add debug info
Richard Purdie (15):
buildhistory: Add simplistic file move detection
bitbake: bin/bitbake: Update to next series release version
perl: Fix host specific modules problems
sanity.conf: Require bitbake 1.47.0 as the minimum version
patchelf: Upgrade 0.10 -> 0.11
test-manual: Add SPDX license headers
Makefile: Drop obsolete edison/denzil branch conditionals
bitbake: tests/fetch: Switch from git.infradead.org to a YP mirror
pseudo: Fix attr errors due to incorrect library resolution issues
oeqa/selftest/runcmd: Add better debug for thread count mismatch failures
oeqa/utils/command: Improve stdin handling in runCmd
vulkan-headers: Fix upstream branch deletion issue
recipes: Fix Upstream-Status Accepted -> Backport
scripts/install-buildtools: Update to 3.2 M1 buildtools
scripts/install-buildtools: Handle new format checksum files
Robert P. J. Day (1):
python: use official "pypi.org" URLs for HOMEPAGE
Ross Burton (8):
install-buildtools: fail if an error occurs
install-buildtools: remove hardcoded x86-64 architecture
install-buildtools: add option to disable checksum validation
common-licenses: add BSD-2-Clause-Patent
gstreamer1.0-plugins-bad: add support for vdpau
go-binary-native: add binary Go to bootstrap
tcmode-default: use go-binary-native by default
go-native: merge bb/inc and add comment
Ryan Rowe (1):
python3: fix PGO for non-reproducible biniaries
Sakib Sajal (1):
qemu: uprev v4.2.0 -> v5.0.0
Samuli Piippo (2):
cmake: allow chainloading of the toolchain file
perl: use relative paths in the perl wrapper
Steve Sakoman (1):
buildtools-tarball: export OPENSSL_CONF in environment setup
Tanu Kaskinen (1):
pulseaudio: remove unnecessary libltdl copying
Trevor Gamblin (1):
python3-setuptools: patch entrypoints for faster initialization
Tuomas Salokanto (1):
recipetool: create: fix SRCBRANCH not being passed to params
Valentin Longchamp (2):
tools-profile: disable valgrind for powerpc soft-float
valgrind: disable it for powerpc soft-float
Wang Mingyu (5):
powertop: upgrade 2.12 -> 2.13
man-db: upgrade 2.9.2 -> 2.9.3
valgrind: upgrade 3.16.0 -> 3.16.1
man-pages: upgrade 5.06 -> 5.07
harfbuzz: upgrade 2.6.7 -> 2.6.8
Yi Zhao (2):
iptables: fix invalid symbolic link for ip6tables-apply
iptables: split iptables-apply to its own package
Yongxin Liu (1):
linux-firmware: add ice for Intel E800 series driver
Yuki Hoshino (1):
sysvinit-inittab: Add support for tty devices with 10 or more number.
akuster (9):
bind: update to 9.11.19
adt-manual: Add SPDX license headers
bsp-guide: Add SPDX license headers
brief-yoctoprojectsqa: Add SPDX license headers
dev-manual: Add SPDX License headers
kernel-dev: Add SPDX license headers
profile-manual: Add SPDX licence headers
sdk-manual: Add SPDX license headers
toaster-manaul: Add SPDX license headers
haiqing (1):
libpam: Remove option 'obscure' from common-password
hongxu (1):
kmod: add nativesdk support
zangrc (1):
ethtool:upgrade 5.6 -> 5.7
Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
Change-Id: I1190ca17297b1167286cfc06033e8485396c7cce
Diffstat (limited to 'poky/bitbake/lib/bb/fetch2')
-rw-r--r-- | poky/bitbake/lib/bb/fetch2/gitsm.py | 45 | ||||
-rw-r--r-- | poky/bitbake/lib/bb/fetch2/perforce.py | 81 |
2 files changed, 116 insertions, 10 deletions
diff --git a/poky/bitbake/lib/bb/fetch2/gitsm.py b/poky/bitbake/lib/bb/fetch2/gitsm.py index 56bd5f048..d6e5c5c05 100644 --- a/poky/bitbake/lib/bb/fetch2/gitsm.py +++ b/poky/bitbake/lib/bb/fetch2/gitsm.py @@ -143,12 +143,43 @@ class GitSM(Git): try: # Check for the nugget dropped by the download operation known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \ - (ud.basecmd), d, workdir=ud.clonedir) + (ud.basecmd), d, workdir=ud.clonedir) - if ud.revisions[ud.names[0]] not in known_srcrevs.split(): - return True + if ud.revisions[ud.names[0]] in known_srcrevs.split(): + return False except bb.fetch2.FetchError: - # No srcrev nuggets, so this is new and needs to be updated + pass + + need_update_list = [] + def need_update_submodule(ud, url, module, modpath, workdir, d): + url += ";bareclone=1;nobranch=1" + + try: + newfetch = Fetch([url], d, cache=False) + new_ud = newfetch.ud[url] + if new_ud.method.need_update(new_ud, d): + need_update_list.append(modpath) + except Exception as e: + logger.error('gitsm: submodule update check failed: %s %s' % (type(e).__name__, str(e))) + need_update_result = True + + # If we're using a shallow mirror tarball it needs to be unpacked + # temporarily so that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and not os.path.exists(ud.clonedir): + tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) + self.process_submodules(ud, tmpdir, need_update_submodule, d) + shutil.rmtree(tmpdir) + else: + self.process_submodules(ud, ud.clonedir, need_update_submodule, d) + if len(need_update_list) == 0: + # We already have the required commits of all submodules. Drop + # a nugget so we don't need to check again. + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) + + if len(need_update_list) > 0: + logger.debug(1, 'gitsm: Submodules requiring update: %s' % (' '.join(need_update_list))) return True return False @@ -163,9 +194,6 @@ class GitSM(Git): try: newfetch = Fetch([url], d, cache=False) newfetch.download() - # Drop a nugget to add each of the srcrevs we've fetched (used by need_update) - runfetchcmd("%s config --add bitbake.srcrev %s" % \ - (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=workdir) except Exception as e: logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) raise @@ -181,6 +209,9 @@ class GitSM(Git): shutil.rmtree(tmpdir) else: self.process_submodules(ud, ud.clonedir, download_submodule, d) + # Drop a nugget for the srcrev we've fetched (used by need_update) + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) def unpack(self, ud, destdir, d): def unpack_submodules(ud, url, module, modpath, workdir, d): diff --git a/poky/bitbake/lib/bb/fetch2/perforce.py b/poky/bitbake/lib/bb/fetch2/perforce.py index f57c2a4f5..6f3c95b6c 100644 --- a/poky/bitbake/lib/bb/fetch2/perforce.py +++ b/poky/bitbake/lib/bb/fetch2/perforce.py @@ -1,6 +1,20 @@ """ BitBake 'Fetch' implementation for perforce +Supported SRC_URI options are: + +- module + The top-level location to fetch while preserving the remote paths + + The value of module can point to either a directory or a file. The result, + in both cases, is that the fetcher will preserve all file paths starting + from the module path. That is, the top-level directory in the module value + will also be the top-level directory in P4DIR. + +- remotepath + If the value "keep" is given, the full depot location of each file is + preserved in P4DIR. This option overrides the effect of the module option. + """ # Copyright (C) 2003, 2004 Chris Larson @@ -17,6 +31,36 @@ from bb.fetch2 import FetchError from bb.fetch2 import logger from bb.fetch2 import runfetchcmd +class PerforceProgressHandler (bb.progress.BasicProgressHandler): + """ + Implements basic progress information for perforce, based on the number of + files to be downloaded. + + The p4 print command will print one line per file, therefore it can be used + to "count" the number of files already completed and give an indication of + the progress. + """ + def __init__(self, d, num_files): + self._num_files = num_files + self._count = 0 + super(PerforceProgressHandler, self).__init__(d) + + # Send an initial progress event so the bar gets shown + self._fire_progress(-1) + + def write(self, string): + self._count = self._count + 1 + + percent = int(100.0 * float(self._count) / float(self._num_files)) + + # In case something goes wrong, we try to preserve our sanity + if percent > 100: + percent = 100 + + self.update(percent) + + super(PerforceProgressHandler, self).write(string) + class Perforce(FetchMethod): """ Class to fetch from perforce repositories """ def supports(self, ud, d): @@ -58,14 +102,33 @@ class Perforce(FetchMethod): logger.debug(1, 'Determined P4PORT to be: %s' % ud.host) if not ud.host: raise FetchError('Could not determine P4PORT from P4CONFIG') - + + # Fetcher options + ud.module = ud.parm.get('module') + ud.keepremotepath = (ud.parm.get('remotepath', '') == 'keep') + if ud.path.find('/...') >= 0: ud.pathisdir = True else: ud.pathisdir = False + # Avoid using the "/..." syntax in SRC_URI when a module value is given + if ud.pathisdir and ud.module: + raise FetchError('SRC_URI depot path cannot not end in /... when a module value is given') + cleanedpath = ud.path.replace('/...', '').replace('/', '.') cleanedhost = ud.host.replace(':', '.') + + # Merge the path and module into the final depot location + if ud.module: + if ud.module.find('/') == 0: + raise FetchError('module cannot begin with /') + ud.path = os.path.join(ud.path, ud.module) + + # Append the module path to the local pkg name + cleanedmodule = ud.module.replace('/...', '').replace('/', '.') + cleanedpath += '--%s' % cleanedmodule + ud.pkgdir = os.path.join(ud.dldir, cleanedhost, cleanedpath) ud.setup_revisions(d) @@ -95,10 +158,20 @@ class Perforce(FetchMethod): pathnrev = '%s' % (ud.path) if depot_filename: - if ud.pathisdir: # Remove leading path to obtain filename + if ud.keepremotepath: + # preserve everything, remove the leading // + filename = depot_filename.lstrip('/') + elif ud.module: + # remove everything up to the module path + modulepath = ud.module.rstrip('/...') + filename = depot_filename[depot_filename.rfind(modulepath):] + elif ud.pathisdir: + # Remove leading (visible) path to obtain the filepath filename = depot_filename[len(ud.path)-1:] else: + # Remove everything, except the filename filename = depot_filename[depot_filename.rfind('/'):] + filename = filename[:filename.find('#')] # Remove trailing '#rev' if command == 'changes': @@ -150,10 +223,12 @@ class Perforce(FetchMethod): bb.utils.remove(ud.pkgdir, True) bb.utils.mkdirhier(ud.pkgdir) + progresshandler = PerforceProgressHandler(d, len(filelist)) + for afile in filelist: p4fetchcmd = self._buildp4command(ud, d, 'print', afile) bb.fetch2.check_network_access(d, p4fetchcmd, ud.url) - runfetchcmd(p4fetchcmd, d, workdir=ud.pkgdir) + runfetchcmd(p4fetchcmd, d, workdir=ud.pkgdir, log=progresshandler) runfetchcmd('tar -czf %s p4' % (ud.localpath), d, cleanup=[ud.localpath], workdir=ud.pkgdir) |