Working with Katello – part 2. We will be importing CentOS errata into Pulp.
This article is part of the Homelab Project with KVM, Katello and Puppet series.
Homelab
We have Katello installed on a CentOS 7 server:
katello.hl.local (10.11.1.4) – see here for installation instructions
See the image below to identify the homelab part this article applies to.
Credits
This is possible thanks to the following sites:
- https://cefs.steve-meier.de/ – CEFS: CentOS Errata for Spacewalk
- https://github.com/rdrgmnzs/pulp_centos_errata_import – the script that imports CentOS errata into Pulp/Katello
Installation
Install the following packages:
# yum install git \ pulp-admin-client \ pulp-rpm-admin-extensions \ pulp-rpm-consumer-extensions \ pulp-rpm-handlers \ pulp-rpm-yumplugins \ pulp-rpm-admin-extensions \ pulp-consumer-client \ python-pulp-agent-lib \ perl-Text-Unidecode \ perl-XML-Simple \ perl-XML-Parser
Configure CentOS Errata Import
Make sure that the CentOS repositories have been synced. We’ve done this in the previous article.
Clone the repository:
# cd /opt && git clone https://github.com/rdrgmnzs/pulp_centos_errata_import.git # cd ./pulp_centos_errata_import
Download the latest errata archive and extract the file:
# wget -N https://cefs.steve-meier.de/errata.latest.xml.bz2 # bunzip2 ./errata.latest.xml.bz2
Configure Pulp authetication:
# mkdir -m0700 ~/.pulp
# cat /etc/pki/katello/certs/pulp-client.crt /etc/pki/katello/private/pulp-client.key > ~/.pulp/user-cert.pem
# chmod 0400 ~/.pulp/user-cert.pem
Import CentOS Errata into Pulp
Chances are that we won’t know which repository IDs to use when importing CentOS errata, therefore we can run the following command without specifying any (it may take longer):
# perl ./errata_import.pl --errata=errata.latest.xml
Alternatively, we can view a list of Pulp repositories by using following command:
# pulp-admin repo list | less
And then import errata into repositories required, e.g.:
# perl ./errata_import.pl \ --errata=errata.latest.xml \ --include-repo=83418f8f-e069-4b24-8e67-4b07af5e87d2 \ --include-repo=0abbd033-67cc-4e42-bff0-237d52f1bcdb
There was a case when we got the error saying “Skipping errata […] — No packages found”. If that happens, try passing the repository ID you want to load the errata for with the –include-repo flag.
After importing errata, use the mighty Hammer to force post-sync actions (Katello 3.0 and up):
# hammer settings set \ --name "force_post_sync_actions" \ --value "true"
We have to sync the repositories so that errata is published (and visible in the Katello interface).
# hammer repository synchronize \ --name "base_x86_64" \ --product "el7_repos"
If all goes well, we should see some errata:
# hammer repository info \ --name "base_x86_64" \ --product "el7_repos" ID: 1 Name: base_x86_64 Label: base_x86_64 Organisation: Lisenet Red Hat Repository: no Content Type: yum Checksum Type: sha256 Mirror on Sync: no URL: http://mirror.centos.org/centos/7/os/x86_64/ Publish Via HTTP: yes Published At: http://katello.hl.local/pulp/repos/lisenet/Library/custom/el7_repos/base_x86_64/ Relative Path: lisenet/Library/custom/el7_repos/base_x86_64 Download Policy: on_demand Product: ID: 1 Name: el7_repos GPG Key: ID: 1 Name: RPM-GPG-KEY-CentOS-7 Sync: Status: Success Last Sync Date: 28 days Created: 2018/02/20 20:56:52 Updated: 2018/02/21 23:16:44 Content Counts: Packages: 9591 Package Groups: 84 Errata: 830
Note how “Mirror on Sync” is set to “no”. This is to prevent loaded errata from being cleared on next sync.
Here is some info for a randomly selected errata ID:
# hammer erratum info --id 4191 ID: 4191 Errata ID: CESA-2018:0260 Title: CentOS systemd Security Update Type: security Severity: Moderate Issued: 2018-02-01 Updated: 2018-02-01 Description: Not available Summary: Solution:
I’m still having issues with importing,
It seems that it imported ~4k erratas for EPEL, but nothing for the others (especially base)
running it displays a whole bunch of :
followed by another whole bunch of (which take a VERY LONG TIME):
INFO: Creating errata for CEBA-2018:0728 (CentOS python-slip BugFix Update) (3 of 3) INFO: +----------------------------------------------------------------------+ Unit Upload +----------------------------------------------------------------------+ Extracting necessary metadata for each request... ... completed Creating upload requests on the server... [==================================================] 100% Initializing upload ... completed Starting upload of selected units. If this process is stopped through ctrl+c, the uploads will be paused and may be resumed later using the resume command or canceled entirely using the cancel command. Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completedCommand I’m trying now is:
the ID is for base repo. I tried without it as well before that which gave me 4k erratas on EPEL.
Any ideas?
Thanks
I think there is some confusion here. The script imports CentOS errata, but it has nothing to do with EPEL errata. While EPEL repository provides security metadata, CentOS repositories don’t supply it. This is the reason why we want to import CentOS errata from CentOS-Announce mailing list.
From the logs that you’ve posted, it looks like CentOS errata has already been applied. Try forcing post-sync actions and then re-sync the repository.
That’s why I was confused too. Right now I see available Errata like “FEDORA-EPEL-2018-ad387c7768” in my system but nothing under base repo.
Once I applied force post sync actions and did repo sync, I started seeing base repo errata.
By the way thanks so much for this series. I’m learning a lot from it.
No worries, these instructions are provided in the blog post.
Conflicts for Katello 3.7 when installing the packages
——————-
–> Processing Conflict: katello-agent-3.3.4-1.el7.noarch conflicts pulp-consumer-client
–> Finished Dependency Resolution
Error: katello-agent conflicts with pulp-consumer-client-2.16.4-1.el7.noarch
You could try using –skip-broken to work around the problem
You could try running: rpm -Va –nofiles –nodigest
I used Katello 3.6 and didn’t have this problem I’m afraid. I later upgraded Katello from 3.6 to 3.7 and errata routine still worked. Also, I don’t have the katello-agent package installed on the server.
It looks like some packages are obsoleted by katello-agent: pulp-rpm-handlers and python-pulp-agent-lib. This may very well be the case here.
@Eric, et al.:
As of today, running the “yum” command, verbatim, as shown in the article, results in the following error:
This is due to various changes since this article was first posted. In order to complete this step successfully, simply remove “pulp-rpm-handlers” and “python-pulp-agent-lib” from the list of packages to be installed; i.e.:
# yum install git pulp-admin-client pulp-rpm-admin-extensions \ pulp-rpm-consumer-extensions pulp-rpm-yumplugins pulp-consumer-client \ perl-Text-Unidecode perl-XML-Simple perl-XML-ParserAdditionally, you should be able to remove “git” from the package list as it is likely already installed by this point (if you have been following the previous articles).
(N.B.: The comment form likely messed up the formatting of the above; hopefully you can figure it out!)
@Tomas:
Thanks for the great series of articles!
Thanks for the tips, this is helpful!
Thanks for this tuto but I have the same problem than Eric with Katello Agent 3.8.
Thanks for your feedback!
Try removing pulp-rpm-handlers and python-pulp-agent-lib from the list of packages to be installed as per suggestion in another comment.
Very nice article!
I am on 3.8 and it looks like force_post_sync_actions is no longer there..
Thanks for your feedback. I’ve got several comments saying that there have been some changes between Katello versions. This is unfortunate but inevitable, because the project is in active development.
My homelab is on Katello 3.6. According to this changelog, force_post_sync_actions has been deprecated, and has since been replaced by the skip-metadata-check hammer flag.
Users have to set skip-metadata-check to force post-sync actions.
Hi,
Did you ever get this working? I am having troubles importing errata also.
I’ve tried the following but still no luck:
hammer repository synchronize –skip-metadata-check true –id 3
i’ve also tried to recreate the repository using: –mirror-on-sync=false to prevent loaded errata from being cleared on next sync but no luck either. Thanks.
Phat
Hi all:
The post is 2 years earlier there!
I just get katello-3.14.1 installed and try to import the ERRATA, but however unfortunately I find neither force_post_sync_actions nor skip-metadata-check are there
[root@foreman pulp_centos_errata_import]# hammer settings list | grep force_post_sync_actions
[root@foreman pulp_centos_errata_import]# hammer settings list | grep skip-metadata-check
[root@foreman pulp_centos_errata_import]#
any idea ?
I think that force_post_sync_actions has been replaced with skip-metadata-check hammer flag to force post-sync actions.
Hi,
I am on 3.8 and tried to use the ‘–skip-metadata-check true’ to force post-sync actions, however it does not seem to be getting the errata data – any idea?
hammer repository synchronize –product ‘Centos 7’ –name ‘Centos 7 – Base – x86_64’ –skip-metadata-check true –organization MyOrg
hammer repository info –id 1
ID: 1
Name: Centos 7 – Base – x86_64
Label: Centos_7_-_Base_-_x86_64
Organization: MyOrg
Red Hat Repository: no
Content Type: yum
Mirror on Sync: yes
URL: http://mirror.centos.org/centos/7/os/x86_64/
Publish Via HTTP: yes
Published At: katello.example.com/pulp/repos/Vendavo/Library/custom/Centos_7/Centos_7_-_Base_-_x86_64/
Relative Path: MyOrg/Library/custom/Centos_7/Centos_7_-_Base_-_x86_64
Download Policy: on_demand
Product:
ID: 1
Name: Centos 7
GPG Key:
Sync:
Status: Success
Last Sync Date: less than a minute
Created: 2018/11/13 20:54:23
Updated: 2018/11/13 21:09:36
Content Counts:
Packages: 9911
Source RPMS: 0
Package Groups: 88
Errata: 0
Looks like errata import was fixed recently in hammer-cli:
https://github.com/Katello/hammer-cli-katello/commit/1136d549365f94952215fff07b997b4de0b42497
Sorry, please disregard my previous comment, apparently all errata appeared after disabling ‘Mirror on Sync’.
Thanks. Yes, as mentioned in the blog post, “Mirror on Sync” needs setting to “no” in order to prevent loaded errata from being cleared on next sync
Hello
i follow the above all is workig as expected, i could see the errata when pulp-admin repo list | less but it is not synching, mirror on sync is turned off.
Id: 3-CentOS7_Content_View-v1_0-3c9a1001-6382-4a4c-bbea-d9dc35d
88411
Display Name: OS_x86_64
Description: None
Content Unit Counts:
Distribution: 1
Erratum: 887
Package Group: 88
hammer repository info –name “OS_x86_64” –product “CentOS 7 x86_64” –organization pvt
Name: OS_x86_64
Label: OS_x86_64
Sync:
Status: Success
Last Sync Date: 10 minutes
Created: 2019/06/13 17:55:36
Updated: 2019/06/14 07:39:13
Content Counts:
Packages: 10019
Source RPMS: 0
Package Groups: 88
Errata: 0
Hi,
After completion of errata import, im able to see the errata in content view but the same time which is not visible for content hosts or any individual host. how can we fix it.
Foreman 1.24 and katello 3.14
and client side is
katello-host-tools-fact-plugin-3.0.2-1.el7.noarch
katello-ca-consumer-insightsof.ctrls.in-1.0-1.noarch
katello-client-repos-3.4.0-3.el7.noarch
katello-host-tools-3.5.1-2.el7.noarch
There are 292 total Errata in this organization but none match the above filters.
I’m sorry, I don’t think that I had this problem before.
Hi Tomas, Have you tried using foreman-proxmox plugin, https://github.com/theforeman/foreman_fog_proxmox
I tried, but it requires higher version of ruby. I wonder if you encounter same issue
Hi Hamid, no, I never tried that plugin. I actually moved away from using Proxmox to KVM.
Hi,
First of all, thanks for this, most of the steps worked, and as u mentioned the settings command is replaced by :hammer repository synchronize –skip-metadata-check true –name “base_x86_64_7.8” –product “el7_repos”
I had a question, after the above synch of the repo do we have to publish and promote the CV again?
Hi, yes, after you sync the repo, you have to publish it again.
Also, beware if you install katello 3.16 or later this process will not work. Starting 3.16 the pulp data =base has been moved to postgresql db, so the pulp-admin will not report any repositories and hence importing/updating errata will not happen.
[root@katello tools]# mongo
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.9
> use pulp_database;
switched to db pulp_database
> db.repos.find()
{ “_id” : ObjectId(“5f1cad6e4a54e704e2a80ecb”), “repo_id” : “1-centos-7-v1_0-puppet-a7227dc4-5668-43ae-bbf6-d3a0a67b0f32”, “display_name” : “centos-7 Puppet Environment”, “notes” : { “_repo-type” : “puppet-repo” }, “scratchpad” : { }, “content_unit_counts” : { }, “_ns” : “repos” }
And the actual Yum repos are stored in PostgreSQL:
[root@katello tools]# su – postgres
[postgres@katello ~]$ psql pulpcore
psql (12.1)
Type “help” for help.
pulpcore=# select pulp_id, name from core_repository;
pulp_id | name
————————————–+—————
e0c17697-0649-49ad-b6c3-50606b1b5531 | base-38848
a5b2c791-36ce-43d9-aa73-b9cd8adc6aeb | updates-47970
bd454296-a9f4-47a8-af00-9311585ec40c | epel-98560
(3 rows)
Thanks for your comment, appreciated.
Experiencing issues with this – any updates for Katello 4.2?
ran the perl script and included the repo names found from querying the postgresql repository as mentioned in above comment.
Script output states “No Packages found” for all errata”
Repos are CentOS 7 x86_64 Base and Updates
Foreman 3.0.1
Katello 4.2
Sorry to hear that, I’m on Katello 3.x with no immediate plans to migrate to 4.x I’m afraid.
I am having issues while trying to import errata to katello:
INFO: Creating errata for CESA-2020:4060 (Important CentOS kernel Security Update) (13 of 14)
Use of uninitialized value in concatenation (.) or string at /opt/pulp_errata_import/errata_import.pl line 247.
Use of uninitialized value in concatenation (.) or string at /opt/pulp_errata_import/errata_import.pl line 247.
Use of uninitialized value in concatenation (.) or string at /opt/pulp_errata_import/errata_import.pl line 247.
Use of uninitialized value in concatenation (.) or string at /opt/pulp_errata_import/errata_import.pl line 247.
INFO: Command: erratum
Description: creates a new erratum
Available Arguments:
–bg – if specified, the client process will end immediately
(the task will continue to run on the server)
–repo-id – (required) unique identifier; only alphanumeric, ., -,
and _ allowed
-v – display extra information about the upload process
–erratum-id, -i – (required) id of the erratum to create
–title, -n – (required) title of the erratum
–description, -d – (required) description of the erratum
–version – (required) version of the erratum
–release – (required) release of the erratum
–type, -t – (required) type of the erratum; examples: “bugzilla”,
“security”, “enhancement”
–status, -s – (required) status of the erratum; examples: “final”
–updated, -u – (required) timestamp the erratum was last updated;
expected format “YYYY-MM-DD HH:MM:SS”
–issued – (required) timestamp the erratum was issued; expected
format “YYYY-MM-DD HH:MM:SS”
–reference-csv, -r – path to a CSV file containing reference information. An
example of a reference would be information of a
bugzilla issue. Format for each entry is:
“href,type,id,title”
–pkglist-csv, -p – (required) path to a CSV file containing package list
information. Format for each entry is:
“name,version,release,epoch,arch,filename,checksum,check
sum_type,sourceurl”
–from – (required) erratum issuer, used in the ‘from_str’ for
the erratum; typically an email address
–pushcount – pushcount for the erratum; an integer, defaults to 1 if
unspecified
–reboot-suggested – if specified, the erratum will be marked as
‘reboot-suggested’
–restart-suggested – if specified, the erratum will be marked as
‘restart-suggested’
–relogin-suggested – if specified, the erratum will be marked as
‘relogin-suggested’
–severity – severity of the erratum
–rights – rights for the erratum
–summary – summary for the erratum
–solution – solution for the erratum
The following options are required but were not specified:
–release
/bin/perl /opt/pulp_errata_import/errata_import.pl –errata=/opt/pulp_errata_import/errata.latest.xml –include-repo=4bd040c9-d19c-4da8-8564-43e179581ef2
It just started to happen at around december 3rd, before then, it was working like a charm.
I am using katello-3.14.1-1.el7.noarch and Centos7 at the moment. pulp-server-2.21.5-1.el7.noarch
Have you tried specifying the release?
Found the problem, our WAF was blocking a specific request made by:
pulp-admin rpm repo content rpm –repo-id=4bd040c9-d19c-4da8-8564-43e179581ef2 –match=filename=kernel-tools-libs-devel-3.10.0-1160.el7.x86_64 –fields=name,version,release,epoch,arch,checksum,checksumtype
When that got blocked, the script just received a 403 error instead of the table it was expecting thus making the variables it uses at around that erroring line 247 being blank
Its all good now after whitelisting the pulp urls
Thanks, that makes sense. From my experience, WAFs are notorious for blocking legitimate traffic. Great care should be taken when configuring WAF rules.