Log in

Adding and building a module for Fedora

Modularity New-Module-Process

Adding and building a module for Fedora

This describes the process of adding a new module to the Fedora Modularity project, how to build it locally and how to build it in Fedora infrastructure

Process and policy for how to add a module to Fedora

Adding a module repository is a manual process atm. Find someone from the release-engineering group to add the new repository and give you write access. Later on this will be automated by MBS, the module build server, but this is still being worked on.

Writing a new modulemd file

A modulemd file is a yaml file that contains the module metadata like description, license and dependencies. The sample file in the upstream git repository of modulemd contains a complete documentation of the required and optional yaml tags. The Modularity team uses a shorter modulemd file to test builds, but it can also be used as a base for new modules. Another good example is base-runtime.yml Lets use the vim modulemd as an example for this document. It is in the /home/karsten/Modularity/modules/vim/ directory on my system.
document: modulemd
version: 1
    summary: The best text editor and IDE
    description: The classic, extensible text editor, the legend.
        module: [ MIT ]
            base_runtime: master
            base_runtime: master
        community: https://fedoraproject.org/wiki/Modularity                    
        documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules                                                                   
        tracker: https://taiga.fedorainfracloud.org/project/modularity          
                - vim-enhanced
                - vim-common
                - vim-filesystem
                - vim-minimal
            - vim-common
                rationale: Provides API for this module
                ref: f25                                                        
                buildorder: 10                                                  
                rationale: build dependency
                ref: f25
                rationale: build dependency
                ref: f25
                rationale: build dependency
                ref: f25
                rationale: build dependency
                ref: f25
All dependencies of vim need to be listed under components/rpms, except those that are already included in Base Runtime. Here's how you can get the list of vim dependencies that are not in Base Runtime:
wget https://raw.githubusercontent.com/asamalik/fake-base-runtime-module-image/master/packages/gen-core-binary-pkgs.txt
for i in `repoquery --requires --recursive --resolve --qf "%{SOURCERPM}\n" \
      vim-enhanced vim-minimal vim-common vim-filesystem \
    | sed -e "s/-[^-]*-[^-]*$//" | sort -n | uniq` ; do
    grep -wq $i gen-core-binary-pkgs.txt || echo $i

verifying the syntax of the new modulemd file and bug fixing

Once the modulemd file is finished, it is a good idea to check if there any errors in the yaml syntax. The check_modulemd program checks modulemd files for errors. You need to install some packages to use this:
python2-aexpect - dependency for python-avocado
python2-avocado - avocado testing framework
python2-modulemd - Module metadata manipulation library
python-enchant - spell checker library (needed only for check_modulemd.py)
hunspell-en-US - English dictionary (needed only for check_modulemd.py)
Then run
./run-checkmmd.sh /home/karsten/Modularity/modules/vim/vim.yaml
and check the output for errors:
Running: avocado run ./check_modulemd.py --mux-inject 'run:modulemd:/home/karsten/Modularity/modules/vim/vim.yaml'
JOB ID : 51581372fec0086a50d9be947ea06873b33dd0e5
JOB LOG : /home/karsten/avocado/job-results/job-2017-01-19T11.28-5158137/job.log
TESTS : 11
(01/11) ./check_modulemd.py:ModulemdTest.test_debugdump: PASS (0.16 s)
(02/11) ./check_modulemd.py:ModulemdTest.test_api: PASS (0.15 s)
(03/11) ./check_modulemd.py:ModulemdTest.test_components: PASS (0.16 s)
(04/11) ./check_modulemd.py:ModulemdTest.test_dependencies: WARN (0.02 s)
(05/11) ./check_modulemd.py:ModulemdTest.test_description: PASS (0.16 s)
(06/11) ./check_modulemd.py:ModulemdTest.test_description_spelling: PASS (0.16 s)
(07/11) ./check_modulemd.py:ModulemdTest.test_summary: PASS (0.16 s)
(08/11) ./check_modulemd.py:ModulemdTest.test_summary_spelling: WARN (0.02 s)
(09/11) ./check_modulemd.py:ModulemdTest.test_rationales: ERROR (0.04 s)
(10/11) ./check_modulemd.py:ModulemdTest.test_rationales_spelling: PASS (0.16 s)
(11/11) ./check_modulemd.py:ModulemdTest.test_component_availability: WARN (0.02 s)
TESTS TIME : 1.20 s
So this isn't quite right yet, lets have a look at the logfile mentioned in the output.
grep -i error /home/karsten/avocado/job-results/job-2017-01-19T11.28-5158137
TestError: Rationale for component RPM generic-release should end with a period: build dependency
It seems that rationales need to end with a period. Change all those lines so that they look like this:
rationale: Provides API for this module.
ref: f25
buildorder: 10
rationale: build dependency.
ref: f25
rationale: build dependency.
ref: f25
rationale: build dependency.
ref: f25
rationale: build dependency.
ref: f25
rationale: build dependency.
ref: f25
Another run of check_modulemd.py completes without errors.

Building the module locally

The build_module script from the build-module repository on github makes local module builds really easy. It sets up the environment and then builds a module and its components locally with mock. One requirement is to have docker installed and running on your system. It is also required that the name of the new modulemd file, the repository name of that module and the name of the module itself match in order to use the build_module script. As build_module builds the latest commit in the master branch of the module git repository, changes need to be checked into git, a push to upstream (dist-git) is not required at this stage. The basic usage of build_module is
./build_module /home/karsten/Modularity/modules/vim/ /tmp/results
This will download a container with the Module build service and rebuild the dependencies that are listed in the modulemd file. This step can take quite some time, depending on the module and how many components need to be built. When build_module is done there will be a couple of rebuilt rpms in /tmp/results/module-vim-master-*/results/:
cd /tmp/results/module-vim-master-*/results/
find . -name "*.rpm"

Putting the packages into a container

For this step you'll need to create a rpm repository of the new packages.
cd /tmp/results/module-vim-master-20170119120233/
mkdir vim-module-repo
cp results/*.rpm vim-module-repo
cd vim-module-repo
createrepo .
The /tmp/results/module-vim-master-20170119120233/vim-module-repo needs to be uploaded somewhere public so that docker can access it. A good place for that is the fedorapeople.org webspace that each Fedora developer has.
scp -r /tmp/results/module-vim-master-20170119120233/vim-module-repo karsten.fedorapeople.org:public_html/
You'll also need a dnf/yum configfile (/home/karsten/Modularity/modules/vim/vimmodule.repo) that points at this new repo:
cat /home/karsten/Modularity/modules/vim/vimmodule.repo
name=VIM module
Now put everything into a Dockerfile. We're using Adam Samalik's fake-gen-core-module as there is no usable base-runtime module yet:
cat /home/karsten/Modularity/modules/vim/Dockerfile
FROM asamalik/fake-gen-core-module
ADD vimmodule.repo /etc/yum.repos.d/vimmodule.repo
RUN dnf -y update vim-minimal
RUN dnf -y install vim-enhanced

Building the module in Fedora infrastructure using a local module-build-service instance

This step uses a local module-build-service and other components in containers and passes results on to the Fedora staging infrastructure. A checkout of the module-build-service is required. Change into your local copy of this repository and run
docker-compose down
docker-compose up --build
This will start the module-build-service frontend and scheduler as well as fedmsg and you'll see when messages about module builds are coming in over the Federated Message Bus. The local module-build-service will connect to product-definition-center (PDC) on the modularity developer server modularity.fedorainfracloud.org At the moment an account on modularity.fedorainfracloud.org is required for this step as remote port forwarding needs to be established via ssh:
cat ~/.ssh/config
Hostname modularity.fedorainfracloud.org
user fedora
RemoteForward 3007
Some changes need to be made to the local module-build-service (aka fm-orchestrator) git repository so that it allows to build from a git repositoy on github:
diff --git a/conf/config.py b/conf/config.py
index 97eed6e..2ddb61b 100644
--- a/conf/config.py
+++ b/conf/config.py
@@ -35,7 +35,7 @@ class BaseConfiguration(object):
PDC_URL = 'http://modularity.fedorainfracloud.org:8080/rest_api/v1'
- SCMURLS = ["git://pkgs.stg.fedoraproject.org/modules/"]
+ SCMURLS = ["git://pkgs.stg.fedoraproject.org/modules/","https://github.com/KarstenHopp/"]

# How often should we resort to polling, in seconds
# Set to zero to disable polling
diff --git a/contrib/submit-build.json b/contrib/submit-build.json
index 0e312a5..e46bf8a 100644
--- a/contrib/submit-build.json
+++ b/contrib/submit-build.json
@@ -1,3 +1,3 @@
- "scmurl": "git://pkgs.stg.fedoraproject.org/modules/testmodule.git?#620ec77"
+ "scmurl": "git://github.com/KarstenHopp/vim-module.git?#cdbc4bf"
Now you need to get a kerberos ticket for the Fedora staging environment. If you haven't already done so, add
kdc = https://id.stg.fedoraproject.org/KdcProxy
to the realms section of your /etc/krb5.conf file. Then point your web browser at https://admin.stg.fedoraproject.org/accounts and log in with your Fedora credentials so that the account will get synced from the Fedora production environment. Run
(replace 'karsten' with your FAS account) and get a kerberos ticket. In another window, ssh into modularity.fedorainfracloud.org to establish required port forwarding. Now you can run
python submit-build.py
from within the module-build-service git repo. You might need to login again at id.stg.fedoraproject.org, just follow the instructions. Now the module build will be submitted to the Fedora build servers. Log messages will show the progress on your screen.

building the module

When your module is ready to get added to Fedora, you need to have write access to the module dist-git on pkgs.stg.fedoraproject.org and you need to have pushed all your changes to this module git repository. You can build your module using two different methods: A special version of rpkg with module-build support is required for this step. Change the working directory to your local copy of your module repo and simply run
fedpkg module-build
The other method requires that you add the git URL of your latest module commit to the submit-build.json file in the module-build-server git repository and then run
python submit-build.py
What is going on in the Fedora Modularity project

We are currently in a 4 week Modularity Sprint #10, the usual 2 week sprint got extended due to FLOCK and pre-FLOCK developer meetings.
As you can see in the 'In Progress' and 'Complete' columns on our Taiga board, lots of tiny and larger pieces are coming together.
The following text describes how various parts of the Modularity infrastructure work together. Ralph Bean shows these connections in his Modularity infrastructure diagram.
Module build tasks can be submitted to orchestrator (rida) either by posting a short yaml text into the REST API or with 'fedpkg module-build'. The orchestrator will then download and parse the module description file, check the dependencies and schedule koji builds for missing components. The build process can be monitored with the Build Pipeline Overview (BPO) component and the final result will be fed into the Product definition center (PDC) which is a webapp and API designed for storing and querying product metadata.
All of the above is already done and in a usable state after this sprint, although not all patches have been added to the upstream repositories yet. Same features are still missing, but we'll continue adding them in the upcoming weeks.

Langdon demoed how all this works from a user perspective during his talk [1] at FLOCK in Krakow. Unfortunately a recording of his talk isn’t available on YouTube yet. Nils and Adam therefore put together a similar demo, you’ll find it in the Fedora Modularity channel on Youtube

[1] https://flock2016.sched.org/event/6yoy/modularity-why-where-we-are-and-how-to-get-involved
[Link to Part 1

One of our main communication channels is IRC. We've recently decided to standardize the name of our efforts as 'Modularity' where we prior to that sometimes used the term 'Modularization' and sometimes 'Modularity'.
That's why you can now find us in the #fedora-modularity channel on Freenode IRC. (The old channel #fedora-modularization will redirect you to our new channel).If you have any questions about what we do or, even better, would like to join our group and help with programming (mostly python) or if you have some ideas for improvements, this is where you need to go and post your questions: irc://chat.freenode.net:6667/#fedora-modularity
To get opinions of other Fedora developers, a post to the Fedora development list devel@lists.fedoraproject.org is probably a good idea. Please use the [Modularity] prefix in your subject for Modularity related posts.

For those of you going to FLOCK in Krakow (August 2-5, 2016), Langdon White will be there and has a session about Modularity on Wednesday, August 3, 11:00-11:50 with the topic 'Modularity: Why, where we are, and how to get involved'.
The Fedora Modularity Project is an effort to fix several problems that all distributions face. One of them is the disconnect between Fedora's release cycle and the release cycle of larger Fedora components like for example GNOME, KDE or even the kernel. Those components obviously don't have the same lifecycle that Fedora follows and Fedora can't always wait for major components to be released upstream and on the other hand doesn't want to ship outdated software.
An earlier attempt to work around this disconnect were the Fedora Rings with a central core 'base design', a concentric ring #2 around it for 'environments and stacks' and a ring #3 for applications. It wasn't possible to have different release cycles for packages in ring #2 as dependencies wouldn't allow that most of the time.

Another problem that the Modularity project is attempting to fix is the need for parallel installations of different versions of components without having to use compatibilty packages. The reason for this is that users want the latest set of components and they also want to keep their older applications running even though they require older components.

Fedora Modularity takes the 'Fedora Rings' approach to the next level. Instead of having complete rings we're now using something like 'ring segments' around a core module where the segments on the same ring # are independent on each other whereas segments on outer rings depend on certain inner segments.

There's a Wiki page describing how to get involved with Modularity. I'd strongly suggest reading those onboarding docs including the links on that page. As we're using agile software development methods for Modularity with 2 week sprints it is very easy to get involved even if you're unsure if this really is something you'd like to spend more of your time on. Just pick a task that you'll be able to finish during one of the sprints  from the 'New' column of our Taiga board, move it to 'In progress' and assign it to yourself. Also make sure to get involved in our ongoing discussions in the #fedora-modularization channel on Freenode IRC.
At the end of each sprint each of us creates a short demo about the work that was done during that sprint. These videos are available on YouTube and show for example how to install the modularity environment by checking out our git repositories, how to set up and configure product definition-center, how to use fm-tools for modules and so on.

As it difficult to follow the URL's in the videos, here's a short writeup about the git repositories and which git branches you need to get started:
export MODULARITY_DIR="modularity"
sudo rm -rf ~/modularity_workdir 2>/dev/null
# Set up the pungi / pdc modularity environment
#Create a new working directory first and chdir into it
#check out the pungi-modularity git repository
git clone https://pagure.io/pungi-modularity.git
#check out the modularity-prototype branch of Lubos's pungi git repository
git clone --branch modularity-prototype http://pagure.io/forks/lkocman/pungi.git pungi
#check out the modularity branch of Lubos's productmd git repository
git clone --branch modularity http://github.com/lkocman/productmd.git productmd
#fm-metadata got renamed to modulemd, check out the git repository
git clone http://pagure.io/modulemd.git modulemd

#set up some environment variables so that python finds the files
export MODULARITY_DIR="$HOME/modularity-work"
export PATH="$MODULARITY_DIR/pungi/bin:$PATH"

#now do some work
#create a directory to store the compose, I've used ~/modularity_workdir
sudo mkdir ~/modularity_workdir
sudo chown $USER.$USER ~/modularity_workdir
cd $MODULARITY_DIR/pungi/bin
# get all the packages required by the packages referenced 
# in ../../pungi-modularity/pungi-inputs/core.yaml
./pungi-gather-prototype \
  --arch x86_64 \
  --config ../../pungi-modularity/pungi-inputs/core.yaml \
  --target-dir ~/modularity_workdir \
  --source-repo-from-path /home/24/Everything/x86_64/os/ \
  --source-repo-from-path /home/24/Everything/source/tree/ \
# get all the packages required by the packages referenced 
# in ../../pungi-modularity/pungi-inputs/shells.yaml
./pungi-gather-prototype \
  --arch x86_64 \
  --config ../../pungi-modularity/pungi-inputs/shells.yaml \
  --target-dir ~/modularity_workdir \
  --source-repo-from-path /home/24/Everything/x86_64/os/ \
  --source-repo-from-path /home/24/Everything/source/tree/ \
#create repositories with those RPMs
./pungi-createrepo-prototype \
  --arch x86_64  \
  --static-content-manifest ~/modularity_workdir/manifest_core-x86_64-*/  \
  --target-dir ~/modularity_workdir \
  --source-repo-from-path /home/24/Everything/x86_64/os/ \
  --source-repo-from-path /home/24/Everything/source/tree/ \
./pungi-createrepo-prototype \
  --arch x86_64  \
  --static-content-manifest ~/modularity_workdir/manifest_shells-x86_64-*/ \
  --target-dir ~/modularity_workdir \
  --source-repo-from-path /home/24/Everything/x86_64/os/ \
  --source-repo-from-path /home/24/Everything/source/tree/ \
#and finally do a compose
pungi-compose-prototype \
  --release fedora-24 \
  --variants-file /home/karsten/tmp/modularity/pungi-modularity/pungi-inputs/variants-fm.xml \
  --arch x86_64 \
  --target-dir ~/modularity_workdir \

After the last step you have several json files in ~/modularity_workdir/compose*/compose/metadata that describe the exact package versions used for this module and that will be later will be used by the Orchestrator to create koji buildroots with these packages and then build the module components and the module itself.

VIM Tip of the day

Ok, here's a short one:

Why don't you use vimx instead of vim ? It is part of the vim-x11 package and starts gvim in terminal mode without the GUI.

It has one major advantage over vim: It has access to the X clipboard.

So if you have marked some text in your browser, you can access it in vimx with the special register '*'.
To test it, mark something in the browser or in another terminal and then switch to vimx and type :reg to list the contents of vim's registers. There should be one called "* with the marked text.

To use it, just yank it into your text as usual. Move the cursor to where the marked text needs to appear and then press
which means 'paste the contents of register * at the current cursor location'

Tested with KDE, if it doesn't work in Gnome, please let me know.

VIM Tip of the day

VIM Tip of the day

Even as the long term Fedora VIM maintainer I'm using only a minimal subset of VIM's features and commands.
But there are lots of small tips and tricks that I think every VIM user should know about and I'll try to post them occasionally in a small series of blog posts.

So today it is about Ctrl-p. Have you ever tried to remember and pronounce a function name or maybe just a name correctly several times in a document ? That's not necessary, you just have to get it correct the first time and then use Ctrl-p to make VIM do this job for you.

Lets take this text from Volcanology of Iceland which I chose because of those hard to get correct names:

The eruption under Eyjafjallajökull ("glacier of Eyjafjöll") in 2010 was notable because the volcanic ash plume disrupted air travel in northern Europe for several weeks; however this volcano is minor in Icelandic terms. In the past, eruptions of Eyjafjallajökull have been followed by eruption of the larger volcano Katla, but after the 2010 eruption no signs of an imminent eruption of Katla were seen

If you have to type that volcano name again, just (in insert mode)type Ey and then press Ctrl-p.
Vim shows you a text menu where you can choose between


with the cursor keys and selct one with ENTER. VIM inserts the selected word into your document.

That's it, fast and easy ;-)

Another great FUDCon

Thanks a lot to the organizers of the FUDCon Lawrence !

The location was great, near the hotel were lots of pubs and restaurants to meet at in the evening. We had coffee (!!!), more than enough rooms afaik, lunch each day and even sponsored beverages at the FUDpub on Saturday. Even the weather played nice, who would have thought that we would be able to sit outside for Lunch in January ?
We've got a lot of topics covered and will keep you informed on the PPC mailing list about improvements to our wiki pages and the future direction of the PPC secondary arch.
Brent has also mentioned some other topics on the PPC mailinglist, so klick the link and read his post...

Fedora 16 for PowerPC: Alpha got released !

Phil has already posted the announcement, but I think this is also worth a blog entry.
We've finally fixed all our blockers for the alpha release of F-16 for PowerPC, got all packages properly signed and synced to the mirrors !
So, if anyone has a PowerMac or an IBM PowerPC that is catching dust because you don't know what to do with it, now is your chance to get the latest Fedora version on it and help out with testing.

You can grab the DVD or netboot images from one of the mirrors for F-16 Alpha for PPC (32bit) or for PPC64 (64bit). PPC64 is our main platform that we develop and test our packages and installer on, 32bit installs are unfortunately completely untested as we don't have the necessary hardware for that.

Report any issues that you find during testing in bugzilla, but please make sure that you set the architecture to 'powerpc' so that we can find your report.

PPC secondary arch progress

Automatic rebuilds with koji-shadow have started, so I'd expect that the number of older packages continues to drop even more in the next few days:

Martin Gracik has almost completed his work on PPC support for lorax, that's our tool to populate the new unified ramdisk.
This means that we can start looking into boot and install issues now.

PPC secondary arch progress

Steady progress, 2126 packages older than the latest package with the same name on primary arch dist-f15, 7543 have the same N-V-R, 250 are newer than its counterpart (sometimes we have to use packages from dist-f15-updates-candidate).

'Older' means the latest RPM of a package is older on PPC then on the primary archs.
'Same' and 'newer' are for the reader to figure out.
'total_missing' is calculated like this: primary arch has the RPMs foo-1.0-1.fc15, foo-1.0-2.fc15 and foo-1.0-3.fc15, PPC has only foo-1.0-1.fc15, so that's 2 missing builds. And when foo-1.0-3.fc15 is built on PPC, the missing_builds counter goes to zero, it's the worst case number