Packaging for PCLinuxOS


 * Relevant to all editions of PCLinuxOS.

A packager is the life blood of PCLinuxOS. Packager is the one who keeps PCLinuxOS updated. Learning to package is an important step towards making useful contributions to the betterment of PCLinuxOS.

The whole process is easy to learn with a little attention to detail and perseverance. In case you run into difficulties just go to the forums or IRC to get help.

Introduction
Packaging is the process of preparing installable packages from source code of individual software. It is an umbrella word used to cover all the things including building right from the source to adjusting spec files and making a PCLinuxOS package out of RPMs available elsewhere to updating existing PCLinuxOS packages.

In order to become a successful packager, it is necessary to have patience. Do not give up easily. If you do not succeed, try again and again. It is also very important to read a lot initially until your mind gets the picture.

''Even if initially you do not understand whatever you are reading or supposed to do, continue reading or doing it. The puzzle will be solved in the end''.

Rough overview
'Please understand that this section is only aimed at providing an approximate picture of what and how building a package works. This is unlikely to be perfectly correct and hence you have to verify the correctness of this information on your own'.

During packaging, we convert original software/items from coders into a form which is ready for download and installation by end users.

Now the original software/items could be media (wallpapers, icons, sounds etc.) or software (binary executables, source code) The most common use of packaging however is to convert source into installable .rpms for end users.


 * Software from source - Most of the software written by humans is in the form of a human readable language such as Assembly, C / C++, Java / Python. However a computer will only understand instructions in machine language which is in binary form. So we need to convert the software from human readable languages to machine language. This is done by a method known generally as compiling / building.


 * - In case of C / C++ etc., this conversion is done in the process of packaging by compiling / building.
 * - Java looks like a gray area what with there being an intermediate conversion to bytecode. Please read this guide for more information on packaging Java programs - Creating an RPM for a Java Application' - http://javaworkshop.wordpress.com/2008/10/22/rolling-up-an-rpm-for-a-java-application/
 * - In case of interpreted languages, this conversion is done on the fly during their execution in the user's computer. Hence here the source is packaged as is.


 * Now this building can happen at 3 different levels :


 * - At the original author of software : Most commercial software is distributed pre-built.
 * - At the distributor : That is what our Bacon Brigade does.
 * - At the end user : Luckily users of PCLinuxOS do not have to do this. This is the standard method in source based distros.


 * So in our case the following is done :
 * - Software is procured from source by the packager.
 * - Source is extracted to BUILD area and build commands run there to get the target software.
 * - A directory structure mimicking the user's computer is created inside BUILDROOT. The target software is now installed into BUILDROOT as if it is getting installed into the user's computer.
 * - The resulting files are picked out of this BUILDROOT environment and put together inside an archive and this is called a package. This will be found in /src/rpm/RPMS/i586 or x64 depending on the target.
 * - A copy of the source code along with the .spec file used to create this package is placed in /src/rpm/SPRM. This is what is used by the maintainer of PCLinuxOS to recreate the same package and enter it into the official repository after quality control checks.


 * Software binaries and media - Things are slightly different when it comes to binary packages and media as they already come in compiled binary form. They mostly only need to be placed into their proper location.

Approach to learning
'Please read through the complete guide. Parts of the guide are verbatim or barely modified posts from the forum by the authors referred. Considering the high level of knowledge of the respective authors, it might seem intimidating at first. (I had a similar experience too). Just read the rest of the guide'.


 * To begin with you should learn rebuilding something which is already present in our repository. Look for software of size less than 1 MB at first.
 * Next you should try to package something similar by your own. Try very simple items containing just 1 or 2 files.
 * Next you should try updating something in the present repository. (You will need to study the version numbers of existing packages and compare them to those from the official websites in order to know which can be updated.)
 * Now you could try and package a software not present in the repository or without looking at the already packaged version.
 * Once you are comfortable here, try packaging libraries after understanding about them.
 * Now you could try your hand at very complex and interlinked packaging tasks such as whole desktop environments, wine etc.
 * At this level, you could try to package binary software such as graphic drivers etc.

New packagers - How to get started
Get an idea of our repository set-up by reading this page Repository and Sections and Mirrors.

Posted by Neal on the forum


 * Go to http://ftp.nluug.nl/pub/os/Linux/distr/pclinuxos/pclinuxos/ and scroll down to the SRPMS.(*) sections. Clicking any of those links will open that section, where you will find SRPMs for PCLinuxOS.
 * These are the best src.rpms to start out with learning to package upto PCLinuxOS standards. Study them. Look closely at how the spec files are written.
 * You should not start with an SRPM from another distro, as these will require changes you may be unaware of. They often use different naming conventions, so that the BuildRequires and Requires in their spec files will call for something that has a different name in PCLinuxOS. They may install to different directories, which will cause difficulties, also.
 * The point is that you should start with learning to rebuild PCLinuxOS packages, proceed to learning how to upgrade our packages and only after learning how to create a PCLinuxOS package should you attempt to convert an SRPM from elsewhere.
 * Same folder from other mirrors of PCLinuxOS can also be used instead of NLUGG.

Setting up the build environment
Please follow these steps : apt should be your local repository for self created packages. rpm is where your packaging will happen. tmp is a temporary directory.
 * Download and install the KDE MiniMe or LXDE Mini flavour on a separate partition.
 * Install pkgutils-kde or pkgutils-thunar which will also install pkgutils.
 * Install jpackage-utils
 * Select the user name by clicking on it when prompted to setup that user as packager. All the packaging related directories will be created under src directory in this user's /home.
 * NOTE : Even if there is only one user in this installation, it is necessary to click and highlight the user name. Else the process of setup will be cancelled and you will need to re-do it from commandline using mkrepo command.
 * Avoid using special characters in the user name of packaging environment.
 * If everything went correctly, the /home/xxxx where xxxx is the packaging user name will have a new directory called src. If you go deeper into it, there should be apt rpm tmp.
 * /home/xxxx/src/rpm/RPMS will store built .rpms either in i586 or x86_64 depending on which architecture you are packaging for.
 * /home/xxxx/src/rpm/SOURCES will store your sources in tar.xz format for building.
 * /home/xxxx/src/rpm/SPECS will store the .spec files needed for the build.
 * /home/xxxx/src/rpm/SRPMS will store the final .srpms. These will be needed when you have succeeded in building, testing the package and ensuring that it is bug free, to upload for the consideration of PCLinuxOS maintainer.
 * There will be 2 hidden files under /home/xxxx namely .rpmmacros and .rpmrc

Building - step by step
Now to the actual process of building a PCLinuxOS package. We need the following :


 * Source - It will be available for download from the original developer website or at the .srpm link mentioned above. It may be available as tar.gz or tar.xz or in any other compressed format. However PCLinuxOS specifications require that you convert any of them to the tar.xz format. This can be done easily by right clicking on the downloaded file and using convert to tar.xz option. Now this source file in tar.xz format has to be placed in the /home/xxxx/src/rpm/SOURCES directory.
 * Spec - This is the file which contains instructions about the actual process of building. It can either be built anew or updated suitably from an older version .srpm existing in our repositories.
 * Dependencies - Any package which we are trying to build may need the existence of other packages. Dependencies can be either build-time or run-time. Build-time dependencies are usually in the form of -devel packages while run-time dependencies are usually in the form of libraries or other packages. Dependencies can be ascertained by reading the release notes of original developer, Readme.txt or equivalent text file inside the source package or by going through older .spec file. An easy way to install dependencies needed for build is to right-click on the .spec file and select Install dependencies. Alternatively, one can make a list of dependencies and install them one-by-one through Synaptic or apt-get install from command line.
 * If you are rebuilding a package from .srpm, you may either right-click on the .srpm and select Install for build option or go to command line, navigate to the directory with the .srpm and type :

rpm -ivh nameofthepackage.srpm


 * Once everything is ready, you may right click on the .spec file and select Build all or Build all, log file options to start building. Alternatively, navigate to the /home/xxxx/src/rpm/SPECS directory and type :

rpmbuild -ba nameofthepackage.spec
 * Time taken to complete build will depend on the size and complexity of the package, power of the build machine etc.
 * Right-click build will open a special purpose shell which will not close on completion. End of build is indicated when you see a exit 0 as the last line.
 * If the build was started from commandline, you should be back at the command prompt once the build is complete.
 * Watch the output of the build carefully. Generally any WARNING will not cause the build to fail, but may cause some problems with the package later on. However any ERROR will generally cause the build to abort. Note down these for further research.
 * Run gbd from the command line to update your local repository and then use Synaptic to install this and check if things work properly.

Gathering information
Before or during building / troubleshooting a build, you would need to have some information regarding settings, options etc. There are many places to look for such information and the location would also depend on the language, type of package etc.
 * Homepage - If the package has a well maintained homepage, it should be your first port of call.
 * Source - The source tarball or the directory will have files such as README, INSTALL, CHANGELOG which are also very good sources of information.
 * configure.ac - This file could provide very useful information about BuildRequires.
 * ./configure --help - The options available for build could be ascertained by extracting the package to a directory and then running this command from a terminal.

Dos and Don'ts
Slightly modified version of a post by Texstar on the forum :


 * Name your specfile - pclos-name-of-package.spec (example: pclos-doxygen.spec)
 * This makes it handy if you are updating a package and encounter a build error. You can install another distribution's source package and compare their spec file with yours to see if they added a patch or a different build option to make the new version build.


 * Convert your source tarball to tar.xz from tar.gz and tar.bz2
 * The public servers do not have infinite space and bandwidth. By re-compressing your source tarball with tar.xz you will generate a much smaller SRPM. This saves space on the mirrors and makes uploading the SRPMs from the build server to go much faster as there is less to upload.


 * Use proper versions of your package release
 * If this is a new package going into the PCLinuxOS repository even if it has been based on another distribution's specfile then your release macro for PCLinuxOS is %mkrel 1


 * Make a complete changelog
 * This gives information as to when the package was built, who built it, an email contact, version number and a description of what you did to the package. If you are importing a package from another distribution then all you need to do is add in the description area based on original specfile from fedora, Opensuse, mdv etc...

Example: - Mon Mar 28 2011 Texstar  1.7.4-1pclos2011 - 1.7.4   - add requires for new glibc
 * gcc/glibc updates need additional requires
 * When you update your gcc/glibc you will need to add the following to your specfile to force the user to update the following 2 packages to ensure the new programs compiled with gcc 4.5.2 will install and run properly on their existing system.
 * Requires: glibc >= 2.12.1
 * Requires: libstdc++6 >= 4.5.2

Neal on the forum :


 * Test your packages before uploading them.
 * If you package beta software, be sure you put a note with it saying so. We will want to add it to testing first.
 * Do not package alpha software. Period!
 * Periodically check to see if a package you have submitted has an update available. If it does, create a new package, test and submit it.
 * Don't forget your changelog entry. Be sure you use the day, date (Month and numbered day of the month) and year of the day you build the package. Add your info to the entry. Do not use the info of the previous packager.

Dependency management
Slightly modified version of a post by TerryN on the forum :

Many people who are new to packaging think that dependency management is only about the Provides: and Requires: tags in the spec file. These are only part of the story because rpm does automatic dependency generation as part of the rpmbuild process. This is handled by 2 scripts find-provides and find-requires which you will see run towards the end of the build.


 * find-provides : This script examines all the files in the %files list and if any shared libraries are found the soname of the lib is added to the list of capabilities provided by the rpm. This means that if another package depends on that shared lib your package can be selected by apt/synaptic to satisfy that dependency.
 * - Be very careful when packaging apps where the SOURCES contain pre-built libraries as they will be added and you could find your package being selected as a dependency of a seemingly totally unrelated application (Calibre springs to mind). Worse still, this can lead to instability if those libraries are used with another application (because they were not built on PCLinuxOS).
 * - To avoid this you can add Autoprov: No to the summary section of the spec file (that is where the Provides/BuildRequires/Requires/Obsoletes tags go) for these types of package (and only these types of package!). Autoprov: No
 * - Documentation for these tags is here.


 * find-requires : This script examines all the files in the %files list and effectively runs an ldd ( more about ldd ) on any executable to see what shared-libraries are needed by the executable. The sonames for these libraries are added to the list of capabilities required by the rpm and will need to be resolved when the rpm is installed.
 * - Once again pre-built binaries in the SOURCE can cause issues as they may reference sonames which cannot be resolved by existing packages in the repo.
 * - An example would be because the SOURCE contains binaries for multiple architectures (see example here)


 * Devel dependencies :
 * There is a further twist with shared objects which are to be used as system shared libraries (there are other uses for .so files). The build will produce 3 "copies" of the lib (actually only 1 is a real file the others are links but that doesn't matter for the purposes here) :

/usr/lib/libxyz.so.1.2.3 /usr/lib/libxyz.so.1 /usr/lib/libxyz.so
 * The first 2 (the versioned files) are required at runtime and should be in the main package. The 3rd one (libxyz.so) is only required by the linker when building apps against the shared lib and should therefore be in the -devel package (along with header files).


 * RPM is clever enough to know this and generates "devel" provides/requires for that file. Therefore if you put the unversioned .so in the main package the automatic dependency management will generate a "devel" provides:

[terry@localhost ~]$ rpm -q --provides kde-workspace-core | grep devel devel(libkwinglesutils) devel(libkwinglutils) devel(liboxygenstyleconfig) devel(libpowerdevilconfigcommonprivate) devel(libpowerdevilui)
 * and therefore devel dependencies:

[terry@localhost ~]$ rpm -qR kde-workspace-core | grep devel devel(libdl) devel(libEGL) devel(libgcc_s) devel(libGL) devel(libGLESv2) devel(libICE) devel(libkdecore) devel(libkdeui) devel(libkephal) devel(libkwineffects) devel(libm) devel(libpowerdevilcore) devel(libpowerdevilui) devel(libQtCore) devel(libQtDBus) devel(libQtGui) devel(libQtSvg) devel(libSM) devel(libstdc++) devel(libX11) devel(libXau) devel(libXdmcp) devel(libXext) devel(libXft) devel(libXpm
 * and these devel dependencies will have their own dependencies etc. etc. This could result in a large amount of unnecessary -devel packages and other "build" packages being installed on systems which are not used for software building. This can consume a lot of space and make things like remastering more difficult.


 * If you are sure that nothing will ever want to link/build against such a library you could avoid creating a -devel subpackage by either deleting the .so at the end of the %install section

rm -f %buildroot/%{_libdir}/libxyz.so
 * or using a %exclude' for the file in the %files'' list.

%files ... %exclude %{_libdir}/libxyz.so
 * but if there is any doubt it is safer to provide a -devel package.

To summarize, great care must be taken with dependency management when dealing with system shared libraries. It is complicated, tedious and certainly not glamorous but it IS necessary to avoid the repo descending into bloat and chaos.

Packagers Tips and Hints

 * Instructions specific to particular categories of packages are listed on Handling package categories page.
 * This is a running thread on the forum with many important tidbits of information to guide you in your packaging : http://www.pclinuxos.com/forum/index.php/topic,101266.0.html
 * Old and new libraries and how they affect updates : http://www.pclinuxos.com/forum/index.php/topic,110765.0/topicseen.html

Errors and Warnings
It would have to be the rarest of rare packages that builds without a single Error or Warning. Here is a list of some of them and wherever known, how to deal with them.''' This is seen in case of packages where a separate subdirectory is created for config/build files. Commonly it is called build but the exact name can be made out by reading the build log. This can be fixed by adding -C nameofsubdirectory to the make install macro in %install section of .spec file. TerryN - forum.
 * '''file not found by glob
 * make: *** No rule to make target `install'. Stop.

Reading references

 * Repository and Sections and Mirrors
 * Basic PCLinuxOS packaging manual - http://www.montanalinux.org/files/RPM_Build_Manual_Rev2.pdf
 * Slightly updated version of above manual - https://dl.dropbox.com/u/32533806/Building_RPMS.tar.gz
 * Packaging software with RPM, Part 1: Building and distributing packages - http://www.ibm.com/developerworks/library/l-rpm1/
 * Packaging software with RPM, Part 2: Upgrading and uninstalling software - http://www.ibm.com/developerworks/library/l-rpm2/
 * Packaging software with RPM, Part 3: Accommodating software dependencies' - http://www.ibm.com/developerworks/library/l-rpm3/
 * How to make good RPM packages - http://www.redhat.com/promo/summit/2008/downloads/pdf/Wednesday_130pm_Tom_Callaway_OSS.pdf
 * RPM Guide - http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/index.html
 * Maximum RPM - http://www.rpm.org/max-rpm-snapshot/index.html
 * Macros - http://www.zarb.org/~jasonc/macros.php
 * Problems and issues with packaging - http://wiki.mandriva.com/en/Development/Packaging/Problems
 * Program Library HOWTO - http://tldp.org/HOWTO/Program-Library-HOWTO/index.html
 * Creating an RPM for a Java Application - http://javaworkshop.wordpress.com/2008/10/22/rolling-up-an-rpm-for-a-java-application/