What is in this release:
The release for 0.3 contains a few scripts, chkpkg-koji.py, asmcheck.sh, and asmanalyzer.py. The chkpkg-koji.py is the main new feature of this release which has been completed. The second script asmcheck.sh has not actually been changed since release for 0.2, but I decided to include it in this release since the scripts work together. The final script asmanalyzer.py has some small modifications which connect the chkpkg-koji.py and asmcheck.sh script together and performs logging of output for all scripts. Along with this release I has included the sample .csv files created from the process, which ran over only a few files.
Brief overview of asmcheck.sh:
This script was originally created by Jon Masters to scan source rpm files for assembly source files and inline assembly code. My small modifications simple store the data in a preferred format and saves it to files. The asmcheck.sh script will generate 6 out of 7 tables that are going to be used for analysis, the one table it does not generate is the asm-build.csv file. Again, nothing has changed in this script for this release, it is simply here because it is required to run all the proper scans and work together with the other scripts.
Overview and modifications of asmanalyzer.py:
This script has been modified to take an additional argument and pass it on to chkpkg-koji.py. This was done so all the configurations are completed by running this single script, asmanalyzer.py. This is the script that is run to start the process of generating the .csv files. There are two different way you can run this script: first is to run it as “./asmanalyzer.py tag_name packages directory”, second is to run it with the “-s”. When run without the -s switch it will download the selected packages source rpms, using yumdownloader –source, and begin starting the two other scripts. When run with the -s option, it will not download anything, it will only scan the specified directory for source rpms and begin analyzing. After looking over this script, it is a mess and I should take some time to fix it up with some of the new things I’ve learned with python.
./asmanalyzer.py [options] tag_name packages directory options - There is only one feature changing option, -s. When you run -s don't use the package argument. This will scan local files instead of downloading. tag_name - This is used to specify which fedora to build on(f17, f18, f19). packages - Used the same was you would use yum or yumdownloader. directory - The specified directory to scan and download to.
Overview of chkpkg-koji.py:
This script does not need to be directly run, and is going to be run most often through the asmanalyzer.py script. But you may still run this script manually for results. This script created the asm-build.csv file. This script is given a few arguments: the package name, package version, and tag_name(this is the same as asmanalyzer.py). Given these three arguments it will return the status of the most recent build of the package and version on what fedora was selected through the tag_name. The difference between this script and the release for 0.2 query-koji.py is quite significant. The chkpkg-koji.py script can query for a package based on package_name, package_version, and the tag. While the query-koji.py script did almost the same thing but could only query for packages based on the task_id. As I have complained before, koji is very poorly documented, and finding a way to do this was a pain. Querying on task_id alone was obviously a problem since you cannot tell what package hides behind each task_id. I will explain more about my experiences with koji in a later post. I should probably also mention that this script actually used a portion of the code from query-koji.py, and especially the knowledge gained from writing the query-koji.py.
For this release my goals has been met, I have finished the generation of all the tables(.csv files) and finished all the scripts(except maybe some tiny tweaking later on). The next step to take will be to get the database up and running, test to make sure the format made works(or tweak a little), and then finally run the scripts across the fedora package set(which will take a long time).
I plan to continue working on this project over my holidays(when I have time), since I feel like I’m so close to finishing. May also need to add a few comments in the script, but first I will explain more about the koji API in my next blog.
Here is the script for chkpkg-koji.py for anyone who does not wish to download and unpack my package:
#!/usr/bin/env python # Provide script with a package_name, version, and tag_name and it will # search through all current koji packages for successful build information. # The process is not very efficient, first it checks the package for the # build_id, then it checked the build id for the task_id, and finally, it # checks the task_id to check if it successfully built. import koji, optparse usage = 'usage: %prog [options] package_name version tag_name' parser = optparse.OptionParser(usage=usage) (options, args) = parser.parse_args() class checkKojiPackage: def __init__(self): if len(args) > 3 or len(args) < 3: parser.error('Provide a package_name, the version of the package, and a tag such as: f17, f18, f19') package_name = str(args) package_version = str(args) tag_name = str(args) # The period in the follow list is used to connect platform with the server kojilist = ['', 'arm.', 'ppc.', 's390.', 'sparc.'] list_len = 0 build_status = "" report = "" for platform in kojilist: build_status = self.checkPackage(package_name, package_version, platform, tag_name) list_len += 1 if list_len != len(kojilist): report = report + build_status + "," else: report = report + build_status self.sendToTable(report) # Writing the build table def sendToTable(self, output): table = open('asm-build.csv', 'a+') table.write(output + "\n") table.close() # Loop through koji servers and check if package exists, extract the the task_id def checkPackage(self, package_name, package_version, platform, tag_name): server = "http://" + platform + "koji.fedoraproject.org/kojihub/" session = koji.ClientSession(server) tables = session.queryHistory(package=package_name) build_status = "" print "\n[" + "- " *10 + server + " -" *10 + "]\n" #print histdata for table in tables: build = tables[table] pre_release = 0 for x in build: try: if int(x['release']) < pre_release: break elif str(x['version']) == package_version and x['tag.name'] == tag_name: pre_release = int(x['release']) build_info = session.getBuild(x['build_id']) task_id = build_info['task_id'] task = session.getTaskInfo(task_id, request=True) task_state = task['state'] task_request = task['request'] task_method = task['method'] print "\nbuild_id: " + str(x['build_id']) print "task_id: " + str(task_id) print "version: " + str(x['version']) print "release: " + str(x['release']) print "tag_name: " + str(x['tag.name']) # If FREE or OPEN is true then the package has not finished building on koji if task_state in (koji.TASK_STATES['FREE'], koji.TASK_STATES['OPEN']): print "Task: " + str(task_id) + "\nStatus: Not completed" build_status = "in-progress" # If CLOSED is true then the package finished building successfully elif task_state == koji.TASK_STATES['CLOSED']: print "Task: " + str(task_id) + "\nStatus: Successful" build_status = "successful" # If CANCELED is true then the package was cancelled before it was completed elif task_state == koji.TASK_STATES['CANCELED']: print "Task: " + str(task_id) + "\nStatus: Cancelled" build_status = "cancelled" # If FAILED is true then the package failed the build before it was completed elif task_state == koji.TASK_STATES['FAILED']: print "Task: " + str(task_id) + "\nStatus: Failed" build_status = "failed" else: print "Task: " + str(task_id) + "\nStatus: Other" build_status = "other" # Only really care about build and buildArch, as these will be building the rpm files if task_method == "build" or task_method == "buildArch": print "Method: " + task_method #print str(task_request) else: print "Method: " + str(task_method) #print str(task_request) except: pass # End of for statement print build_status if build_status: return build_status else: return "no-package" if __name__ == '__main__': checkKojiPackage()