Strange buildbot python behavior

Question

EDIT for some explanations: The buildbot is a continous integration system in python which can be controlled by a web interface. In this web interface you have a "Waterfall" page where you can choose a specific builder and trigger a build with a "Force build" button.

URL: http://trac.buildbot.net/

The background: We have a cluster of builders which are continous (check every few minutes if a change happened and if yes, rebuild) or nightly (build every night). Our system so far has only one specific enterprise builder for every project. This was done by enforcing that every project must be found under a constant URL like

https://myrepositioryurl/{$projectname}.

Then when a project needs an enterprise build you need to choose one project XYZ and the buildbot assumes that the project needs to be checked out under

https://myrepositioryurl/{$projectname}.

This is very rigid and I wanted to reconfigure the buildbot that projects can be under different URLs. During the setup of the builders which is started by "buildbot start master" a config file of our projects is read and stored in a ConfigParser object. In the following source code it is the clzoptions var und my URL I want to use should be

https://mytesturl/XYZ.

for project XYZ. I added now my different URLs under a "svnBaseURL" entry for a test project. Now I have encountered something which I do not quite understand in my python class which creates the builders. First the source:

import os
import logging

from xcodebuild import xcodebuild
from projects import Projects

from buildbot.config import BuilderConfig
from buildbot.process.factory import BuildFactory
from buildbot.steps.source import SVN
from buildbot.steps.shell import ShellCommand, WithProperties
from buildbot.process.properties import Property


class builders(object):
    clzoptions = Projects().options


    def __init__(self):
        aProject = Projects()
        self.options = aProject.options


    def enterprise_checkout_url(self, curProjectName):
        return curProjectName

    def create_enterprise_builder(self):
        factory = BuildFactory()
        factory.addStep(ShellCommand(name='svn checkout',
                                     haltOnFailure=True,
                                     description='svn checkout',
                                     descriptionDone='svn checkout done',
                                     command=['svn', 'checkout', '--username', 'admin', self.enterprise_checkout_url(WithProperties('%(project)s')), '.']))



        builderConfig = BuilderConfig(name="foobuilder",
                                      category="OnDemand",
                                      slavenames=[ "buildslave01" ],
                                      factory=factory)
        return builderConfig



    def get_all_builders(self):
        builders = []

        builders.append(self.create_enterprise_builder())

        return builders

I have melted it down to the core problem, there are many more builders inside. The key function is self.enterprise_checkout_url(WithProperties('%(project)s')).

If I call that builder with the project name "XYZ" in the Waterfall, I get as result

svn checkout --username admin XYZ .

for the ShellCommand. While this is nonsensical because it is not an URL, I see that the parameter curProjectName evaluates to "XYZ". Easy so far,right ? Lets change now that function...

def enterprise_checkout_url(self, curProjectName):
  return builders.clzoptions.get("XYZ", "svnBaseURL"))

and get

svn checkout --username admin https://mytesturl/XYZ .

This is nearly the thing I need,

https://mytesturl/XYZ

is the right path. But the key is constant, I need it to be variable. But at least I know that the dictionary exists and has the correct entry for XYZ.

Now the problem I simply do not understand.

Lets try now

def enterprise_checkout_url(self, curProjectName):
      return builders.clzoptions.get(curProjectName, "svnBaseURL"))

and oops, he does not build

ConfigParser.NoSectionError: No section: <buildbot.process.properties.WithProperties instance at 0x1073739e0>

Ok, during the compile phase curProjectName may not be set, how about:

def enterprise_checkout_url(self, curProjectName):
    projects = builders.clzoptions.sections()
    for project in projects:
      if project == curProjectName:
        return builders.clzoptions.get(project, "svnBaseURL" )

which compiles. I am getting all my projects, test if the curProjectName is right and then return my svnBaseURL with the project key which should be equal to curProjectName. But I get:

<type 'exceptions.TypeError'>: 'NoneType' object is not iterable

Your turn. I have tried to use str(), repr(), eval() on curProjectName, but to no avail. I cannot access both the existing dictionary and curProjectName.

Answer 1

Does this help you ?

class builders(object):

    builders = []
    print 'id of buiders just created ==',id(builders)

    def __init__(self,x):
        self.the = x

    def enterprise_checkout_url(self, curProjectName):
        return curProjectName

    def create_enterprise_builder(self,yy):
        builderConfig = dict(descriptionDone='-SVN-',
                             command=yy)
        return builderConfig

    def get_all_builders(self):
        print 'id of builders inside get_all_builders ==',id(builders)
        print 'id of builders.builders inside get_all_builders ==',id(builders.builders)

        builders.builders.append(self.create_enterprise_builder((self.the)))

        return builders.builders

print 'id of class builders ==',id(builders)
print '\n################################\n'

b = builders('BOF')
print b.get_all_builders()

print '\n=================================\n'

b2 = builders('MOTO')
print b2.get_all_builders()

result

id of buiders just created == 18709040
id of class builders == 13819408

################################

id of builders inside get_all_builders == 13819408
id of builders.builders inside get_all_builders == 18709040
[{'descriptionDone': '-SVN-', 'command': 'BOF'}]

=================================

id of builders inside get_all_builders == 13819408
id of builders.builders inside get_all_builders == 18709040
[{'descriptionDone': '-SVN-', 'command': 'BOF'}, {'descriptionDone': '-SVN-', 'command': 'MOTO'}]

EDIT

Ther's a problem with my code.
If the instruction print b2.get_all_builders() is executed two times, the result is

id of buiders just created == 18709040
id of class builders == 13819408

################################

id of builders inside get_all_builders == 13819408
id of builders.builders inside get_all_builders == 18709040
[{'descriptionDone': '-SVN-', 'command': 'BOF'}]

=================================

id of builders inside get_all_builders == 13819408
id of builders.builders inside get_all_builders == 18709040
[{'descriptionDone': '-SVN-', 'command': 'BOF'}, {'descriptionDone': '-SVN-', 'command': 'MOTO'}]

id of builders inside get_all_builders == 13819408
id of builders.builders inside get_all_builders == 18709040
[{'descriptionDone': '-SVN-', 'command': 'BOF'}, {'descriptionDone': '-SVN-', 'command': 'MOTO'}, {'descriptionDone': '-SVN-', 'command': 'MOTO'}]

One of the dictionaries appears two times.

Since I don't understand very well your problem and I'm not sure of what you want exactly, I don't know how to correct it