-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathdev.notes
219 lines (161 loc) · 11.3 KB
/
dev.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
The essential machinery for being able to reflect an external CI/Build system's job activity
in Agile Central is operable.
The primary features of the Jenkins BLD Connector for Agile Central at this time are:
Job Build information from Jenkins is reflected in Agile Central in Build records
(each of which is tied to an AC BuildDefinition record)
Configuration is via a YAML file
3 sections
o AgileCentral system id/creds, workspace and project info
o Jenkins system id/creds, facility for policy based job inclusion/exclusion
via regexes for multiple views/jobs, ability to specify AC targeted project.
o Service - to specify Preview status, logging level, policy for jobs
whose name doesn't appear in the specified project.
provides facility to economically cover interesting jobs on a Jenkins instance
in a single relatively small and easily understood file.
Cacheing of AC BuildDefinition info
reduces time spent getting infrequently changed data
Can extend to have the connector write in the last processed build time on a
per Jenkins job basis in the time_file.
Jenkins Job URI is written to the BuildDefinition at creation time
(if one is created in the course of running the connector)
Jenkins Job Build URI is written to the Build record
(enables AC user to click on link and see Jenkins page for the specific job)
You can run multiple configs per invocation.
*** OBVIATED *** we no longer allow the StrictProject tag in a config file
If your team/project morphs into another AC project, you might not have to
update your config file(s) if you've set the Service -> StrictProject to False
(If you have a BuildDefinition for the job in question associated with another
Project that the account accessing Agile Central can find, and the original
Project is moribund or has been renamed, the connector will post the Builds
to be associated with the BuildDefinition with the Name of the Jenkins job.)
*********************************
Ability to handle Jenkins instances that have the CloudBees Folders plugin.
(in this context a folder is really just a job, that happens to have subsidiary jobs)
The current way to handle these is to have
Folders:
- Folder: <name>
# you can settings associated with the above folder, for instance
#AgileCentral_Project: <some project name>
#exclude: regex1,regex2 etc
- Folder: <some other name>
#---------------------------------------------------------------------------------------------------
What remains to be done?
More industrial sized testing for processing multiple Jenkins Views
Linkage of Builds to Changeset items
Jenkins provides information about any VCS id/comments and affected files that led to
the triggering of a Jenkins Build.
It may be feasible for the Jenkins BLD Connector to determine whether there are any
AC Changesets present that match the Jenkins info and use that information in the
Build record posted to AC in the Changesets collection.
Alternatively, if the VCS Changeset info in Jenkins is not reflected in Agile Central,
it might be feasible to create those AC Changeset items and then include that
information in the Build record posted to AC in the Changesets collection.
Run some tests against almci Jenkins instance, pick some folders, set up some includes/excludes
and have the AgileCentral section point to the trial AC stack,
Put it in the hands of the 10 Foot Pole team and get some feedback.
#-----------------------------------------------------------------------------------------
Running the connector on a periodic basis on Mac OSX
Apple now recommends using launchctl / launchd over crontab / cron
There's a useful article at http://alvinalexander.com/mac-os-x/mac-osx-startup-crontab-launchd-jobs
Basically you copy some XML boilerplate, edit to reflect your conditions and script, write a script that
goes in to your ~/bin directory and tell launchctl to load your scheduled job.
The specifics follow...
We need two files:
#1 a file in $HOME/bin directory that invokes our connector script, bldeif_connector
#2 an XML file in a $HOME/Library/LaunchAgents directory that references the file above
- launchctl load command that load file #2
For file #1 in $HOME/bin , we called it run_bldeif_connector.sh and it has this content:
-----------file starts after this line----------
#!/bin/zsh
BLDEIF_DIR=/Users/pairing/trump/bldeif-0.9.0
PYTHON35="/Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5"
CONFIG_FILE="product_x.yml"
SANITY_MSG="I got started from launchctl to run the bldeif_connector"
(cd $BLDEIF_DIR; $PYTHON35 bldeif_connector $CONFIG_FILE; echo "$SANITY_MSG")
-----------file has ended with the previous line----------
We made this file executable by the owner (pairing), and tested it by running it and saw that
a log file was written with information relevant to the run (successful) and the msg was written to stdout.
For file #2, we put it in $HOME/Library/LaunchAgents (we could have put it in $HOME/Library/LaunchDaemons for normal use)
we called it 'com.ca.bldeif.plist' , it doesn't seem to matter what the permissions are on this file
the content of this rancid XML file follows:
-----------file starts after this line----------
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.ca.bldeif</string>
<key>ProgramArguments</key>
<array>
<string>/Users/pairing/bin/run_bldeif_connector.sh</string>
</array>
<key>Nice</key>
<integer>1</integer>
<key>StartInterval</key>
<integer>60</integer>
<key>RunAtLoad</key>
<true/>
<key>StandardErrorPath</key>
<string>/Users/pairing/trump/bldeif-0.9.0/bldeif.err</string>
<key>StandardOutPath</key>
<string>/Users/pairing/trump/bldeif-0.9.0/bldeif.out</string>
</dict>
</plist>
-----------file has ended with the previous line----------
Now, run the launchctl load command from within the $HOME/Library/LaunchAgents directory thusly:
launchctl load com.ca.bldeif.plit
Now your job will run immediately and then every 60 seconds (according to the entry in the plist file) therafter.
If you want to run the job at a greater interval, use a value expressed in seconds, ie., 1 hour = 60 * 60 = 3600.
seconds are the default unit of measurement for this timing field.
you can override this by putting in a <key>Minute</key> or <key>Hour</key><integer>3</integer><keyMinute</key><integer>55</integer>
says run it every day at 3:55
#-----------------------------------------------------------------------------------------
Running the connector on a periodic basis on Linux
use cron
run crontab -e and put in an entry like:
#--------------------------------------------------------------------------------------------
Installing and running the connector on a Windows platform
Get the Python 3.5 Windows install from www.python.org
We recommend using the 64 bit version
You will need to install a Windows specific module called win32com which you can obtain at:
https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220/pywin32-220.win-amd64-py3.5.exe/download
#----------------------------------------------------------------------------------------------
Running a connector as post-build action:
(3) no post-build plugin, add/configure a Jenkins job whose only task is to run the connector
In Build>Excecute Shell of a job's configuration add something like this:
WORK_DIR=/home/integrations/sw/bldeif-0.9.4
PYTHON=/usr/local/bin/python3.5
cd $WORK_DIR; export PYTHONPATH="$WORK_DIR"; $PYTHON bldeif_connector product_x.yml
(In case we want to pick up a new build add another job under Build Triggers > Build after other projects are built
e.g. in Projects to Watch we added "Cowardly Weasels, black-swan"
A limitation of Projects to Watch input box is that it only accepts first level jobs, and not jobs in folders or non-All views
This job, e.g. Cowardly Weasels or black-swan has to be triggered or manually run separately so that in turn it triggers the connector job.
(2) with post-build plugin, you must instal the PostBuild Script plugin.
for each Jenkins job you want to trigger a connector run. You must add/configure a Jenkins job with Post-build Actions section that provides
a path to a script that invokes the connector's executable,
e.g. File Script Path: /home/jenkins/postbuild/jenk_da_connecteur
content of jenk_da_connecteur file-------------------------------
#!/bin/sh
(cd /home/integrations/sw/bldeif-0.9.4; /usr/local/bin/python3.5 bldeif_connector product_x.yml)
end of file------------------------------------------------------
When a jenkins job is built, as long as this job has a post build script action set that mentions the target script that in turn invokes the
connector's executable, the connector is run after the build is complete, hence the new build is considered by the connector.
Two notable drawbacks:
1) the connector is run as the jenkins user (or whatever user owns the Jenkins process)
2) each Job has to have this post build script action that mentions the target script
For both (2) and (3) the directory where the connector is located has to be opened with chmod to allow users other than the owner of the directory,
in our case, intergrations user, to write to it (the lock file, time file, log file have to be written)
(1) is a variation of (2), but with the added benefit of switching the user from jenkins to integrations. We tried to achive this by following
these SO posts:
http://stackoverflow.com/questions/1988249/how-do-i-use-su-to-execute-the-rest-of-the-bash-script-as-that-user
http://stackoverflow.com/questions/1401002/trick-an-application-into-thinking-its-stdin-is-interactive-not-a-pipe
It did not work. Here is an example of a post-build action script. We notice that user switch did not happen
content of jenk_da_connecteur file-------------------------------
#!/bin/sh
da_connecteur="cd ~/sw/bldeif-0.9.4; /usr/local/bin/python3.5 bldeif_connector product_x.yml"
function faketty { script -qfc "$(printf "%q " "$@")"; }
#faketty sudo -u integrations -H sh -c $da_connecteur
#faketty sudo -u integrations -H sh -c "(cd /home/integrations/sw/bldeif-0.9.4; /usr/local/bin/python3.5 bldeif_connector product_x.yml)"
faketty sudo -u integrations -H sh -c "pwd; ls -la > your_files_are;"
end of file------------------------------------------------------
Parhaps commenting out this bogon in sudoers file "#Defaults requiretty"...but we still get "no tty present and no askpass program specified"