Commit | Line | Data |
---|---|---|
86530b38 AT |
1 | =head1 NAME |
2 | ||
3 | perlhack - How to hack at the Perl internals | |
4 | ||
5 | =head1 DESCRIPTION | |
6 | ||
7 | This document attempts to explain how Perl development takes place, | |
8 | and ends with some suggestions for people wanting to become bona fide | |
9 | porters. | |
10 | ||
11 | The perl5-porters mailing list is where the Perl standard distribution | |
12 | is maintained and developed. The list can get anywhere from 10 to 150 | |
13 | messages a day, depending on the heatedness of the debate. Most days | |
14 | there are two or three patches, extensions, features, or bugs being | |
15 | discussed at a time. | |
16 | ||
17 | A searchable archive of the list is at either: | |
18 | ||
19 | http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/ | |
20 | ||
21 | or | |
22 | ||
23 | http://archive.develooper.com/perl5-porters@perl.org/ | |
24 | ||
25 | List subscribers (the porters themselves) come in several flavours. | |
26 | Some are quiet curious lurkers, who rarely pitch in and instead watch | |
27 | the ongoing development to ensure they're forewarned of new changes or | |
28 | features in Perl. Some are representatives of vendors, who are there | |
29 | to make sure that Perl continues to compile and work on their | |
30 | platforms. Some patch any reported bug that they know how to fix, | |
31 | some are actively patching their pet area (threads, Win32, the regexp | |
32 | engine), while others seem to do nothing but complain. In other | |
33 | words, it's your usual mix of technical people. | |
34 | ||
35 | Over this group of porters presides Larry Wall. He has the final word | |
36 | in what does and does not change in the Perl language. Various | |
37 | releases of Perl are shepherded by a ``pumpking'', a porter | |
38 | responsible for gathering patches, deciding on a patch-by-patch | |
39 | feature-by-feature basis what will and will not go into the release. | |
40 | For instance, Gurusamy Sarathy was the pumpking for the 5.6 release of | |
41 | Perl, and Jarkko Hietaniemi is the pumpking for the 5.8 release, and | |
42 | Hugo van der Sanden will be the pumpking for the 5.10 release. | |
43 | ||
44 | In addition, various people are pumpkings for different things. For | |
45 | instance, Andy Dougherty and Jarkko Hietaniemi share the I<Configure> | |
46 | pumpkin. | |
47 | ||
48 | Larry sees Perl development along the lines of the US government: | |
49 | there's the Legislature (the porters), the Executive branch (the | |
50 | pumpkings), and the Supreme Court (Larry). The legislature can | |
51 | discuss and submit patches to the executive branch all they like, but | |
52 | the executive branch is free to veto them. Rarely, the Supreme Court | |
53 | will side with the executive branch over the legislature, or the | |
54 | legislature over the executive branch. Mostly, however, the | |
55 | legislature and the executive branch are supposed to get along and | |
56 | work out their differences without impeachment or court cases. | |
57 | ||
58 | You might sometimes see reference to Rule 1 and Rule 2. Larry's power | |
59 | as Supreme Court is expressed in The Rules: | |
60 | ||
61 | =over 4 | |
62 | ||
63 | =item 1 | |
64 | ||
65 | Larry is always by definition right about how Perl should behave. | |
66 | This means he has final veto power on the core functionality. | |
67 | ||
68 | =item 2 | |
69 | ||
70 | Larry is allowed to change his mind about any matter at a later date, | |
71 | regardless of whether he previously invoked Rule 1. | |
72 | ||
73 | =back | |
74 | ||
75 | Got that? Larry is always right, even when he was wrong. It's rare | |
76 | to see either Rule exercised, but they are often alluded to. | |
77 | ||
78 | New features and extensions to the language are contentious, because | |
79 | the criteria used by the pumpkings, Larry, and other porters to decide | |
80 | which features should be implemented and incorporated are not codified | |
81 | in a few small design goals as with some other languages. Instead, | |
82 | the heuristics are flexible and often difficult to fathom. Here is | |
83 | one person's list, roughly in decreasing order of importance, of | |
84 | heuristics that new features have to be weighed against: | |
85 | ||
86 | =over 4 | |
87 | ||
88 | =item Does concept match the general goals of Perl? | |
89 | ||
90 | These haven't been written anywhere in stone, but one approximation | |
91 | is: | |
92 | ||
93 | 1. Keep it fast, simple, and useful. | |
94 | 2. Keep features/concepts as orthogonal as possible. | |
95 | 3. No arbitrary limits (platforms, data sizes, cultures). | |
96 | 4. Keep it open and exciting to use/patch/advocate Perl everywhere. | |
97 | 5. Either assimilate new technologies, or build bridges to them. | |
98 | ||
99 | =item Where is the implementation? | |
100 | ||
101 | All the talk in the world is useless without an implementation. In | |
102 | almost every case, the person or people who argue for a new feature | |
103 | will be expected to be the ones who implement it. Porters capable | |
104 | of coding new features have their own agendas, and are not available | |
105 | to implement your (possibly good) idea. | |
106 | ||
107 | =item Backwards compatibility | |
108 | ||
109 | It's a cardinal sin to break existing Perl programs. New warnings are | |
110 | contentious--some say that a program that emits warnings is not | |
111 | broken, while others say it is. Adding keywords has the potential to | |
112 | break programs, changing the meaning of existing token sequences or | |
113 | functions might break programs. | |
114 | ||
115 | =item Could it be a module instead? | |
116 | ||
117 | Perl 5 has extension mechanisms, modules and XS, specifically to avoid | |
118 | the need to keep changing the Perl interpreter. You can write modules | |
119 | that export functions, you can give those functions prototypes so they | |
120 | can be called like built-in functions, you can even write XS code to | |
121 | mess with the runtime data structures of the Perl interpreter if you | |
122 | want to implement really complicated things. If it can be done in a | |
123 | module instead of in the core, it's highly unlikely to be added. | |
124 | ||
125 | =item Is the feature generic enough? | |
126 | ||
127 | Is this something that only the submitter wants added to the language, | |
128 | or would it be broadly useful? Sometimes, instead of adding a feature | |
129 | with a tight focus, the porters might decide to wait until someone | |
130 | implements the more generalized feature. For instance, instead of | |
131 | implementing a ``delayed evaluation'' feature, the porters are waiting | |
132 | for a macro system that would permit delayed evaluation and much more. | |
133 | ||
134 | =item Does it potentially introduce new bugs? | |
135 | ||
136 | Radical rewrites of large chunks of the Perl interpreter have the | |
137 | potential to introduce new bugs. The smaller and more localized the | |
138 | change, the better. | |
139 | ||
140 | =item Does it preclude other desirable features? | |
141 | ||
142 | A patch is likely to be rejected if it closes off future avenues of | |
143 | development. For instance, a patch that placed a true and final | |
144 | interpretation on prototypes is likely to be rejected because there | |
145 | are still options for the future of prototypes that haven't been | |
146 | addressed. | |
147 | ||
148 | =item Is the implementation robust? | |
149 | ||
150 | Good patches (tight code, complete, correct) stand more chance of | |
151 | going in. Sloppy or incorrect patches might be placed on the back | |
152 | burner until the pumpking has time to fix, or might be discarded | |
153 | altogether without further notice. | |
154 | ||
155 | =item Is the implementation generic enough to be portable? | |
156 | ||
157 | The worst patches make use of a system-specific features. It's highly | |
158 | unlikely that nonportable additions to the Perl language will be | |
159 | accepted. | |
160 | ||
161 | =item Is the implementation tested? | |
162 | ||
163 | Patches which change behaviour (fixing bugs or introducing new features) | |
164 | must include regression tests to verify that everything works as expected. | |
165 | Without tests provided by the original author, how can anyone else changing | |
166 | perl in the future be sure that they haven't unwittingly broken the behaviour | |
167 | the patch implements? And without tests, how can the patch's author be | |
168 | confident that his/her hard work put into the patch won't be accidentally | |
169 | thrown away by someone in the future? | |
170 | ||
171 | =item Is there enough documentation? | |
172 | ||
173 | Patches without documentation are probably ill-thought out or | |
174 | incomplete. Nothing can be added without documentation, so submitting | |
175 | a patch for the appropriate manpages as well as the source code is | |
176 | always a good idea. | |
177 | ||
178 | =item Is there another way to do it? | |
179 | ||
180 | Larry said ``Although the Perl Slogan is I<There's More Than One Way | |
181 | to Do It>, I hesitate to make 10 ways to do something''. This is a | |
182 | tricky heuristic to navigate, though--one man's essential addition is | |
183 | another man's pointless cruft. | |
184 | ||
185 | =item Does it create too much work? | |
186 | ||
187 | Work for the pumpking, work for Perl programmers, work for module | |
188 | authors, ... Perl is supposed to be easy. | |
189 | ||
190 | =item Patches speak louder than words | |
191 | ||
192 | Working code is always preferred to pie-in-the-sky ideas. A patch to | |
193 | add a feature stands a much higher chance of making it to the language | |
194 | than does a random feature request, no matter how fervently argued the | |
195 | request might be. This ties into ``Will it be useful?'', as the fact | |
196 | that someone took the time to make the patch demonstrates a strong | |
197 | desire for the feature. | |
198 | ||
199 | =back | |
200 | ||
201 | If you're on the list, you might hear the word ``core'' bandied | |
202 | around. It refers to the standard distribution. ``Hacking on the | |
203 | core'' means you're changing the C source code to the Perl | |
204 | interpreter. ``A core module'' is one that ships with Perl. | |
205 | ||
206 | =head2 Keeping in sync | |
207 | ||
208 | The source code to the Perl interpreter, in its different versions, is | |
209 | kept in a repository managed by a revision control system ( which is | |
210 | currently the Perforce program, see http://perforce.com/ ). The | |
211 | pumpkings and a few others have access to the repository to check in | |
212 | changes. Periodically the pumpking for the development version of Perl | |
213 | will release a new version, so the rest of the porters can see what's | |
214 | changed. The current state of the main trunk of repository, and patches | |
215 | that describe the individual changes that have happened since the last | |
216 | public release are available at this location: | |
217 | ||
218 | http://public.activestate.com/gsar/APC/ | |
219 | ftp://ftp.linux.activestate.com/pub/staff/gsar/APC/ | |
220 | ||
221 | If you're looking for a particular change, or a change that affected | |
222 | a particular set of files, you may find the B<Perl Repository Browser> | |
223 | useful: | |
224 | ||
225 | http://public.activestate.com/cgi-bin/perlbrowse | |
226 | ||
227 | You may also want to subscribe to the perl5-changes mailing list to | |
228 | receive a copy of each patch that gets submitted to the maintenance | |
229 | and development "branches" of the perl repository. See | |
230 | http://lists.perl.org/ for subscription information. | |
231 | ||
232 | If you are a member of the perl5-porters mailing list, it is a good | |
233 | thing to keep in touch with the most recent changes. If not only to | |
234 | verify if what you would have posted as a bug report isn't already | |
235 | solved in the most recent available perl development branch, also | |
236 | known as perl-current, bleading edge perl, bleedperl or bleadperl. | |
237 | ||
238 | Needless to say, the source code in perl-current is usually in a perpetual | |
239 | state of evolution. You should expect it to be very buggy. Do B<not> use | |
240 | it for any purpose other than testing and development. | |
241 | ||
242 | Keeping in sync with the most recent branch can be done in several ways, | |
243 | but the most convenient and reliable way is using B<rsync>, available at | |
244 | ftp://rsync.samba.org/pub/rsync/ . (You can also get the most recent | |
245 | branch by FTP.) | |
246 | ||
247 | If you choose to keep in sync using rsync, there are two approaches | |
248 | to doing so: | |
249 | ||
250 | =over 4 | |
251 | ||
252 | =item rsync'ing the source tree | |
253 | ||
254 | Presuming you are in the directory where your perl source resides | |
255 | and you have rsync installed and available, you can `upgrade' to | |
256 | the bleadperl using: | |
257 | ||
258 | # rsync -avz rsync://ftp.linux.activestate.com/perl-current/ . | |
259 | ||
260 | This takes care of updating every single item in the source tree to | |
261 | the latest applied patch level, creating files that are new (to your | |
262 | distribution) and setting date/time stamps of existing files to | |
263 | reflect the bleadperl status. | |
264 | ||
265 | Note that this will not delete any files that were in '.' before | |
266 | the rsync. Once you are sure that the rsync is running correctly, | |
267 | run it with the --delete and the --dry-run options like this: | |
268 | ||
269 | # rsync -avz --delete --dry-run rsync://ftp.linux.activestate.com/perl-current/ . | |
270 | ||
271 | This will I<simulate> an rsync run that also deletes files not | |
272 | present in the bleadperl master copy. Observe the results from | |
273 | this run closely. If you are sure that the actual run would delete | |
274 | no files precious to you, you could remove the '--dry-run' option. | |
275 | ||
276 | You can than check what patch was the latest that was applied by | |
277 | looking in the file B<.patch>, which will show the number of the | |
278 | latest patch. | |
279 | ||
280 | If you have more than one machine to keep in sync, and not all of | |
281 | them have access to the WAN (so you are not able to rsync all the | |
282 | source trees to the real source), there are some ways to get around | |
283 | this problem. | |
284 | ||
285 | =over 4 | |
286 | ||
287 | =item Using rsync over the LAN | |
288 | ||
289 | Set up a local rsync server which makes the rsynced source tree | |
290 | available to the LAN and sync the other machines against this | |
291 | directory. | |
292 | ||
293 | From http://rsync.samba.org/README.html : | |
294 | ||
295 | "Rsync uses rsh or ssh for communication. It does not need to be | |
296 | setuid and requires no special privileges for installation. It | |
297 | does not require an inetd entry or a daemon. You must, however, | |
298 | have a working rsh or ssh system. Using ssh is recommended for | |
299 | its security features." | |
300 | ||
301 | =item Using pushing over the NFS | |
302 | ||
303 | Having the other systems mounted over the NFS, you can take an | |
304 | active pushing approach by checking the just updated tree against | |
305 | the other not-yet synced trees. An example would be | |
306 | ||
307 | #!/usr/bin/perl -w | |
308 | ||
309 | use strict; | |
310 | use File::Copy; | |
311 | ||
312 | my %MF = map { | |
313 | m/(\S+)/; | |
314 | $1 => [ (stat $1)[2, 7, 9] ]; # mode, size, mtime | |
315 | } `cat MANIFEST`; | |
316 | ||
317 | my %remote = map { $_ => "/$_/pro/3gl/CPAN/perl-5.7.1" } qw(host1 host2); | |
318 | ||
319 | foreach my $host (keys %remote) { | |
320 | unless (-d $remote{$host}) { | |
321 | print STDERR "Cannot Xsync for host $host\n"; | |
322 | next; | |
323 | } | |
324 | foreach my $file (keys %MF) { | |
325 | my $rfile = "$remote{$host}/$file"; | |
326 | my ($mode, $size, $mtime) = (stat $rfile)[2, 7, 9]; | |
327 | defined $size or ($mode, $size, $mtime) = (0, 0, 0); | |
328 | $size == $MF{$file}[1] && $mtime == $MF{$file}[2] and next; | |
329 | printf "%4s %-34s %8d %9d %8d %9d\n", | |
330 | $host, $file, $MF{$file}[1], $MF{$file}[2], $size, $mtime; | |
331 | unlink $rfile; | |
332 | copy ($file, $rfile); | |
333 | utime time, $MF{$file}[2], $rfile; | |
334 | chmod $MF{$file}[0], $rfile; | |
335 | } | |
336 | } | |
337 | ||
338 | though this is not perfect. It could be improved with checking | |
339 | file checksums before updating. Not all NFS systems support | |
340 | reliable utime support (when used over the NFS). | |
341 | ||
342 | =back | |
343 | ||
344 | =item rsync'ing the patches | |
345 | ||
346 | The source tree is maintained by the pumpking who applies patches to | |
347 | the files in the tree. These patches are either created by the | |
348 | pumpking himself using C<diff -c> after updating the file manually or | |
349 | by applying patches sent in by posters on the perl5-porters list. | |
350 | These patches are also saved and rsync'able, so you can apply them | |
351 | yourself to the source files. | |
352 | ||
353 | Presuming you are in a directory where your patches reside, you can | |
354 | get them in sync with | |
355 | ||
356 | # rsync -avz rsync://ftp.linux.activestate.com/perl-current-diffs/ . | |
357 | ||
358 | This makes sure the latest available patch is downloaded to your | |
359 | patch directory. | |
360 | ||
361 | It's then up to you to apply these patches, using something like | |
362 | ||
363 | # last=`ls -t *.gz | sed q` | |
364 | # rsync -avz rsync://ftp.linux.activestate.com/perl-current-diffs/ . | |
365 | # find . -name '*.gz' -newer $last -exec gzcat {} \; >blead.patch | |
366 | # cd ../perl-current | |
367 | # patch -p1 -N <../perl-current-diffs/blead.patch | |
368 | ||
369 | or, since this is only a hint towards how it works, use CPAN-patchaperl | |
370 |