Commit | Line | Data |
---|---|---|
920dae64 AT |
1 | """New import scheme with package support. |
2 | ||
3 | Quick Reference | |
4 | --------------- | |
5 | ||
6 | - To enable package support, execute "import ni" before importing any | |
7 | packages. Importing this module automatically installs the relevant | |
8 | import hooks. | |
9 | ||
10 | - To create a package named spam containing sub-modules ham, bacon and | |
11 | eggs, create a directory spam somewhere on Python's module search | |
12 | path (i.e. spam's parent directory must be one of the directories in | |
13 | sys.path or $PYTHONPATH); then create files ham.py, bacon.py and | |
14 | eggs.py inside spam. | |
15 | ||
16 | - To import module ham from package spam and use function hamneggs() | |
17 | from that module, you can either do | |
18 | ||
19 | import spam.ham # *not* "import spam" !!! | |
20 | spam.ham.hamneggs() | |
21 | ||
22 | or | |
23 | ||
24 | from spam import ham | |
25 | ham.hamneggs() | |
26 | ||
27 | or | |
28 | ||
29 | from spam.ham import hamneggs | |
30 | hamneggs() | |
31 | ||
32 | - Importing just "spam" does not do what you expect: it creates an | |
33 | empty package named spam if one does not already exist, but it does | |
34 | not import spam's submodules. The only submodule that is guaranteed | |
35 | to be imported is spam.__init__, if it exists. Note that | |
36 | spam.__init__ is a submodule of package spam. It can reference to | |
37 | spam's namespace via the '__.' prefix, for instance | |
38 | ||
39 | __.spam_inited = 1 # Set a package-level variable | |
40 | ||
41 | ||
42 | ||
43 | Theory of Operation | |
44 | ------------------- | |
45 | ||
46 | A Package is a module that can contain other modules. Packages can be | |
47 | nested. Package introduce dotted names for modules, like P.Q.M, which | |
48 | could correspond to a file P/Q/M.py found somewhere on sys.path. It | |
49 | is possible to import a package itself, though this makes little sense | |
50 | unless the package contains a module called __init__. | |
51 | ||
52 | A package has two variables that control the namespace used for | |
53 | packages and modules, both initialized to sensible defaults the first | |
54 | time the package is referenced. | |
55 | ||
56 | (1) A package's *module search path*, contained in the per-package | |
57 | variable __path__, defines a list of *directories* where submodules or | |
58 | subpackages of the package are searched. It is initialized to the | |
59 | directory containing the package. Setting this variable to None makes | |
60 | the module search path default to sys.path (this is not quite the same | |
61 | as setting it to sys.path, since the latter won't track later | |
62 | assignments to sys.path). | |
63 | ||
64 | (2) A package's *import domain*, contained in the per-package variable | |
65 | __domain__, defines a list of *packages* that are searched (using | |
66 | their respective module search paths) to satisfy imports. It is | |
67 | initialized to the list consisting of the package itself, its parent | |
68 | package, its parent's parent, and so on, ending with the root package | |
69 | (the nameless package containing all top-level packages and modules, | |
70 | whose module search path is None, implying sys.path). | |
71 | ||
72 | The default domain implements a search algorithm called "expanding | |
73 | search". An alternative search algorithm called "explicit search" | |
74 | fixes the import search path to contain only the root package, | |
75 | requiring the modules in the package to name all imported modules by | |
76 | their full name. The convention of using '__' to refer to the current | |
77 | package (both as a per-module variable and in module names) can be | |
78 | used by packages using explicit search to refer to modules in the same | |
79 | package; this combination is known as "explicit-relative search". | |
80 | ||
81 | The PackageImporter and PackageLoader classes together implement the | |
82 | following policies: | |
83 | ||
84 | - There is a root package, whose name is ''. It cannot be imported | |
85 | directly but may be referenced, e.g. by using '__' from a top-level | |
86 | module. | |
87 | ||
88 | - In each module or package, the variable '__' contains a reference to | |
89 | the parent package; in the root package, '__' points to itself. | |
90 | ||
91 | - In the name for imported modules (e.g. M in "import M" or "from M | |
92 | import ..."), a leading '__' refers to the current package (i.e. | |
93 | the package containing the current module); leading '__.__' and so | |
94 | on refer to the current package's parent, and so on. The use of | |
95 | '__' elsewhere in the module name is not supported. | |
96 | ||
97 | - Modules are searched using the "expanding search" algorithm by | |
98 | virtue of the default value for __domain__. | |
99 | ||
100 | - If A.B.C is imported, A is searched using __domain__; then | |
101 | subpackage B is searched in A using its __path__, and so on. | |
102 | ||
103 | - Built-in modules have priority: even if a file sys.py exists in a | |
104 | package, "import sys" imports the built-in sys module. | |
105 | ||
106 | - The same holds for frozen modules, for better or for worse. | |
107 | ||
108 | - Submodules and subpackages are not automatically loaded when their | |
109 | parent packages is loaded. | |
110 | ||
111 | - The construct "from package import *" is illegal. (It can still be | |
112 | used to import names from a module.) | |
113 | ||
114 | - When "from package import module1, module2, ..." is used, those | |
115 | modules are explicitly loaded. | |
116 | ||
117 | - When a package is loaded, if it has a submodule __init__, that | |
118 | module is loaded. This is the place where required submodules can | |
119 | be loaded, the __path__ variable extended, etc. The __init__ module | |
120 | is loaded even if the package was loaded only in order to create a | |
121 | stub for a sub-package: if "import P.Q.R" is the first reference to | |
122 | P, and P has a submodule __init__, P.__init__ is loaded before P.Q | |
123 | is even searched. | |
124 | ||
125 | Caveats: | |
126 | ||
127 | - It is possible to import a package that has no __init__ submodule; | |
128 | this is not particularly useful but there may be useful applications | |
129 | for it (e.g. to manipulate its search paths from the outside!). | |
130 | ||
131 | - There are no special provisions for os.chdir(). If you plan to use | |
132 | os.chdir() before you have imported all your modules, it is better | |
133 | not to have relative pathnames in sys.path. (This could actually be | |
134 | fixed by changing the implementation of path_join() in the hook to | |
135 | absolutize paths.) | |
136 | ||
137 | - Packages and modules are introduced in sys.modules as soon as their | |
138 | loading is started. When the loading is terminated by an exception, | |
139 | the sys.modules entries remain around. | |
140 | ||
141 | - There are no special measures to support mutually recursive modules, | |
142 | but it will work under the same conditions where it works in the | |
143 | flat module space system. | |
144 | ||
145 | - Sometimes dummy entries (whose value is None) are entered in | |
146 | sys.modules, to indicate that a particular module does not exist -- | |
147 | this is done to speed up the expanding search algorithm when a | |
148 | module residing at a higher level is repeatedly imported (Python | |
149 | promises that importing a previously imported module is cheap!) | |
150 | ||
151 | - Although dynamically loaded extensions are allowed inside packages, | |
152 | the current implementation (hardcoded in the interpreter) of their | |
153 | initialization may cause problems if an extension invokes the | |
154 | interpreter during its initialization. | |
155 | ||
156 | - reload() may find another version of the module only if it occurs on | |
157 | the package search path. Thus, it keeps the connection to the | |
158 | package to which the module belongs, but may find a different file. | |
159 | ||
160 | XXX Need to have an explicit name for '', e.g. '__root__'. | |
161 | ||
162 | """ | |
163 | ||
164 | ||
165 | import imp | |
166 | import sys | |
167 | import __builtin__ | |
168 | ||
169 | import ihooks | |
170 | from ihooks import ModuleLoader, ModuleImporter | |
171 | ||
172 | ||
173 | class PackageLoader(ModuleLoader): | |
174 | ||
175 | """A subclass of ModuleLoader with package support. | |
176 | ||
177 | find_module_in_dir() will succeed if there's a subdirectory with | |
178 | the given name; load_module() will create a stub for a package and | |
179 | load its __init__ module if it exists. | |
180 | ||
181 | """ | |
182 | ||
183 | def find_module_in_dir(self, name, dir): | |
184 | if dir is not None: | |
185 | dirname = self.hooks.path_join(dir, name) | |
186 | if self.hooks.path_isdir(dirname): | |
187 | return None, dirname, ('', '', 'PACKAGE') | |
188 | return ModuleLoader.find_module_in_dir(self, name, dir) | |
189 | ||
190 | def load_module(self, name, stuff): | |
191 | file, filename, info = stuff | |
192 | suff, mode, type = info | |
193 | if type == 'PACKAGE': | |
194 | return self.load_package(name, stuff) | |
195 | if sys.modules.has_key(name): | |
196 | m = sys.modules[name] | |
197 | else: | |
198 | sys.modules[name] = m = imp.new_module(name) | |
199 | self.set_parent(m) | |
200 | if type == imp.C_EXTENSION and '.' in name: | |
201 | return self.load_dynamic(name, stuff) | |
202 | else: | |
203 | return ModuleLoader.load_module(self, name, stuff) | |
204 | ||
205 | def load_dynamic(self, name, stuff): | |
206 | file, filename, (suff, mode, type) = stuff | |
207 | # Hack around restriction in imp.load_dynamic() | |
208 | i = name.rfind('.') | |
209 | tail = name[i+1:] | |
210 | if sys.modules.has_key(tail): | |
211 | save = sys.modules[tail] | |
212 | else: | |
213 | save = None | |
214 | sys.modules[tail] = imp.new_module(name) | |
215 | try: | |
216 | m = imp.load_dynamic(tail, filename, file) | |
217 | finally: | |
218 | if save: | |
219 | sys.modules[tail] = save | |
220 | else: | |
221 | del sys.modules[tail] | |
222 | sys.modules[name] = m | |
223 | return m | |
224 | ||
225 | def load_package(self, name, stuff): | |
226 | file, filename, info = stuff | |
227 | if sys.modules.has_key(name): | |
228 | package = sys.modules[name] | |
229 | else: | |
230 | sys.modules[name] = package = imp.new_module(name) | |
231 | package.__path__ = [filename] | |
232 | self.init_package(package) | |
233 | return package | |
234 | ||
235 | def init_package(self, package): | |
236 | self.set_parent(package) | |
237 | self.set_domain(package) | |
238 | self.call_init_module(package) | |
239 | ||
240 | def set_parent(self, m): | |
241 | name = m.__name__ | |
242 | if '.' in name: | |
243 | name = name[:name.rfind('.')] | |
244 | else: | |
245 | name = '' | |
246 | m.__ = sys.modules[name] | |
247 | ||
248 | def set_domain(self, package): | |
249 | name = package.__name__ | |
250 | package.__domain__ = domain = [name] | |
251 | while '.' in name: | |
252 | name = name[:name.rfind('.')] | |
253 | domain.append(name) | |
254 | if name: | |
255 | domain.append('') | |
256 | ||
257 | def call_init_module(self, package): | |
258 | stuff = self.find_module('__init__', package.__path__) | |
259 | if stuff: | |
260 | m = self.load_module(package.__name__ + '.__init__', stuff) | |
261 | package.__init__ = m | |
262 | ||
263 | ||
264 | class PackageImporter(ModuleImporter): | |
265 | ||
266 | """Importer that understands packages and '__'.""" | |
267 | ||
268 | def __init__(self, loader = None, verbose = 0): | |
269 | ModuleImporter.__init__(self, | |
270 | loader or PackageLoader(None, verbose), verbose) | |
271 | ||
272 | def import_module(self, name, globals={}, locals={}, fromlist=[]): | |
273 | if globals.has_key('__'): | |
274 | package = globals['__'] | |
275 | else: | |
276 | # No calling context, assume in root package | |
277 | package = sys.modules[''] | |
278 | if name[:3] in ('__.', '__'): | |
279 | p = package | |
280 | name = name[3:] | |
281 | while name[:3] in ('__.', '__'): | |
282 | p = p.__ | |
283 | name = name[3:] | |
284 | if not name: | |
285 | return self.finish(package, p, '', fromlist) | |
286 | if '.' in name: | |
287 | i = name.find('.') | |
288 | name, tail = name[:i], name[i:] | |
289 | else: | |
290 | tail = '' | |
291 | mname = p.__name__ and p.__name__+'.'+name or name | |
292 | m = self.get1(mname) | |
293 | return self.finish(package, m, tail, fromlist) | |
294 | if '.' in name: | |
295 | i = name.find('.') | |
296 | name, tail = name[:i], name[i:] | |
297 | else: | |
298 | tail = '' | |
299 | for pname in package.__domain__: | |
300 | mname = pname and pname+'.'+name or name | |
301 | m = self.get0(mname) | |
302 | if m: break | |
303 | else: | |
304 | raise ImportError, "No such module %s" % name | |
305 | return self.finish(m, m, tail, fromlist) | |
306 | ||
307 | def finish(self, module, m, tail, fromlist): | |
308 | # Got ....A; now get ....A.B.C.D | |
309 | yname = m.__name__ | |
310 | if tail and sys.modules.has_key(yname + tail): # Fast path | |
311 | yname, tail = yname + tail, '' | |
312 | m = self.get1(yname) | |
313 | while tail: | |
314 | i = tail.find('.', 1) | |
315 | if i > 0: | |
316 | head, tail = tail[:i], tail[i:] | |
317 | else: | |
318 | head, tail = tail, '' | |
319 | yname = yname + head | |
320 | m = self.get1(yname) | |
321 | ||
322 | # Got ....A.B.C.D; now finalize things depending on fromlist | |
323 | if not fromlist: | |
324 | return module | |
325 | if '__' in fromlist: | |
326 | raise ImportError, "Can't import __ from anywhere" | |
327 | if not hasattr(m, '__path__'): return m | |
328 | if '*' in fromlist: | |
329 | raise ImportError, "Can't import * from a package" | |
330 | for f in fromlist: | |
331 | if hasattr(m, f): continue | |
332 | fname = yname + '.' + f | |
333 | self.get1(fname) | |
334 | return m | |
335 | ||
336 | def get1(self, name): | |
337 | m = self.get(name) | |
338 | if not m: | |
339 | raise ImportError, "No module named %s" % name | |
340 | return m | |
341 | ||
342 | def get0(self, name): | |
343 | m = self.get(name) | |
344 | if not m: | |
345 | sys.modules[name] = None | |
346 | return m | |
347 | ||
348 | def get(self, name): | |
349 | # Internal routine to get or load a module when its parent exists | |
350 | if sys.modules.has_key(name): | |
351 | return sys.modules[name] | |
352 | if '.' in name: | |
353 | i = name.rfind('.') | |
354 | head, tail = name[:i], name[i+1:] | |
355 | else: | |
356 | head, tail = '', name | |
357 | path = sys.modules[head].__path__ | |
358 | stuff = self.loader.find_module(tail, path) | |
359 | if not stuff: | |
360 | return None | |
361 | sys.modules[name] = m = self.loader.load_module(name, stuff) | |
362 | if head: | |
363 | setattr(sys.modules[head], tail, m) | |
364 | return m | |
365 | ||
366 | def reload(self, module): | |
367 | name = module.__name__ | |
368 | if '.' in name: | |
369 | i = name.rfind('.') | |
370 | head, tail = name[:i], name[i+1:] | |
371 | path = sys.modules[head].__path__ | |
372 | else: | |
373 | tail = name | |
374 | path = sys.modules[''].__path__ | |
375 | stuff = self.loader.find_module(tail, path) | |
376 | if not stuff: | |
377 | raise ImportError, "No module named %s" % name | |
378 | return self.loader.load_module(name, stuff) | |
379 | ||
380 | def unload(self, module): | |
381 | if hasattr(module, '__path__'): | |
382 | raise ImportError, "don't know how to unload packages yet" | |
383 | PackageImporter.unload(self, module) | |
384 | ||
385 | def install(self): | |
386 | if not sys.modules.has_key(''): | |
387 | sys.modules[''] = package = imp.new_module('') | |
388 | package.__path__ = None | |
389 | self.loader.init_package(package) | |
390 | for m in sys.modules.values(): | |
391 | if not m: continue | |
392 | if not hasattr(m, '__'): | |
393 | self.loader.set_parent(m) | |
394 | ModuleImporter.install(self) | |
395 | ||
396 | ||
397 | def install(v = 0): | |
398 | ihooks.install(PackageImporter(None, v)) | |
399 | ||
400 | def uninstall(): | |
401 | ihooks.uninstall() | |
402 | ||
403 | def ni(v = 0): | |
404 | install(v) | |
405 | ||
406 | def no(): | |
407 | uninstall() | |
408 | ||
409 | def test(): | |
410 | import pdb | |
411 | try: | |
412 | testproper() | |
413 | except: | |
414 | sys.last_type, sys.last_value, sys.last_traceback = sys.exc_info() | |
415 | ||
416 | print sys.last_type, ':', sys.last_value | |
417 | ||
418 | pdb.pm() | |
419 | ||
420 | def testproper(): | |
421 | install(1) | |
422 | try: | |
423 | import mactest | |
424 | print dir(mactest) | |
425 | raw_input('OK?') | |
426 | finally: | |
427 | uninstall() | |
428 | ||
429 | ||
430 | if __name__ == '__main__': | |
431 | test() | |
432 | else: | |
433 | install() |