Commit | Line | Data |
---|---|---|
86b1eead ES |
1 | .if n .ds y + |
2 | .if n .ds z ++ | |
3 | .if t .ds y \(dg | |
4 | .if t .ds z \(dd | |
5 | .ds a ``An Introduction to the Berkeley Network'' | |
6 | .ds b ``Network System Manual'' | |
7 | .TL | |
8 | The Berkeley Network \- | |
9 | A Retrospective | |
10 | .AU | |
11 | Eric Schmidt | |
12 | .AI | |
13 | Computer Science Division | |
14 | Department of Electrical Engineering and Computer Science | |
15 | University of California, Berkeley | |
16 | Berkeley, California 94720 | |
17 | .AB | |
18 | The Berkeley Network connects a number of | |
19 | .UX | |
20 | machines on the Berkeley campus. | |
21 | It provides facilities for file transfer, sending and reading mail, | |
22 | and remote printing. | |
23 | Operating in a batch mode, network requests are transferred one by one | |
24 | through an inter-connected network until they reach their final destination. | |
25 | .PP | |
26 | This document describes the history and goals of this project, | |
27 | the design decisions faced, discusses issues in portable software | |
28 | development in networks, and discusses the future of this project. | |
29 | .AE | |
30 | .SH | |
31 | .ce | |
32 | Introduction | |
33 | .sp .5 | |
34 | .PP | |
35 | This document is intended for readers with an interest in networking | |
36 | who are familiar with two other documents about the network, | |
37 | \*a, and the \*b, by this author. | |
38 | It is not necessary to read this document to set up and maintain the network, | |
39 | although systems persons will benefit if they are familiar with the | |
40 | concepts presented here. | |
41 | .PP | |
42 | The sections are presented as follows: | |
43 | .DS | |
44 | Principals | |
45 | History | |
46 | Overall System Description | |
47 | Protocol Explanation | |
48 | Portability | |
49 | Security | |
50 | Future Plans | |
51 | Summary | |
52 | .DE | |
53 | .PP | |
54 | The most important section is the last, | |
55 | which details a set of principles the author | |
56 | has learned during this project. | |
57 | .SH | |
58 | .ce | |
59 | .sp 1 | |
60 | Principals | |
61 | .sp .5 | |
62 | .PP | |
63 | This project was a collaboration of many individuals. | |
64 | Dr. Robert S. Fabry participated in the initial design | |
65 | and has exerted the strongest influence on polishing the final product. | |
66 | Bill Joy and Ed Gould provided valuable technical expertise every step of | |
67 | the way, primarily about developing systems programs. | |
68 | The support staff of the EECS Department and the Computer Center | |
69 | (Bob Kridle, Jeff Schreibman, Vance Vaughan, Robyn Allsman, and Ricki Blau) | |
70 | were involved in setting up and administering this multi-domain project. | |
71 | The lowest-level concepts the author used came from experience | |
72 | at \s-2XEROX PARC\s0, primarily from discussions | |
73 | with David Boggs, one of the \s-2ETHERNET\s0\*y designers. | |
74 | .FS | |
75 | \*y ``Ethernet: Distributed Packet Switching for Local Computer Networks'', by Robert Metcalfe and David Boggs, CACM, July 1976. | |
76 | .FE | |
77 | .SH | |
78 | .ce | |
79 | .sp 1 | |
80 | History | |
81 | .sp .5 | |
82 | .PP | |
83 | The network project can be divided into two distinct phases. | |
84 | The first, from January 1978 to May 1978 (4\(12 months) | |
85 | involved designing and implementing the network facilities | |
86 | now available. | |
87 | The second, from October 1978 to March 1979 (5 months), saw the addition | |
88 | of many more machines to the network with emphasis on | |
89 | portability, security, and minor design changes. | |
90 | The network has been in almost continuous service to users since May 1, 1978. | |
91 | .SH | |
92 | First Phase | |
93 | .PP | |
94 | An initial design was worked out with Dr. Fabry in January 1978. | |
95 | A suitable connection was made between the Computer Center ``A'' and ``Q'' | |
96 | machines (then called ``D''). | |
97 | .DS | |
98 | .cs R 24 | |
99 | ||
100 | ||
101 | A Q | |
102 | ||
103 | ||
104 | .cs R | |
105 | .DE | |
106 | .PP | |
107 | Development proceeded on two fronts \- | |
108 | a set of daemons were written to transfer files, using | |
109 | .UX | |
110 | pipes for debugging. | |
111 | The lowest-level protocols were designed and implemented of the terminal-type | |
112 | connection between A and Q. | |
113 | These were debugged using simple programs to send and receive packets, | |
114 | and a pair | |
115 | of programs to transfer a file from one to the other. | |
116 | This was the first experience with a distributed software development \- | |
117 | the Q machine is a DEC PDP-11/34 with no lineprinter, a non-standard | |
118 | tape drive, and terminal access only by telephone, so | |
119 | most development had to proceed on the A machine. | |
120 | .PP | |
121 | During this phase the goals of the project increased in scope. | |
122 | The implementor and only user had to use the network to transfer | |
123 | network source and worked out simple ways to automate this | |
124 | (the ``.netrc'' file is an example). | |
125 | .PP | |
126 | When it appeared usable by more than the implementor, | |
127 | the connection was changed to be between A and the Computer Center ``C'' | |
128 | machine: | |
129 | .DS | |
130 | .cs R 24 | |
131 | ||
132 | ||
133 | A C | |
134 | ||
135 | ||
136 | .cs R | |
137 | .DE | |
138 | .PP | |
139 | Up until now, the network had required an account on both machines. | |
140 | It became clear this was a handicap since | |
141 | the A machine had too many accounts and the password file was immense. | |
142 | Certain ``free'' commands were allowed, without an account. | |
143 | .PP | |
144 | The Cory machine was soon added to the network: | |
145 | .DS | |
146 | .cs R 24 | |
147 | ||
148 | ||
149 | Cory A C | |
150 | ||
151 | ||
152 | .cs R | |
153 | .DE | |
154 | .PP | |
155 | This produced major changes in the design \- | |
156 | initially we had assumed network users would have accounts on all machines. | |
157 | This was unreasonably strict | |
158 | and a solution (kludge) was worked out where a request was | |
159 | examined and forwarded through the queue(s) on the intermediate machine. | |
160 | The network became table-driven | |
161 | to accomplish the routing, and distributed software | |
162 | development became more difficult because of the increased number of machines. | |
163 | The implementor quickly discovered only one solution: | |
164 | Always have a designated source machine for all changes. | |
165 | To this day, software changes are made only on this ``source'' machine, | |
166 | the others are guaranteed to have copies. | |
167 | This makes remote updating (copying new versions around the network) | |
168 | possible. | |
169 | .PP | |
170 | Documentation was written and sent off to about fifteen | |
171 | faculty, staff, and graduate student users. | |
172 | A few bugs were fixed and the system frozen from the end | |
173 | of May to October. | |
174 | End of Phase 1. | |
175 | .SH | |
176 | Second Phase | |
177 | .PP | |
178 | While the implementor was away at a summer job, the connection between | |
179 | the A and C machines was switched to reduce the loading on the A machine: | |
180 | .DS | |
181 | .cs R 24 | |
182 | ||
183 | ||
184 | Cory C A | |
185 | ||
186 | ||
187 | .cs R | |
188 | .DE | |
189 | .PP | |
190 | Unknown to the implementor, the network source was modified in | |
191 | divergent and incompatible ways, and the commands were moved to a different | |
192 | place. | |
193 | These changes invalidated certain assumptions about full pathnames and | |
194 | some commands such as inter-machine mail stopped working. | |
195 | .PP | |
196 | The Computer Center had also made absolutely incompatible changes | |
197 | to some system calls. | |
198 | This began a path of software divergence that became very painful | |
199 | and is still not completely solved. | |
200 | .PP | |
201 | Fortunately, the Computer Center placed Robyn Allsman in charge of maintaining | |
202 | the network on their machines \- | |
203 | to lighten that routine part of the load from the implementor. | |
204 | .PP | |
205 | The Computer Center acquired a ``D'' machine, and the EECS Division | |
206 | a DEC VAX 11/780, running an experimental Version 7 | |
207 | .UX | |
208 | system. | |
209 | The implementor decided to use the (at this point) unused VAX | |
210 | to to do software development and incorporate the Version 7 | |
211 | changes into the network code. | |
212 | By this time, the protocols were stable which made it possible | |
213 | to run a Version 7 network | |
214 | on the VAX connected to the old Version 6 code | |
215 | on the existing network, to facilitate debugging. | |
216 | Because of improved terminal availability and better machine | |
217 | response, many new ways were used to debug the network \- | |
218 | using pipes, using TTY lines wired together on the VAX and over the | |
219 | usual machine-to-machine link. | |
220 | A file was added (``initfile'') which allowed quick reconfiguration | |
221 | of the daemons when system parameters were changed. | |
222 | A temporary connection between the VAX and C machines was arranged. | |
223 | .DS | |
224 | .cs R 24 | |
225 | ||
226 | ||
227 | VAX | |
228 | ||
229 | ||
230 | ||
231 | ||
232 | ||
233 | ||
234 | ||
235 | ||
236 | Cory C A | |
237 | ||
238 | ||
239 | .cs R | |
240 | .DE | |
241 | .PP | |
242 | The network code had to be able to run on three different types of | |
243 | machines \- | |
244 | the VAX running | |
245 | .UX | |
246 | Version 7, the Cory machine running Version 6, and the anomalous | |
247 | Computer Center machines. | |
248 | There was no conversion package available at the time,\*y | |
249 | .FS | |
250 | \*y The ``retrofit'' library, by Bill Joy, is now available. | |
251 | .FE | |
252 | and the old network code had not used any system header | |
253 | files, so after a great deal of experimentation, conditional | |
254 | compilation was used as much as possible and a procedural interface | |
255 | was used to elide system differences. | |
256 | .PP | |
257 | The new | |
258 | .UX | |
259 | command | |
260 | .I make | |
261 | (I) was used with a ``makefile'' to organize this regime. | |
262 | The old network code was used to bootstrap | |
263 | the D machine onto a network running the new network code exclusively. | |
264 | .DS | |
265 | .cs R 24 | |
266 | ||
267 | ||
268 | VAX | |
269 | ||
270 | ||
271 | ||
272 | ||
273 | ||
274 | ||
275 | ||
276 | ||
277 | Cory C A | |
278 | ||
279 | ||
280 | ||
281 | ||
282 | ||
283 | ||
284 | ||
285 | ||
286 | D | |
287 | ||
288 | ||
289 | .cs R | |
290 | .DE | |
291 | .PP | |
292 | Shortly thereafter, over the Christmas break, the VAX and Cory connections | |
293 | went down for security reasons (discussed in the ``Security'' section). | |
294 | After seven weeks, they reentered the network in a new configuration. | |
295 | .DS | |
296 | .cs R 24 | |
297 | ||
298 | ||
299 | Cory C D | |
300 | ||
301 | ||
302 | ||
303 | ||
304 | ||
305 | ||
306 | ||
307 | ||
308 | VAX A | |
309 | ||
310 | ||
311 | ||
312 | .cs R | |
313 | .tr -- | |
314 | .DE | |
315 | .PP | |
316 | Shortly after that, they were down again because of a lightning strike | |
317 | for another week but have been operational since then. | |
318 | During the last time period the network was made less Berkeley-specific | |
319 | and a copy was run on the Rand Corporation UNIX machines. | |
320 | Documentation was rewritten and prepared for release. | |
321 | The network queues were converted to send shortest-job first. | |
322 | Extensive monitoring of system load, network performance, | |
323 | and network use was added. | |
324 | The format of the | |
325 | .I netq | |
326 | command was changed to summarize more information on output. | |
327 | The E machine, and then later the Survey Research Center (SRC) | |
328 | machine, were added. | |
329 | .DS | |
330 | .cs R 24 | |
331 | ||
332 | ||
333 | A | |
334 | ||
335 | ||
336 | ||
337 | ||
338 | ||
339 | ||
340 | ||
341 | ||
342 | Cory C D | |
343 | ||
344 | ||
345 | ||
346 | ||
347 | ||
348 | ||
349 | ||
350 | ||
351 | VAX E SRC | |
352 | ||
353 | ||
354 | .cs R | |
355 | .DE | |
356 | .PP | |
357 | Tuning was still important \- | |
358 | serious overloading problems caused by sluggish response | |
359 | stopped the network between Cory and C for a week. | |
360 | Network parameters were tuned to help solve this problem. | |
361 | The complexity of software development and maintenance became too | |
362 | great for unstructured changes. | |
363 | Versions on all the machines except the VAX were frozen | |
364 | for a month at a time. | |
365 | The protocols were almost immutable. | |
366 | People were delegated responsibility to observe and straighten | |
367 | out, if necessary, problems with the net queues. | |
368 | .PP | |
369 | As this is written, the software is stable and the | |
370 | user-documentation is finished and being sold, | |
371 | and there is hope of adding more | |
372 | .UX | |
373 | machines to the network. | |
374 | .SH | |
375 | .ce | |
376 | .sp 1 | |
377 | Overall System Description | |
378 | .sp .5 | |
379 | .PP | |
380 | The Berkeley network operates in a batch/ request mode, and is similar | |
381 | in concept to a line printer queue. | |
382 | ``Requests'' are queued up at the source, where they are sent in shortest | |
383 | job first order through an interconnected network of machines to | |
384 | their destination. | |
385 | At each intermediate node, they are queued as if they were originated locally. | |
386 | .PP | |
387 | The network consists of a set of user-executed commands, | |
388 | a queue of requests to be sent, and a continuously-running | |
389 | program called a | |
390 | .I daemon | |
391 | which transmits requests in the queue and listens for any request being sent | |
392 | to it. | |
393 | There are many network commands \- | |
394 | one to send mail, one to read mail, one to copy files, etc. | |
395 | They all use the | |
396 | .I net | |
397 | command to access the network. | |
398 | The | |
399 | .I net | |
400 | command takes a command, assorted parameters, with any input | |
401 | data, and puts a request in the queue. | |
402 | These requests are composed of a header, the command to be executed, | |
403 | and any data for input to the command. | |
404 | The header contains network information such | |
405 | as the destination machine, login name, and password. | |
406 | This request is stored in the queue as a normal ASCII | |
407 | file, owned by the invoker. | |
408 | This way | |
409 | .UX | |
410 | commands can be used to examine the file. | |
411 | .PP | |
412 | The daemon examines the queue to see if there is anything to send. | |
413 | If so, it begins sending to a remote daemon, | |
414 | using a protocol to establish this dialog, | |
415 | involving retransmissions and timeouts. | |
416 | The remote daemon accepts the requests, | |
417 | parses the header information, and takes any | |
418 | data for the command and puts it in a file. | |
419 | The main loop of the daemon then returns to a waiting state. | |
420 | .PP | |
421 | The command execution is done by `forking' a series of processes. | |
422 | One of these is the user's login shell, | |
423 | which is given the command to execute. | |
424 | Another is a process which waits for the command | |
425 | to be executed, then examines the output of the command. | |
426 | The output is typically sent back to the user, via a | |
427 | .I net | |
428 | command, executed by the daemon. | |
429 | .PP | |
430 | In the reverse transmission, the command is called ``mwrite'', | |
431 | and it is routed and handled exactly as in the forward mode, | |
432 | except no password is required. | |
433 | The ``mwrite'' command is executed on the original machine | |
434 | with input data which is a copy of the output of | |
435 | the remote command. | |
436 | The user is either ``written'' or ``mailed'' to, | |
437 | depending on certain options. | |
438 | If ``written'', the user's screen is filled with the output\*y | |
439 | .FS | |
440 | \*y Standard output and standard error files. | |
441 | .FE | |
442 | of the command executed remotely, just as if he had executed it locally. | |
443 | Otherwise, it is in his mailbox, as mail from the remote account he used. | |
444 | The user's terminal must be write-able (see the | |
445 | .I mesg | |
446 | (I) option), the originating user must still be logged in, | |
447 | and he must not have logged off and on again. | |
448 | .PP | |
449 | The output from the command is preceded by a line of information | |
450 | listing the command, the time it was sent, and the time elapsed. | |
451 | .PP | |
452 | Our design then tries to simulate local | |
453 | .UX | |
454 | commands as much as possible. | |
455 | With defaults set correctly\*y | |
456 | .FS | |
457 | \*y With a ``.netrc'' file, see below. | |
458 | .FE | |
459 | the user in principle need only precede the command he is executing | |
460 | with the command | |
461 | .I net. | |
462 | .SH | |
463 | Copying Remote Files | |
464 | .PP | |
465 | The most frequent use of the network is file-transfer using the | |
466 | .I netcp | |
467 | command. | |
468 | The | |
469 | .I netcp | |
470 | command is based on the | |
471 | .I cp | |
472 | (I) command. | |
473 | Its two arguments are a source and destination file, optionally with | |
474 | remote machine names prepended: | |
475 | .DS | |
476 | \fBnetcp\fI\ \ from\-file\ \ to\-file\fR | |
477 | .DE | |
478 | where the names are local or remote. | |
479 | Since | |
480 | .DS | |
481 | Cory:/usr/pascal/sh | |
482 | .DE | |
483 | is a file on the Cory machine, | |
484 | .DS | |
485 | % netcp\ \ junk\ \ Cory:/usr/pascal/sh | |
486 | .DE | |
487 | will transfer the file ``junk'' to the named file on the Cory machine\*z. | |
488 | .FS | |
489 | \*z For more examples, see the \*a document. | |
490 | .FE | |
491 | .PP | |
492 | The way the transfer is accomplished depends on the type of file copy: | |
493 | .IP 1. | |
494 | Copy local file to remote file \- | |
495 | .br | |
496 | On the remote machine a | |
497 | .I cat | |
498 | (I) command is executed on the remote file with the local | |
499 | file as standard input. | |
500 | .IP 2. | |
501 | Copy remote file to local file \- | |
502 | .br | |
503 | A | |
504 | .I cat | |
505 | command is executed on the remote machine from the remote file | |
506 | to standard output. | |
507 | This standard output is sent back to the local machine | |
508 | into a | |
509 | .I | |
510 | response file, | |
511 | .R | |
512 | instead of being written or mailed to the user. | |
513 | .IP 3. | |
514 | Copy remote file to another remote file \- | |
515 | .br | |
516 | If both are on the same machine, a | |
517 | .I cp | |
518 | command is sent. | |
519 | Otherwise, a | |
520 | .I netcp | |
521 | command is sent to the remote machine where the | |
522 | .I from\-file | |
523 | exists, | |
524 | to copy that file to the \fIto\-file\fP's machine. | |
525 | .PP | |
526 | This last case is experimental. | |
527 | Unfortunately, the system is structured only to carry one login | |
528 | name and password to a remote machine. | |
529 | Since the last option involves three machines, | |
530 | the second remote machine is handled imperfectly at best. | |
531 | .SH | |
532 | Sending Mail | |
533 | .PP | |
534 | The | |
535 | .I mail | |
536 | (I) command on the network machines has been modified to examine the names of | |
537 | the recipients of a particular message. | |
538 | If their names have a remote prefix, | |
539 | .I mail | |
540 | executes an internal command ``sendmail'', | |
541 | which in turn executes a | |
542 | .I net | |
543 | command. | |
544 | This net command sends a mail command to the remote machine, | |
545 | with the message as input. | |
546 | Since the recipient would like to know which machine | |
547 | the message came from, a simple program ``mmail'' | |
548 | is executed to insert a pseudo-header indicating the real | |
549 | source of the mail. | |
550 | The net command logs in as user ``network'', | |
551 | so remote mail does not require an account on the destination machine. | |
552 | This facility has proven invaluable. | |
553 | .SH | |
554 | Reading Mail | |
555 | .PP | |
556 | The | |
557 | .I netmail | |
558 | command uses the | |
559 | .I net | |
560 | command to send a command to read mail for a specified user | |
561 | on a remote machine. | |
562 | Since the existing mail programs on different machines vary in their | |
563 | options, it was decided the only thing that would work on both | |
564 | .UX | |
565 | Version 6 and 7 systems was to copy the mail from the remote | |
566 | to the local machine. | |
567 | If the user subsequently logs in on the remote machine, | |
568 | his mail will be there, as if it were unread. | |
569 | An internal program ``prmail'' is used to copy the user's mail | |
570 | back to the local machine. | |
571 | It knows the location of the mailboxes and the user's name. | |
572 | .PP | |
573 | The mail programs at Berkeley are being modified to search a database | |
574 | to see whether a user would like to receive all his mail on | |
575 | another machine and automatically forward it. | |
576 | This will diminish the need for the | |
577 | .I netmail | |
578 | command. | |
579 | .SH | |
580 | Printing on remote lineprinters | |
581 | .PP | |
582 | The | |
583 | .I netlpr | |
584 | command takes a list of arguments as files to be printed on a remote | |
585 | lineprinter. | |
586 | Unfortunately, there can only be one standard input file for the remote | |
587 | command, so each file is sent as a | |
588 | .I net | |
589 | command executing the command | |
590 | .I lpr. | |
591 | .SH | |
592 | Other System Components | |
593 | .sp .5 | |
594 | .LP | |
595 | The ``.netrc'' file. | |
596 | .PP | |
597 | A user must specify defaults for frequently repeated options | |
598 | on a per-machine basis. | |
599 | This is done in a file ``.netrc'' in the user's login directory, | |
600 | and is fully described in the \*a. | |
601 | .sp .5 | |
602 | .LP | |
603 | The | |
604 | .I netq | |
605 | command. | |
606 | .PP | |
607 | To see the network queue, the user must type the | |
608 | .I netq | |
609 | command. | |
610 | It lists the queue for each directly-connected machine, in the order | |
611 | requests will be sent. | |
612 | Each request is listed, one per line, giving | |
613 | the local and remote machines, the destination machine, the time sent, | |
614 | and the command to be executed. | |
615 | Commands which are internal to the network are called ``responses'' | |
616 | in the | |
617 | .I netq | |
618 | listing. | |
619 | .sp .5 | |
620 | .LP | |
621 | The | |
622 | .I netrm | |
623 | command. | |
624 | .PP | |
625 | Requests may be removed from the send queue using the | |
626 | .I netrm | |
627 | command. | |
628 | Since the originator of the file owns the queue file, a simple user-id | |
629 | check suffices to validate permission. | |
630 | Unfortunately, this notion breaks down for queue files | |
631 | of requests on intermediate machines. | |
632 | On such a machine an appropriate user does not exist, | |
633 | and the files are owned by ``root''. | |
634 | There is an option to | |
635 | .I netrm | |
636 | to remove all the user's queue files, to make | |
637 | .I netrm | |
638 | easier to use. | |
639 | .sp .5 | |
640 | .LP | |
641 | The | |
642 | .I netlog | |
643 | command and other information. | |
644 | .PP | |
645 | A number of log files are kept by the network. | |
646 | Users may execute a | |
647 | .I netlog | |
648 | command which prints the last few entries of a log of commands | |
649 | sent and received. | |
650 | Also listed is the user, the time of the entry, | |
651 | and the return code of the command. | |
652 | .PP | |
653 | A more unreadable logfile is `/usr/net/plogfile?'. | |
654 | This log file has all the information of | |
655 | .I netlog, | |
656 | in a more cryptic form, along with trace information | |
657 | about net errors. | |
658 | The beginning and ending of sending and receiving are noted. | |
659 | This way the exact state of the network can be determined. | |
660 | .PP | |
661 | Hourly and daily statistics in a file `/usr/net/netstat?'. | |
662 | The number of bytes transmitted, the number of commands, and a | |
663 | breakdown of their type, and system load is recorded. | |
664 | This information is recorded in both hourly and daily form | |
665 | to track the performance of the network under different system loads. | |
666 | .PP | |
667 | Every hour, a | |
668 | .I netq | |
669 | command is executed and the number of queue entries is recorded in a | |
670 | file `/usr/net/netqstats'. | |
671 | This gives an estimate of the queue length. | |
672 | .PP | |
673 | Finally, the login names of each local user are recorded | |
674 | in a file `/usr/net/usernames'. | |
675 | Periodically, these names are sorted and duplicates removed. | |
676 | This gives a complete listing of network users, | |
677 | useful for sending network-specific mail and for general interest. | |
678 | .SH | |
679 | .sp 1 | |
680 | .ce | |
681 | Protocol Explanation | |
682 | .sp .5 | |
683 | .PP | |
684 | The network uses three distinct levels of protocol. | |
685 | The highest level of protocol (the ``command'' protocol) | |
686 | refers to the organization of the information sent to the remote | |
687 | machine. | |
688 | An intermediate level splits such a stream into distinct numbered packets | |
689 | with a small header in each packet. | |
690 | The lowest level refers to the appearance of these packets on the hardware | |
691 | connection. | |
692 | At the present, this is a modified 6-bit ASCII code. | |
693 | Each of these layers is distinct, and presents the | |
694 | interface through procedure calls. | |
695 | .SH | |
696 | The Command Protocol | |
697 | .PP | |
698 | Each machine sends a request using a precise command protocol | |
699 | involving a header, the command, and any associated data. | |
700 | .TS | |
701 | center allbox; | |
702 | l l l l. | |
703 | length header command ... data ... | |
704 | .TE | |
705 | All but the length field is formatted by the | |
706 | .I net | |
707 | command before the file is queued for transmission. | |
708 | The length is used to detect abnormally short, and | |
709 | poorly-formed, requests. | |
710 | The header includes all the information to route and verify the | |
711 | request. | |
712 | It includes | |
713 | .RS | |
714 | .IP a) | |
715 | the origin and destination machines, | |
716 | .IP b) | |
717 | the login names on both machines, | |
718 | .IP c) | |
719 | a version stamp for this command protocol, | |
720 | .IP d) | |
721 | the time sent, | |
722 | .IP e) | |
723 | information about the originating terminal, and | |
724 | .IP f) | |
725 | the pseudo-command. | |
726 | .RE | |
727 | .PP | |
728 | The pseudo-command is read by | |
729 | .I netq, | |
730 | and instead of printing the actual command being | |
731 | executed, prints something more appropriate. | |
732 | All the commands which use | |
733 | .I net | |
734 | (e.g. | |
735 | .I netcp) | |
736 | use the pseudo-command to list themselves rather | |
737 | than the command they are executing on the remote machine. | |
738 | .PP | |
739 | In order to be able to print the data on a normal | |
740 | .UX | |
741 | terminal for debugging, the fields within the header are variable-length ASCII, | |
742 | separated by colons (`:'). | |
743 | This forces the daemon to parse the header information and | |
744 | requires that literal colons (e.g. in the command being sent) | |
745 | be properly escaped within the protocol, | |
746 | but I felt the alternatives of using byte counts or fixed-length fields | |
747 | were too difficult to debug. | |
748 | The shortest header is approximately 85 bytes. | |
749 | Fortunately, this means the shortest command | |
750 | will fit into a single packet.\*y | |
751 | .FS | |
752 | \*y (less than 100 bytes, see below). | |
753 | .FE | |
754 | .SH | |
755 | The Packet Protocol | |
756 | .PP | |
757 | The above information is conveyed to each machine as a stream. | |
758 | This is done using subroutine calls to read and write data | |
759 | of arbitrary length over the link. | |
760 | The write procedure breaks the information into a set | |
761 | of numbered packets, | |
762 | with a length and exclusive-or check-sum in a header: | |
763 | .TS | |
764 | center allbox; | |
765 | l l l l l. | |
766 | length seqnum type chksum ... data ... | |
767 | .TE | |
768 | .PP | |
769 | The length, type and checksum are one byte each, | |
770 | and the sequence number is two bytes. | |
771 | Since the packets are variable length the checksum is in the | |
772 | header rather than at the end of the packet to avoid the extra computation | |
773 | required to access it. | |
774 | .PP | |
775 | Each packet is transmitted over a link to a listener. | |
776 | Normally an acknowledgement packet is sent back. | |
777 | If there is an error, nothing is sent back, | |
778 | and the sending end will retransmit after a certain number of seconds. | |
779 | .PP | |
780 | There are no windowing or piggyback acknowledgements for two reasons: | |
781 | 1) this scheme is very simple to implement and | |
782 | 2) the error rate if each packet were not acknowledged would be | |
783 | high because of the hardware involved. | |
784 | If the future, I hope that both hardware and kernel device drivers | |
785 | will allow this improvement. | |
786 | .PP | |
787 | The so-called ``rendezvous'' protocol, whereby two daemons agree to communicate, | |
788 | is a simple ``contention'' scheme. | |
789 | When one daemon wants to transmit, | |
790 | it sends a special packet ``reset'' to the possible receiver, | |
791 | then transmits his first packet. | |
792 | Normally a daemon able to receive listens for a ``reset''. | |
793 | If it receives one, it enters a section of code designed | |
794 | to receive a header command, and data, and ultimately will execute it. | |
795 | If not, after a prescribed time interval is checks to see if there are any | |
796 | requests to send. | |
797 | Should both send at once, the characters may be garbled, | |
798 | or both may receive resets at the same time. | |
799 | In each case they both will retransmit. | |
800 | Each has a randomizing factor to break any ties which might develop. | |
801 | .PP | |
802 | In retrospect, this protocol is very primitive. | |
803 | Now that the network is in production use, | |
804 | the extra acknowledgements and separate ``reset'' are too | |
805 | expensive. | |
806 | A redesign would modify the protocol to transmit | |
807 | more than one packet before acknowledging it (ACKs), | |
808 | use negative ACKs to indicate immediately that an error has occurred, | |
809 | and eliminate the separate ``reset'' entirely. | |
810 | .PP | |
811 | The ``rendezvous'' protocol consumes a fair amount of time | |
812 | when both daemons choose to send packets. | |
813 | The alternative of constantly sending status packets when | |
814 | the daemons would be idle was never seriously considered | |
815 | because it was felt that the daemon should have as light a system | |
816 | load as possible; it seems now the daemons are busy most of the time | |
817 | and thus the initial connection tradeoffs | |
818 | should have been studied more closely. | |
819 | .SH | |
820 | The Low-Level Protocol | |
821 | .PP | |
822 | The network transmits over TTY lines through terminal interfaces | |
823 | and system drivers which behave as if the characters coming from another | |
824 | machine are from a terminal. | |
825 | This mode was chosen because it is absolutely the simplest, | |
826 | cheapest interconnection scheme possible. | |
827 | Unfortunately, the | |
828 | .UX | |
829 | terminal drivers cannot accept 8-bit bytes unless they are in | |
830 | .I raw | |
831 | mode. | |
832 | This was judged to be a high system load, so the | |
833 | TTY lines operate in | |
834 | .I cooked, | |
835 | the reverse of | |
836 | .I raw, | |
837 | mode. | |
838 | In this mode certain bit combinations, | |
839 | e.g. ASCII newline and escape, do not | |
840 | transmit through the terminal driver to the user's program | |
841 | but rather are interpreted as control information. | |
842 | .PP | |
843 | After much experimentation, the following transmission method was chosen. | |
844 | Each 3 byte triple is viewed as 24 bits, and broken into 4 6-bit groups. | |
845 | Each 6-bit number is in the range 0-63, and is added to a constant | |
846 | representing the lowest acceptable character code (a blank) | |
847 | in the ASCII sequence. | |
848 | This is sent as an ASCII character to a receiver | |
849 | who gathers 4 bytes, subtracts the increment, and shifts the 4 6-bit groups | |
850 | into 3 bytes. | |
851 | This represents a 3 to 4 expansion of all characters over the | |
852 | link, or a 33% loss of bandwidth. | |
853 | .PP | |
854 | In retrospect, this expansion has a considerable cost. | |
855 | The most scarce resource in the network | |
856 | is the actual hardware speed of the links. | |
857 | The alternative of using raw mode was never seriously considered. | |
858 | .PP | |
859 | The implementor's hope is that better hardware will make better | |
860 | middle- and lower-level protocols easier. | |
861 | Until then, the difficulties of using TTY lines efficiently | |
862 | make further protocol improvements unlikely to yield big increases in speed. | |
863 | .SH | |
864 | A Note About Features this Protocol Lacks | |
865 | .PP | |
866 | In | |
867 | .UX | |
868 | a process may only read or write one I/O device at a time. | |
869 | A daemon approach requires a single process reading and writing on a link to | |
870 | another machine. | |
871 | This process must decide who will receive this packet. | |
872 | I judged (correctly) that this decision was hard to schedule using | |
873 | .UX | |
874 | pipes and signals, | |
875 | and only allow one kind of communication between daemons. | |
876 | This also makes it almost impossible | |
877 | to forward packets through intermediate machines. | |
878 | Thus intermediate machines copy whole requests before sending them again. | |
879 | .PP | |
880 | If the design specification required a simple packet-oriented driver | |
881 | within the system, the | |
882 | .UX | |
883 | kernel could decide which of several special files this was destined to, | |
884 | and allow much more intermixing of traffic than before. | |
885 | I did not realize the importance of this and, in retrospect, | |
886 | would have chosen the other of the two approaches. | |
887 | .SH | |
888 | .sp 1 | |
889 | .ce | |
890 | Portability | |
891 | .sp .5 | |
892 | .PP | |
893 | The acquisition of VAX/UNIX (Version 7) and the divergence of the Computer | |
894 | Center and Cory Hall Version 6 systems made the portability of the | |
895 | network source code important. | |
896 | Until then, the source code on all machines was identical. | |
897 | Fortunately, the | |
898 | .UX | |
899 | implementors encountered these same problems and developed a number of | |
900 | facilities the network now uses. | |
901 | .PP | |
902 | Since many system calls use machine and version dependent data fields, | |
903 | so-called ``include'' files are available to hide the system differences | |
904 | and help standardize the system interface. | |
905 | The conditional compilation feature of the C language was used to select | |
906 | which kinds of code to generate, when the ``include'' files were | |
907 | not sufficient. | |
908 | Roughly, the following command included at the beginning of each C module: | |
909 | .DS | |
910 | # include <whoami.h> | |
911 | .DE | |
912 | would define which system, by name, the code was run on. | |
913 | For example, the above defines ``VAX'' on the VAX machine, | |
914 | and then lines such as | |
915 | .DS | |
916 | # ifdef VAX | |
917 | . | |
918 | . | |
919 | # endif | |
920 | .DE | |
921 | control the code generated. | |
922 | In the network, sequences such as this in turn define other sequences, | |
923 | such as | |
924 | .DS | |
925 | # ifdef \s-2CORY\s0 | |
926 | # define \s-2FUID\s0 | |
927 | # define \s-2OLDTTY\s0 | |
928 | # define \s-2PASSWDF\s0 | |
929 | # endif | |
930 | .DE | |
931 | defines the unusual features of the Cory machine: | |
932 | the combined user-id and group-id returned by the | |
933 | .I getuid() | |
934 | system call, that it uses the old 1-character terminal names, and | |
935 | that it has a split password file for security reasons. | |
936 | Each of these symbols, e.g. ``FUID'', is tested | |
937 | in order to compile the correct code for that feature. | |
938 | .PP | |
939 | To help in isolating the changes, attempts were made to create a procedural | |
940 | interface to hide machines differences. | |
941 | These procedures are all in one file. | |
942 | Only one or two cases exist of conditional sections of code | |
943 | not in ``mach.c'' or ``mach.h'', its header file. | |
944 | .PP | |
945 | One problem these features pose is testing changes \- | |
946 | the conditional sections hide errors in inapplicable code. | |
947 | A regimen was adopted: | |
948 | Testing was first done on the VAX (Version 7), | |
949 | then, after it was stable for a few days, | |
950 | moved to Cory, where typically there was some Version 6-dependent | |
951 | error, and after that was fixed and stable, it was moved to the | |
952 | Computer Center machines. | |
953 | This notion of a ``testing'' machine is very important \- | |
954 | the VAX always has an up-to-date copy of the network | |
955 | source, even though other machines may lag in improvements. | |
956 | .PP | |
957 | There is now a ``retrofit library'' | |
958 | that simulates many of the features ``mach.c'' provides. | |
959 | It was not stable enough when the network was converted to Version 7, | |
960 | otherwise I would have used it. | |
961 | .PP | |
962 | At this point, when the entire source for the software | |
963 | for the network is transferred between machines only the first five | |
964 | lines of the ``makefile'' need be changed. | |
965 | .SH | |
966 | .sp 1 | |
967 | .ce | |
968 | .sp .5 | |
969 | Security | |
970 | .PP | |
971 | Over Christmas vacation of 1978, a 15-year old high school student | |
972 | repeatedly broke into the Computer Center and Cory machines. | |
973 | He was able to use the network to gain access to privileged files on | |
974 | the VAX, and the fear of protection ``holes'' caused the staff to take | |
975 | the network down for seven weeks. | |
976 | .PP | |
977 | There were two security problems posed by the network: | |
978 | .RS | |
979 | .IP 1. | |
980 | The threat to the ``root'' account on another system. | |
981 | .IP 2. | |
982 | The threat to a user's remote accounts. | |
983 | .RE | |
984 | .NH 0 | |
985 | Threat to ``root'' | |
986 | .PP | |
987 | Originally the network would allow a user logged in as ``root'' | |
988 | on one machine to execute any command as ``root'' on another network | |
989 | machine. | |
990 | As far as we know, this feature was never used to break into a system. | |
991 | However, the network has been changed to prevent a user from logging | |
992 | in as ``root'' on another machine, regardless of the password. | |
993 | This check is performed on the sending machine. | |
994 | Since ``root'' could conceivably circumvent this check by altering | |
995 | the command, the receiving end of a command checks the user-id | |
996 | of all commands being executed. | |
997 | If it is zero (i.e. ``root'') only a set of five commands is allowed, | |
998 | all needed by the network internally, and believed ``safe''. | |
999 | .PP | |
1000 | We believe this makes the network ``safer'' than many local machine | |
1001 | features such as the use of dial-up lines. | |
1002 | .NH | |
1003 | Threat to user's remote accounts. | |
1004 | .PP | |
1005 | If a user places remote passwords in his ``.netrc'' file, | |
1006 | an illicit superuser could get the password to all the user's remote | |
1007 | accounts. | |
1008 | Even if the user does not care, system managers dislike this because | |
1009 | the illicit superuser could now use a legal account on another system | |
1010 | to break into it. | |
1011 | .PP | |
1012 | We have no good solutions to this. | |
1013 | Users are now warned of this danger in the documentation, | |
1014 | and a ``.netrc'' file with passwords must be | |
1015 | readable only by the owner of the file. | |
1016 | .PP | |
1017 | Various solutions have been proposed: | |
1018 | .RS | |
1019 | .IP a) | |
1020 | Disallow passwords in ``.netrc'' files. | |
1021 | .br | |
1022 | Unfortunately, heavy network users would have to repeatedly | |
1023 | type their password. | |
1024 | .IP b) | |
1025 | Encrypt the ``.netrc'' file. | |
1026 | .br | |
1027 | A program would have to exist to decrypt the file. | |
1028 | A superuser could get access to whatever key technique that program used, | |
1029 | if it were on the local machine. | |
1030 | A public-key encryption scheme would make this option possible. | |
1031 | We decided it was too much work to implement this. | |
1032 | .IP c) | |
1033 | Once-a-session passwords. | |
1034 | .br | |
1035 | In this scheme, a user would register his password when he logged in, | |
1036 | then use the network without needing to type in a password. | |
1037 | When he logged off, the password would be removed. | |
1038 | We discarded this because we could not guarantee the password would | |
1039 | disappear unless we wrote a daemon, which itself could be compromised. | |
1040 | The best solution along this line uses the ``alias'' feature | |
1041 | of the C shell. | |
1042 | Each net command is aliased with itself and the | |
1043 | \fB\-l\fP, \fB\-p\fP options. | |
1044 | When the user logs in, | |
1045 | he sets a shell variable to his password. | |
1046 | When he logs off, his shell dies and the passwords are forgotten. | |
1047 | .RE | |
1048 | .PP | |
1049 | I believe the current alternatives are sufficient for a conscientious | |
1050 | user to protect himself and still have an easy-to-use network interface. | |
1051 | .SH | |
1052 | .sp 1 | |
1053 | .ce | |
1054 | Future Plans | |
1055 | .sp .5 | |
1056 | .NH 0 | |
1057 | Hardware | |
1058 | .PP | |
1059 | The net has suffered with low speed hardware. | |
1060 | Short-term plans include speeding up the current terminal | |
1061 | interface hardware from 1200 Baud to 9600 Baud and writing a driver | |
1062 | for the device to bypass the internal | |
1063 | .UX | |
1064 | character queues. | |
1065 | This driver will improve the reliability of transmission and decrease the character | |
1066 | interrupt overhead. | |
1067 | The speedup from 1200 Baud to 9600 Baud may overload the systems due | |
1068 | to the number of hardware interrupts it causes. | |
1069 | .PP | |
1070 | In the longer term, the EECS Department is considering acquiring | |
1071 | direct-memory-access (DMA) devices such as the Logical Network Interface (LNI) | |
1072 | or the Digital Corp. DMC-11 links for high-speed transmission. | |
1073 | These devices are capable of over 1-megabit speeds, and | |
1074 | would increase the speed of the current network by factors of hundreds. | |
1075 | .NH | |
1076 | Adding More Machines to the Network | |
1077 | .PP | |
1078 | The \s-2INGRES\s0 Research group and various other research units within | |
1079 | the EECS department have expressed interest in being added to the network. | |
1080 | .NH | |
1081 | Remote Use of the Typesetter Facilities | |
1082 | .PP | |
1083 | The Computer Center A machine has a Graphics Systems phototypesetter | |
1084 | and the VAX Research machine has a Versatec 36'' dot-matrix plotter | |
1085 | with a | |
1086 | .I troff | |
1087 | simulator. | |
1088 | Software now being debugged will allow remote use of the typesetter | |
1089 | by running the | |
1090 | .I troff | |
1091 | program locally and sending the typesetter device codes to the remote | |
1092 | machine. | |
1093 | .PP | |
1094 | This will distribute the typesetting load and help overloading on | |
1095 | the A and VAX machines. | |
1096 | It will also allow the use of | |
1097 | .I troff | |
1098 | macro packages only available on some machines. | |
1099 | .NH | |
1100 | Remote Mail Forwarding | |
1101 | .PP | |
1102 | The | |
1103 | .UX | |
1104 | mail programs will be modified to forward mail to another account on another | |
1105 | machine, allowing a user with accounts on many machines to read it all on one | |
1106 | designated machine. | |
1107 | .SH | |
1108 | .sp 1 | |
1109 | .ce | |
1110 | Summary Points | |
1111 | .sp .5 | |
1112 | .PP | |
1113 | The author would like to stress these points about his experience: | |
1114 | .IP 1. | |
1115 | Success in building networking software depends on having ready access | |
1116 | to the correct hardware. | |
1117 | The minimum is two terminals connected to two usable machines with two | |
1118 | magnetic tape drives or some other existing means of software transfer. | |
1119 | .IP 2. | |
1120 | Design in portability and security. | |
1121 | More careful attention to machine dependence and security | |
1122 | in the first phase would have saved much later re-programming. | |
1123 | .IP 3. | |
1124 | Develop good local debugging techniques. | |
1125 | The ``self-loop'' trick for network debugging depends on the accuracy | |
1126 | of that simulation. | |
1127 | .UX | |
1128 | pipes, for example, were not sufficient to simulate TTY lines | |
1129 | because TTY lines are 7-bits with a restricted ASCII range. | |
1130 | .IP 4. | |
1131 | Encourage users to use the system. | |
1132 | Their feed back is important. | |
1133 | However, it is necessary to have an unused network link for new | |
1134 | protocol development, etc. | |
1135 | .IP 5. | |
1136 | There is a fine line between support of an existing network and research. | |
1137 | In the best of all possible worlds, support of research-developed | |
1138 | software would be the responsibility of the systems staff | |
1139 | for the machines it runs on. | |
1140 | This is seldom the case. | |
1141 | .IP 6. | |
1142 | The concept of layers of networks was very helpful in this project. | |
1143 | There appear to be these levels: | |
1144 | .DS L | |
1145 | .ce 99 | |
1146 | .ls 2 | |
1147 | .sp 2 | |
1148 | user interface | |
1149 | queues and daemons | |
1150 | command protocol | |
1151 | packet protocol | |
1152 | transmission protocol | |
1153 | teletype lines | |
1154 | .sp 2 | |
1155 | .ls 1 | |
1156 | .ce 0 | |
1157 | .DE | |
1158 | These levels are quite distinct. | |
1159 | If a new, better network not involving queues is built, the transfer of files | |
1160 | could still be by | |
1161 | .I netcp. | |
1162 | If state-of-the-art link hardware is used, perhaps all of the levels | |
1163 | below the command protocol could be discarded. | |
1164 | .IP 7. | |
1165 | The chief virtue of the current system is its extreme flexibility | |
1166 | and low start-up costs. | |
1167 | No modifications to the | |
1168 | .UX | |
1169 | kernel are required and all local features are conditionally | |
1170 | specified in a header file. | |
1171 | .IP 8. | |
1172 | Networks are hard to build because | |
1173 | .RS | |
1174 | .IP a) | |
1175 | They involve mutually-cooperating copies of software on (usually) | |
1176 | differing computers. | |
1177 | .IP b) | |
1178 | Many options are not practical because of compatibility considerations \- | |
1179 | new networks need drivers in unchangeable systems, and new protocols have to | |
1180 | accept the old protocols until the old protocols are extinct. | |
1181 | .RE |