Mobile code systems take two fundamental approaches to the security problem. Most of them attempt to keep programs from doing anything dangerous, or at least from doing anything dangerous without asking you about it first. For instance, JavaScript isn't allowed to write files to your disk without your consent; Java isn't allowed to open network connections to any server other than the one that your original connection was to. This approach results in web pages that can't quite do what you want but can still do some things that you don't want.
To avoid this problem, ActiveX takes the second approach and uses digital signatures to attempt to give you some idea where a program comes from, in the hope that this will tell you whether or not to trust it. There are no restrictions on what an ActiveX control can do. This approach results in web pages that you're not certain whether or not to trust. Who is the person who signed it? Do you trust that person to be both well intentioned and competent to write security-critical programs? If a program is not signed, is it because the author is evil, or because the author is lazy, uneducated about digital signatures, or arguing with the signature authority?
These two approaches are being combined, allowing you to decide exactly what a program can do based on what you know about who wrote it. This is still imperfect, but it's the most promising approach, and both Java and ActiveX are moving in this direction.
JavaScript attempts to provide security by limiting what it's possible to do. For instance, JavaScript does not have commands for reading and writing files. Furthermore, programs written in JavaScript are supposed to be able to access only limited amounts of data: information about user preferences, available plug-ins and capabilities of the browser, and links and forms in the document that contains the script and in windows created by that web page. JavaScript programs can communicate to the outside world only by submitting forms.
A web page with a JavaScript program can contain something that is technically a form without being visible, and JavaScript can fill in that form with any information it can get to and submit it automatically. If your web browser is configured to ask for confirmation before submitting forms, you will see a warning; if not, the entire transaction will be silent. Furthermore, submitting a form simply consists of sending information in a predefined format to a URL, which can specify any IP address and port number. This means that submitting forms effectively lets JavaScript send any information it has to anybody it thinks should have it, possibly without notifying you. This is one of the reasons that web browsers warn you about form submissions, and it is a good reason for leaving these warnings in place.
Most JavaScript security problems fall into one of two categories: denial of service attacks that hang or crash the browser (or, if you're really unlucky, the entire computer), and bugs in the data access limitations that allow JavaScript programs to read arbitrary files and return the data to the site that the page came from. In addition, there have been occasional buffer overflow problems with JavaScript implementations.
The denial of service attacks are, as usual, unavoidable, but they're no worse than annoying. Ones that actually crash the browser or manage to affect the machine are straightforward implementation bugs, and fixes are usually rapidly available. The standard way to hang the browser is to code an infinite loop.
While JavaScript programs cannot directly open files, these programs can cause the web browser to open them by giving the browser local URLs to open (for instance, file:/etc/passwd). When the JavaScript security is correctly implemented, this is not a problem, since JavaScript still can't get the data out of the newly opened URLs. However, there have been a series of bugs with the handling of this particular limitation, where the browser becomes confused about the correct context for scripts or data embedded in odd places. For instance, scripts in the titles of pages may be executed in the process of reading bookmarks, and scripts can manage to execute in windows that are supposed to be displaying source code. There have been dozens of bugs in context handling, all of which have roughly similar effects; the script returns data it shouldn't have to the server it was on. That data ranges from information about sites that have been visited recently to the contents of arbitrary files on the disk.
The buffer overflow problems are, strictly speaking, not JavaScript vulnerabilities at all. They occur when the interpreter gets invalid JavaScript and the interpreter itself fails to enforce buffer limitations. There is a known problem with buffer overflows on some JavaScript calls in some versions of Internet Explorer 4.0.
JavaScript can use ActiveX or Java, if they are enabled, to get access to capabilities that JavaScript alone would not have. If ActiveX or Java is enabled in the browser, JavaScript can use them without containing any visible ActiveX or Java objects. This means that filtering out ActiveX and Java is not sufficient to protect a site. You must also filter out scripting languages or disable ActiveX and Java in all browsers.
There don't appear to be any reported VBScript security problems; this does not indicate that VBScript is secure. What it does indicate is that JavaScript is implemented by more browsers than VBScript, while ActiveX is more powerful than VBScript. Therefore, attackers are going to concentrate on JavaScript and ActiveX until they've exhausted their obvious possibilities. Because VBScript provides the same capabilities that JavaScript does, VBScript vulnerabilities are likely to fall into the same sort of categories as JavaScript vulnerabilities, involving mostly denial of service attacks and ways to read data.
In general, programming languages are divided into interpreted languages and compiled languages. An interpreted language, like Perl or Visual Basic, is one where you write a program, and when you want to run it, you give it to an interpreter in the same form that you wrote it. The interpreter, which is run every time you run a program in the language, is responsible for turning human-readable commands into instructions the computer can execute. A compiled language, like C, is one where you write a program and run it through a compiler once to get an executable. You can then run the executable all by itself; you don't need another program to make it run.
Interpreted languages are machine independent. You have a different interpreter for each kind of machine but run the same programs. Compiled languages, on the other hand, are machine dependent; once you compile a program, the result will run on only one kind of machine. On the other hand, a program that is running through an interpreter is slower than one that has been compiled. In addition, when you give somebody a program in an interpreted language, it's easy for them to modify it and reuse it, while a compiled program is much more difficult to alter.
Java uses an intermediate option, sometimes called byte compiling. A program written in Java is compiled into machine-independent Java byte code, which is then turned into computer instructions by an interpreter usually called the Java Virtual Machine. This gives some of the advantages of compiled code (it's faster to run than an interpreted language, and the code is not in an easily modifiable form) and some of the advantages of interpreted code (it's machine independent). It also gives it many of the disadvantages of both; it's slower than compiled code, you have to have an interpreter to do anything with it, and you can have problems with bugs in either the compiler or the interpreter. Just as it is possible for a determined person to write a program directly in machine language, without using a traditional compiler, it's possible for a determined person to write a program directly in Java byte code, without using a Java compiler, and the result may be acceptable to the interpreter even if it couldn't be generated from any program the compiler would accept.
There are security features in both the Java byte-code compiler and the Java runtime interpreter. In general, you should think of the Java compiler as providing security to Java programmers; it helps people write Java programs that cannot be attacked by hostile users. (For instance, Java programs are not susceptible to buffer overflow problems.) The Java interpreter provides security to Java users; it is supposed to keep hostile Java programs from damaging machines. Because people can write Java byte code directly, you can't rely on the compiler to protect you from malicious programs.
Instead, what you're relying on is something called the security manager, which is part of the runtime interpreter. The security manager is responsible for determining what a program is allowed to do. It does this by looking at each separate action the program attempts to take and comparing that to the security policy that's in force. Normally, there are two possible security policies: one that covers normal programs, which doesn't put any limitations in place, and one that covers programs that have been downloaded from the network, which restricts what files can be read and written, how much memory and disk space a program can use, and what network connections it can make.
The security manager doesn't directly control the operations that the program performs. Instead, it controls what functions the program can call. For instance, insecure programs are not normally allowed to write to disk, but if there is a library that is supposed to be safe for use by insecure programs, an insecure program can call that library and have the library write to disk. Effectively, this is the same sort of restriction as allowing a child to eat cookies, but only if an adult gets the cookies; you are requiring transactions to go through a theoretically trustworthy intermediary that will impose limits on them.
There are two main risks to this model. First, there is the risk that the security manager will permit something that it should have denied. Second, there is the risk that a theoretically secure library will contain insecure operations. Java has had problems with both of these, but mostly with the security manager. The security manager has a very complex task, and it is extremely difficult to implement correctly. Since the original release of Java, people have found both implementation and design flaws with some regularity. Although these have been rapidly fixed for the most part, and the rate has considerably slowed down, it is reasonable to expect that there will be continuing problems.
ActiveX is an extension and update of Microsoft's Object Linking and Embedding system (OLE). ActiveX controls can be written in any of a number of languages, including C and Visual Basic, and can be accessed from an even wider variety of languages. In the context of the Web, they are usually used from HTML, JavaScript, or VBScript, but they can be used from Java as well. No matter what language is used to access an ActiveX control, it is the ActiveX security model that applies to the control, not the calling language's; this is important because it means that an ActiveX control used in a Java program will not be constrained by the Java sandbox.
ActiveX security is provided in two ways. First, there are limitations on when an ActiveX control can be read in; second, there are limitations on when an existing ActiveX control can be executed. These limitations are part of the implementation of the current ActiveX interpreters, not part of the language design, so there is no guarantee that future ActiveX implementations will have the same characteristics.
The most famous aspect of ActiveX security is its use of digital signatures, which is part of the security system for reading in ActiveX controls. An ActiveX control may be signed with a digital signature, which theoretically allows you to identify the author of the control and make a decision as to whether or not to allow the control to be loaded. ActiveX controls do not have to be signed, but unsigned controls are normally treated differently from signed controls. (See Appendix C, "Cryptography", for further discussion of digital signatures and what they mean to you.) By default, unsigned controls from external web pages are rejected, and signed controls request confirmation.
Digital signatures are checked when a control is loaded, but once the control has been loaded, it has to be used in order for anything to happen. The ActiveX model also provides controls over when a control can be run. You can choose to allow controls to run, forbid them from running, or be asked every time something attempts to run a control. By default, controls are allowed to run without confirmation.
In addition, a control can claim to be safe for use with untrusted data and/or safe for use from scripts. By default, a control cannot be run by an external web page unless it claims both these properties. A control that claims both these properties is supposed to be one that will never do anything bad, even if used hostilely. Programs like Internet Explorer that use ActiveX objects look at these properties to decide whether or not to allow a control to be run. Aside from these restrictions, once a control has been loaded for any purpose, it is available from any program.
Note that this means that if a control is present on your local disk, and it claims to be safe for use with untrusted data and safe for use from scripts, any web page can run it, and no request for confirmation will be made before it is run. This can lead to unpleasant surprises. Many vendors have preinstalled ActiveX controls that will allow arbitrary commands to be run and have incorrectly marked them as safe for scripting. Not only have third-party vendors like Compaq shipped machines with dangerous ActiveX controls, but even Microsoft, in Windows 98, provided a control marked as safe for scripting that could be used to run arbitrary programs.
Obviously, the same sort of thing can be done with controls downloaded from web pages. You may download a control from one web site, which is then activated by another. Less obviously, if you go to a page and download a control once, when you revisit the page, it will attempt to download the control again. If you do not let it download the control, you will still have the original copy of the control installed, and the page will happily continue to run that copy, which is presumably not what you had in mind (if you wanted it to run the control, you would have let it download the control). People who have just clicked No on a download dialog box are usually surprised and outraged when the control runs, since the distinction between downloading and running is not normally user-visible.