Principles of Secure Programming
Sven Anders Robbestad
15 October 2019

Design Principles

In 1975 two computer scientists named Jerome Saltzer and Michael Schroeder wrote a paper called The Protection of Information in Computer Systems that listed a set of design principles that from their experience are important for the design of secure software.

https://www.cs.virginia.edu/~evans/cs551/saltzer/

These principles cover not only technical details but also human interaction.

The list is based on the ideas of simplicity and restriction.

Simplicity

• Less can go wrong
• Easier to understand
• Fewer inconsistencies

Restriction

• Minimize access ("need to know")
• Inhibit communication

The list has been modified in a later article, and other computer scientists have suggested additional principles. In this article, I'll cover the original list, including the newer ones.

The list is as follows:

• Least Privilege
• Fail-safe Defaults
• Economy of Mechanism
• Complete Mediation
• Open Design
• Separation of Privilege
• Least Common Mechanism
• Least Astonishment & Psychological Acceptability
• Work factor
• Compromise recording

These principles should underlie all designs and mechanisms including implementation.

They don't tell you how to apply them, because the implementation details are specific for each piece of software you write, or system you design. It follows that applying them requires a good understanding of the goals and mechanisms of the environment.

Principle of Least Privilege

Every program and every user of the system should operate using the least set of privileges necessary to complete the job.

This limits the damage that can result from an accident or error, and reduces the number of potential interactions among privileged programs to the minimum for correct operation.

The function of the entity should control the assigment of rights, not its identity. Rights should be added as needed, and discarded after use.

Examples

• Unix "sudo"
• Windows User Account Control

Both of these allow a user to apply administrative rights temporarily to perform a privileged task.

Principle of Fail-safe defaults

Base access decisions on permission rather than exclusion.

Deny access by default

You should base access decisions based on permission rather than exclusion. The idea is that if any action fails, the system is as secure as when action began.

A design or implementation mistake in a mechanism that explicitly excludes access tends to fail by allowing access, a failure which may go unnoticed in normal use. This principle applies both to the outward appearance of the protection mechanism and to its underlying implementation.

Example

When sanitizing input, by applying fail-safe defaults, you allow only pass the characters you allow, rather than checking for characters you don't allow.

Principle of Economy of mechanism

Keep the design as simple and small as possible

When code is simple, less can go wrong. A simple design is easier to test and easier to validate.

Code should be easy to read, easy to amend. Don't add complexity. Complexity adds confusion. Confusion leads to fragile programs.

This principle also requires that you create interfaces, because when you don't have interfaces, other modules may make erroneous assumption about input parameters.

This well known principle (KISS) is important because: "a design or implementation mistake in a mechanism that explicitly excludes access tends to fail by allowing access, a failure which may go unnoticed in normal use. This principle applies both to the outward appearance of the protection mechanism and to its underlying implementation."

Principle of Complete mediation

Every access to every object must be checked for authority.

This is usually done once, on first action. However, you should strive to validate every time access occurs.

A foolproof method of identifying the source of every request must be devised. It also requires that proposals to gain performance by remembering the result of an authority check be examined skeptically. If a change in authority occurs, such remembered results must be systematically updated.

Example

On Linux systems, access is checked on open, but not thereafter. If permission change, you may get unauthorized access. (This violated the principle).

Principle of Open design

The mechanisms should not depend on the ignorance of potential attackers, but rather on the possession of specific, more easily protected, keys or passwords.

Do not depend on secrecy of design or secrecy of implementation ("Security through obscurity"). This is inherently insecure. Humans are fallible, and your secrets may end up being published or known in ways you don't expect.

It is simply not realistic to attempt to maintain secrecy for any system which receives wide distribution.

However, this does not mean that you should publish passwords or cryptographic keys, nor does it mean that you should publish your source code.

Principle of Separation of privilege

A protection mechanism that requires two keys to unlock it is more robust and flexible than one that allows access to the presenter of only a single key.

A system should not grant permission based on a single condition

A mechanism that requires two keys to unlock it is more robust than one that allows access using a single key.

The idea is that no single accident, deception, or breach of trust is sufficient to compromise the protected information.

Example

• Payments for more than a $10.000 must be signed by two officers of the company. This is separation of privilege because you need two persons to verify the payout.

• Bank safe-deposit boxes where the bank has one key, the customer the other.

• Opening a password-protected file on a system requires a password to open. In addition, you also need access to the file system to read the file. This is separation of privilege, because you need access from both the system and the password protection, but each works separately and doesn't know or care about each other.

Principle of Least common mechanism

Mechanisms used to access resources should not be shared

Minimize the amount of mechanism common to more than one user and depended on by all users.

Don't share what you don't need to share. This applies to information, but also shared mechanisms. Information can flow along shared channels.

For instance, if you and I are working on separate system, but share CPU time, then the CPU is a shared channel by which attacks may happen.

Another example: in the early days of Internet hosting it was common to get a login to a server to upload your web content, either by file transfer or live editing on a terminal. These web server were shared by a number of users, and access to your space was given on login. As it happened, a bug in the implementation meant that you could access other people's account by simply issuing a change folder command. In this day and age, a security flaw such as this would be uncovered in minutes, but back in the day this system kept running for months without issues.

Isolation is the primary mechanism to uphold this principle.

There's basically two methods of isolation, virtual machines and sandboxes. By restricting the execution of software on isolated virtual machines or in isolated sandboxes, it's impossible to target weaknesses outside the environment.

This principle also cover Distributed Denial of Service attacks, in which attackers attempts to deny access to a resource on the internet by flooding the resource with messages (or "packets"). Countermeasures include traffic throttling and proxy server to target suspect connections.

The principle of Least Astonishment

Don't surprise the user

Security mechanisms should be designed such that the user understands how and why the system works the way it does.

The designer should create a system that is easy to understand from the user's point of view and hide the complexity of the security mechanism.

Another way of looking at it: The resource should be no more difficult to access than if the security mechanisms were not present.

This applies to thing such as installing software as well.

Stay in the mental model of the user, and don't require things that will astonish the user.

If you randomly change your login system from requiring a username and a password, to a username, a password and a screenshot of your passport, that would surprise and alarm the user.

Psychological Acceptability

This is a related principle to "least astonishment".

It is essential that the human interface be designed for ease of use, so that users routinely and automatically apply the protection mechanisms correctly.

For instance, asking for a username and password is annoying, but acceptable, if you do it once, or if you say that access is granted for a certain time. On the other hand, if the system requires you to enter your username and password every time you access a resource, you will annoy the user and violate both principles.

Other principles

Saltzer and Schroeder also noted two other principles that works in the context of physical security, but which can only by applied "imperfectly" in software.

Work factor

Compare the cost of circumventing the mechanism with the resources of a potential attacker.

Stronger security measures pose more work for the attacker.

A strong door is harder to breach than a weak door. A moat makes it harder to reach the door. A castle is more secure than a house, and so on.

However, while the work factor can be easily calculated in some cases (a four letter alphabetic password for instance), many computer protection mechanisms are not susceptible to direct work factor calculation. "Defeat can be accomplished only by indirect strategies, such as waiting for an accidental hardware failure or searching for an error in implementation. Reliable estimates of the length of such a wait or search are very difficult to make."

Related, "tiger teams" — groups of professional hackers hired to attack a computer system to determine its security — were used systematically in the early 70s to find flaws in software system which allowed successful penetration. These attacks showed that "work factor" is an inefficient strategy to secure systems.

Compromise recording

It is sometimes suggested that mechanisms that reliably record that a compromise of information has occurred can be used in place of more elaborate mechanisms that completely prevent loss.

The system should keep records of attacks even if the attacks aren't necessarily blocked. The authors were skeptical about this, since the system ought to be able to prevent penetrations in the first place. If the system couldn't prevent a penetration or other attack, then it was possible that the compromise recording itself may be modified or destroyed.

Work factor and Compromise recording today

Most analysts and developers embrace these final two design principles today. The argument underlying complex password selection reflect a work factor calculation, as do the recommendations on choosing cryptographic keys. Compromise recording has become an essential feature of every secure system in the form of event logging and auditing.

Updated list:

Security principle: Minimize secrets – a thoughtful addition to the list that could be prone to misunderstanding. Secrets should be few and changeable, but they should also maximize entropy, and thus increase the attacker's work factor. The simple principle is also true by itself, since each secret increases a system's administrative burden: a late 1990s fighter jet project required dozens of separately-managed crypto keys to comply with data separation requirements that had been added piecemeal.

General principle: Adopt sweeping simplifications – a restatement that acknowledges how hopelessly complex modern systems have become. In the 1970s, a Unix operating system could support a dozen separate users with a megabyte of RAM; a single user on a modern desktop easily consumes a gigabyte of RAM, much of it containing software programs.

General principle: Principle of least astonishment – a concise and much clearer restatement of the "psychological acceptability" principle described in 1975.

General principle: Design for iteration – an important first step towards incorporating continuous improvement as a design principle.

Neither of the uncertain principles listed in 1975 made it into this revised list. Despite this, event logging and auditing is a fundamental element of modern computer security practice. Likewise, work factor calculations continue to play a role in the design of information security systems.

Elementary Information Security

The textbook Elementary Information Security presents a set of eight basic information security principles. While many directly reflect principles from Saltzer and Schroeder, they also reflect more recent terminology and concepts. The notion of "basic principles" stated as brief phrases seems like a natural choice for introducing students to a new field of study.

This yielded the following eight principles:

• Continuous Improvement – continuously assess how well we achieve our objectives and make changes to improve our results. Modern standards for information security management systems, like ISO 27001, are based on continuous improvement cycles. Such a process also implicitly incorporates compromise recording from 1975 and "design for iteration" from 2009. Introduced in Chapter 1, along with a basic six-step security process to use for textbook examples and exercises.

• Defense in Depth – build a system with independent layers of security so that an attacker must defeat multiple independent security measures for the attack to succeed. This echoes "least common mechanism" but seeks to address a separate problem. Defense in depth is also a well-known alternative for stating NIST's Principle 16. Introduced in Chapter 1.

• Open Design – building a security mechanism whose design does not need to be secret. This also repeats a 1975 principle. Introduced in Chapter 2.

• Chain of Control – ensure that either trustworthy software is being executed, or that the software's behavior is restricted to enforce the intended security policy. This is an analogy to the "chain of custody" concept in which evidence must always be held by a trustworthy party or be physically secured. A malware infection succeeds if it can redirect the CPU to execute its code with enough privileges to embed itself in the computer and spread. Introduced in Chapter 2.

• Deny by Default – grant no accesses except those specifically established in security rules. This is a more-specific variant of Saltzer and Schroeder's "fail safe defaults" that focuses on access control. The original statement is less specific, so it applies in safety and control problems. Introduced in Chapter 3.

• Transitive Trust – If A trusts B, and B trusts C, then A also trusts C. In a sense this is an inverted statement of "least common mechanism," but it states the problem in a simpler way for introductory students. Moreover, this is already a widely-used term in computer security. Introduced in Chapter 4.

• Separation of Duty – decompose a critical task into separate elements performed by separate individuals or entities. This reflects the most common phrasing in the security community. Some writers phrase it as "segregation of duty" or "separation of privilege." Introduced in Chapter 8.

The textbook's list focused on memorable phrases that were widely accepted in the computer security community. Principles introduced in earlier chapters always resurface in examples in later chapters. In retrospect, the list is missing at least one pithy and well-known maxim: "Trust, but verify." The book discusses the maxim in Chapter 13, but does not tag it as a basic principle.

Omitted Principles

For better or worse, three of the 1975 principles do not play a central role in modern information security practice. These are simplicity, complete mediation, and psychological acceptability. We examine each below.

There is no real market for simplicity in modern computing. Private companies release product improvements to entice new buyers. The sales bring in revenues to keep the company operating. The company remains financially successful as long as the cycle continues. Each improvement, however, increases the underlying system's complexity. Much of the free software community is caught in a similar cycle of continuous enhancement and release. Saltzer and Kaashoek (2009) call for "sweeping simplifications" instead of overall simplicity, reflecting this change.

Complete mediation likewise reflects a sensible but obsolete view of security decision making. Network access control is spread across several platforms, no one of which makes the whole decision. A packet filter may grant or deny access to packets, but it can't detect a virus-infected email at the packet level. Instead it forwards email to a series of servers that apply virus and spam checks before releasing the email to the destination mailbox. Even then, the end user might apply a digital signature check to perform a final verification of the email's contents.

Psychological acceptability, or the "principle of least astonishment" is an excellent goal, but it is honored more in the breach than in the observance. The current generation of "graphical" file access control interfaces provide no more than rudimentary control over low-level access flags. It takes a sophisticated understanding of the permissions already in place to understand how a change in access settings might really affect a particular user's access.

Only a handful of Saltzer and Schroeder's original 1975 design principles have stood the test of time. Nonetheless, this represents a memorable success. Kerckhoffs, a 19th century French cryptographic expert, published a list of principles for hand-operated cipher systems, some of which we still apply to cryptosystems today. But most experts only recognize a single principle as "Kerckhoffs's Principle," and that is his view on Open Systems: a cryptosystem should not rely on secrecy, since it may be stolen by the enemy. In addition to the Open System principle, both the principle of least privilege and of separation of privilege appeared on the 1975 list and are still widely recognized by security experts.

Perhaps lists of principles belong primarily in the classroom and not in the workplace. The short phrases are easy to remember, but they may promote a simplistic view of technical problems. Students need simplicity to help them build an understanding of a more complex reality.

Check for what is allowed, not what is disallowed

Distributed denial of service attack is a popular way to throttle web servers and bring traffic to a halt by blocking the network channel from receiving legitimate packagaes. A countermeasure to this is by analyzing the incoming traffic and blocking the bad packets at the router level, while still allowing good packets.

Puzzles

Fail-safe defaults: If an action fails, the system ensures it enters a safe state
Violation of open design: Keeping the code in a crytographic library secret
Fail-safe defaults: Deny access by default
Violation of Economy of Mechanism: Complex interfaces and complex code

Sources:

https://cryptosmith.com/2013/10/19/security-design-principles/