24. Deployment Considerations
Any time you embark on an enterprise project, the costs, return on investment, risks, benefits, threats, and workflow (to name a few) should be considered. When deploying a PAM solution, the realization that it may impact the entire organization needs to be addressed with everyone who may potentially be impacted—from employees to vendors. This means that not only administrators will be affected, but also end users who may lose administrative rights. This can affect rank and file workers and executives, all the way through contractors (although I hope your business never gives temporary employees admin rights; sadly it happens). Deciding where to start, how to deploy, how to educate, and the measurable outcome are challenges that must be addressed up-front. If they are not, internal politics, user resistance, and shadow IT may completely circumvent the reasons for embracing PAM in the first place. This chapter covers some of the deployment considerations all executives, security professionals, and operational teams should consider, discuss, and address along their PAM journey.
Lack of visibility and awareness of all the privileged accounts and credentials across an enterprise poses a monolithic challenge, especially for those companies that rely on manual processes and tools. Privileged accounts, many long forgotten, are sprawled across most organizations including desktops, servers, hypervisors, cloud platforms, cloud workloads, network devices, applications, IoT devices, SaaS applications, and more. Different teams may be separately managing (if managing them at all) their own set of credentials, making it difficult to track all the passwords, let alone who has access to them and who uses them. An administrator can easily have access to more than a hundred systems, possibly disposing them to take shortcuts in order for them to maintain their credentials.
With this proliferation of privileges scattered throughout the environment, where do you start? In some cases, organizations will start with the end users and target desktops and remove administrator rights to mitigate threats like ransomware. In other cases, they will start by protecting the ∗nix server environment supporting critical business applications, like trading floors or banking systems. In some, they will need to adhere to third-party vendor monitoring as a compliance requirement. Perhaps they have a nearer-term need to focus on a subset of assets to respond to an audit finding, such as properly securing and managing assets connected to the secured PCI network segment. Whether you begin with servers, desktop, networking devices, and/or other connected devices, your decision is a function of risks, complexity, and cost. Ask yourself where the biggest pain is first, what is the risk of tackling it first, and can it be successful? Once you understand the risk and pain, you start by “ripping the Band-Aid off” or “picking the lowest-hanging fruit” to prove success and gain experience.
For many organizations, quantifying this risk from day one through the sustainment of the solution is a problem. How do you actually measure privileged risk? This will vary from organization to organization and is generally modeled after the regulatory compliance requirements governing your organization. A fault in a control, therefore, gives you an indication of risk. After all, it is not like a CVE1 with a CVSS2 score; measurements you choose to use need a defendable basis.
Privileged Credential Oversight
Even if IT successfully identifies all the privileged credentials strewn across the enterprise, this does not by default translate into knowing what specific activities are performed during a privileged session (i.e., the period during which elevated privileges are granted to an account, service, or process and actively being used). Privileged access to a superuser account should not amount to ceding carte blanche to the user. Moreover, PCI, HIPAA, and other regulations require organizations to not just secure and protect data but be capable of proving the effectiveness of those measures. So, for both compliance and security reasons, IT needs visibility into the activities performed during the privileged session.
Ideally, IT should also have the ability to seize control over a session should inappropriate use of the credentials occur. But, with potentially hundreds or concurrent privileged sessions running across an enterprise, how does IT expeditiously detect and halt malicious activity? While some applications and services (such as Active Directory) can log user actions, and while Windows servers using logon events within Event Log data can reveal some behavioral anomalies, expect full coverage of privileged account activity to require a complete implementation of PAM, not just managing passwords. And contemplate the use cases needed to track oversight, auditability, and the necessary infrastructure when designing your deployment and workflow. Credential oversight needs its own security to prevent misuse of stored sessions, and potentially large quantities of storage if session archives are required to be stored for long periods of time.
IT teams commonly share root, Windows Administrator, and many other privileged passwords so workloads and duties can be seamlessly shared as needed. However, with multiple people sharing an account password, it may be impossible to trace actions performed with an account to a single individual, complicating auditing and accountability. For a successful deployment, assess how often this problem occurs and where it needs to be addressed with PAM. Simply put, determine how often and where in your environment users are sharing privileged accounts and how you can eliminate this poor behavior with PAM. This is true for every user that has a privileged account—from server administrator to network infrastructure engineer to help desk technician.
Privileged credentials are needed to facilitate authentication for app-to-app (A2A) and application-to-database (A2D) communications. Applications, systems, and IoT devices are commonly shipped, and often deployed, with embedded or backdoor default credentials that are easily guessable and pose a formidable risk until they are brought under management. These privileged credentials are frequently stored in plain text, perhaps within a script, code, or a file on the device, or even in documentation. Unfortunately, there is no universally efficient method to detect or centrally manage passwords stored within applications or scripts. Bluntly, every implementation is different. Securing embedded passwords requires separating the password from the code so that when it’s not in use, it’s securely stored in a centralized password safe or secret store. For a successful deployment, identification of all the embedded credentials is critical when implementing PAM and how you handle fault tolerance when they are removed ensures there is no interruption to your business once you do.
IT teams commonly rely on SSH keys to automate secure access to servers, bypassing the need to enter login credentials manually. SSH key sprawl presents a substantive risk for thousands of organizations, which may have upward of a million SSH keys and present viable backdoors for hackers to infiltrate critical servers. Look at your own environment and ask, where are SSH keys, how are they being managed, and what do you do when they expire? Realistically, PAM can manage SSH keys, so environments never get in this situation. It all starts with a discovery of keys and an automated process to bring them under management. This is a deployment consideration that is often overlooked as teams focus on password management for end users, while neglecting to address those SSH keys used by administrators and applications.
Privileged Credentials in the Cloud
The challenges of visibility and auditability are generally exacerbated in the cloud. They are, after all, not your computers and you have limited “rights” for visibility. Cloud and virtualization administrator consoles (as with AWS, Office 365, Azure, etc.) provide vast superuser capabilities, enabling users to rapidly provision, configure, and delete resources at a massive scale. Within these consoles, users can spin up and manage thousands of virtual machines, containers, and other services (each with its own set of privileges and privileged accounts) with just a few clicks. One predicament then arises around how to onboard and manage all the newly created privileged accounts within “everything” that has been created and, just as important, correctly deprovision them once a resource is decommissioned. On top of this, cloud platforms frequently lack native privileged session monitoring capabilities that work at a granular level to audit exactly what was done. And, even for those organizations that have implemented some degree of automation for their password management, if not architected with the cloud in mind, there’s no guarantee a password management solution will be able to manage cloud credentials adequately through all of these resources. For a successful deployment, determine how many cloud services your organization is using, who has privileged access, and how the resources are being accessed, maintained, and monitored. And, remember to ask about the entire workflow, including new account onboarding as well as account offboarding. Many times, the latter is overlooked and your PAM implementation is bogged down with junk accounts it is trying to manage.
The concept of functional accounts is used within privileged access management (PAM) and identity and access management (IAM), referring to accounts used to perform automated account management functions, regardless of being local, centralized, within an operating system, application, on-premise, or in the cloud. Simply put, functional accounts help to manage other accounts. Functional accounts have elevated privileges and, in many implementations, domain administrator or root privileges across multiple resources. Management functions can include, but are certainly not limited to, account creations and deletion, password rotation, account enablement or disablement, and group membership placement or revocation.
A good functional account architecture limits the reach of each instantiation and prefers multiple functional accounts governing zones, resources, assets, and applications vs. a few that have nearly godlike, or domain privileges, across the entire environment. These accounts typically also fall outside of any just-in-time management for identity and privileged access management solutions since they must be considered “always-on” in order to perform their automated functions. The latter makes it easy to understand that if a functional account is compromised, repercussions are quite pronounced, and every account under the functional account’s control (managed account) is in jeopardy too.
As an example, consider a deployment of Windows resources within your environment. The resources could be servers or laptops. In this scenario, a functional account would manage all of the privileged and service accounts assigned to the resource and linked to other systems that must share the same credentials. They can be rotated and checked in and out on-demand, or based on a workflow. All management for these accounts, whether they are local or domain-joined, is accomplished via the functional account. The goal is to ensure the credentials are always unique, never become stale or dormant, and are changed frequently enough to mitigate risks of the privileged credentials being stolen or misused.
Functional accounts should never be associated with any identity. They operate independently.
They are strictly used for automation from an IAM and PAM solution. They should not be used by other applications.
They should never be used for any daily work. Ever!
They should be managed like any other highly privileged account and passwords or certificates combination and be rotated periodically to prevent them from becoming stale. This must be done with great care to ensure dependent management functions do not break due to a missed password change or error.
Functional accounts should be excluded from any just-in-time IAM or PAM initiatives.
Whenever possible, they should be local accounts and not domain accounts. However, certain applications and implementations will necessitate exceptions. Follow this simple rule: if it can be managed or implemented without using a domain account, that is probably a lower-risk method.
Functional accounts are a necessary concept to place privileged accounts under management. While they have elevated privileges to perform their functions, they must be treated as a high security risk and deserve protection that even exceeds that of domain administrator credentials. IAM and PAM solutions can manage these expectations for an environment, but some basic do’s and don’ts should always be honored when considering your deployment.
Traditionally applications only had to store credentials when trying to authenticate against external resources. Some examples are remote databases, file shares, or directory stores. Ensuring that developers securely store these credentials has always been a challenge. Unfortunately, developers have created a large number of applications over the years that store these credentials in plain text (or even poorly hashed) within the configuration files of the application. With the explosion of cloud computing and SaaS and IaaS offerings over the last 5 years, applications are increasingly interacting with many platforms, and not just a single external resource. It is common for configuration files to have many credentials to connect to other external resources, including API keys. One of the promises of zero trust is to eliminate this credential storage model and use a third-party policy engine to broker authentication. Often, API keys are not seen as the sensitive piece of information that should be protected by developers. This is evident by the number of applications where effort is put forth to securely store credentials, yet API keys for cloud resources are left in plain text and, sometimes, even posted in public Internet forums. How many times have developers pushed code to GitHub with API keys included or accidentally exposed API keys while posting source code to Stack Overflow? The carelessness is shocking.
As with traditional resources, when investing in the cloud, we need to push developers to achieve the highest application goals, but with the least amount of privileges. This philosophy is hard to abide by with most public APIs. With traditional usernames and passwords, it is often possible to create role-based access with limited privileges. Developers need to be aware that API keys usually grant applications access to the entire environment. This is contrary to the principle of least privilege. Exposure of an API key cannot be contained to the minimal amount of functionality that the consuming application requires. SendGrid is one of the exceptions to this and does an adequate job providing fine-grained control to limit the functionality that the API key is allowed to consume.
As enterprises continue to migrate workloads to the cloud and advocate for more secure coding, API security and vendor platform security will continue to mature. PAM has a place by ensuring that privileges are not Boolean and any programmatic application access also has a fine-grained privileged model. When considering your PAM deployment, consider how applications authenticate and specifically whether they use API keys. These should not be hard-coded in your applications, but rather centrally managed in a secret store.
There is one thing certain about securing privileged access, by the time you read this book, something will have changed. Since the original release of Privileged Attack Vectors in 2017, concepts like one-time passwords (OTPs) and behavioral authentication have spiked in popularity, while other methods have withered in failure or otherwise fallen into disuse. One new promising concept that was introduced to the mainstream consumer community in 2019 was application-specific passwords. This has not translated yet to the enterprise, but there are definitely merits in the concept that will probably lead to commercial implementations soon.
The concept behind application-specific passwords is simple—an identity has only one account and, therefore, only one username. This is different from the traditional one identity to multiple account model. However, each account can have multiple passwords where each password is unique per application. A user effectively logs on to a management console and generates a random password for a new application. The user registers the application with a unique name like “Outlook” and has a finite amount of time to create an account and authenticate from that application. The application is then fingerprinted, or a key exchanged, during the initial authentication to trust the application. If the password is attempted to be used by another application due to it being stolen or some form of password reuse, it is denied access and the application-specific password potentially locked out. While this creates a potential management nightmare for a user with hundreds of accounts, it does solve key privileged access management problems from lateral movement to password reuse. It is a newer concept worth keeping an eye on and a deployment consideration since most enterprise password management solutions cannot interact with this paradigm yet.