Snowflake: The Breach That Was Really a Password Problem

Published Category: Security 31 min read 6,181 words by Jimmy Nixx

Somewhere in an outer suburb, there is a self-storage place. You know the kind. A grey shed the size of an aircraft hangar, a gate with a keypad, a man called Terry in the office, and three hundred roller doors with a padlock each. The owners are very proud of the security. There are cameras on every corridor. There is a fence with spikes on it. The gate keeps a log of every code entered, to the second.

One Tuesday, a removalist truck backs up to unit 114 and starts loading. The man driving it has a keypad code and a key to the padlock. Terry waves. The cameras record everything in excellent definition. The truck leaves with a filing cabinet, eleven archive boxes and what appears to be a rowing machine.

Then it goes to unit 115. Then 116.

By Friday, the newspapers report a "massive breach at the storage facility". The owners say the fence was never touched. The tenants say they rented space in a secure facility, thank you very much. Somebody points out that the facility offered a second lock as an optional extra, a little fingerprint reader, and almost nobody on the list of burgled units had fitted it. And somebody else asks the question that ruins everyone's afternoon: where did a removalist get three hundred tenants' keys?

The answer, it turns out, is a pile of notebooks. Over several years, people had written their codes and keys in their cars' gloveboxes, and somebody had been quietly stealing the gloveboxes.

Put the kettle down for a moment. Nobody broke the cloud.

That is, more or less, what has happened over the last six weeks to customers of Snowflake, one of the largest cloud data platforms in the world. Live Nation told the US Securities and Exchange Commission that, on 20 May, it found "unauthorized activity within a third-party cloud database environment" holding mostly Ticketmaster data, and a Ticketmaster spokesperson confirmed to TechCrunch that the database was on Snowflake. This week, Neiman Marcus filed a notice for 64,472 people and named the same platform. The Australian Signals Directorate put out a "high" alert for Australian Snowflake customers, a rating that means act within 48 hours. And on 10 June, Mandiant, the incident response firm that led much of the investigation, said it and Snowflake had notified approximately 165 potentially exposed organisations.

The same report says something that should be printed on a tea towel and hung in every IT department: the campaign "is not the result of any particularly novel or sophisticated tool, technique, or procedure". Every incident Mandiant responded to traced back to stolen customer credentials. No zero-day. No master key. Old passwords, a door with one lock, and a lot of patience.

So we are going to follow the notebook from the glovebox to the storage unit. First, what actually happened and when. Then the malware that wrote the notebook, how it works, and where its pages get sold. Then the uncomfortable question of whose lock it was, where I will argue that both sides of that contract have something to answer for. Then the three locks, each taken down to the setting you actually change, and a worked example of what the logs show when it goes wrong. And finally, a list you can work through before the end of the week.

Let's get into it.

Part 1: Six Weeks, in Order

Breach stories arrive out of order. A forum post first, then a denial, then a filing, then a report that explains the first three. So, the sequence as of 27 June, assembled from the people who were there.

Mid-April. Snowflake later said it saw "increased threat activity beginning mid-April 2024" from a set of IP addresses and suspicious clients. In the same month, Mandiant received threat intelligence about database records that turned out to come from one victim's Snowflake account. It investigated and found the account had been accessed with credentials "previously stolen via infostealer malware", and that the account did not have multi-factor authentication (MFA) turned on. One victim, one stolen password, one missing second factor. That is the whole pattern, already visible in April.

14 May. Santander said it had found "unauthorized access to a Santander database hosted by a third-party provider", affecting customers in Chile, Spain and Uruguay and all current and some former staff. Its statement does not name the provider. Later reporting, including a Recorded Future News summary this week, lists Santander among the organisations that have come forward as victims of the campaign against Snowflake customers.

20 to 23 May. Live Nation found the unauthorised activity on 20 May. On 22 May, Mandiant had seen enough to recognise a wider campaign and contacted Snowflake. Snowflake says it became aware of potentially unauthorised access to customer accounts on 23 May.

27 May onwards. Someone offered what they said was Live Nation's user data for sale on the dark web. The administrator of BreachForums claimed to be selling the personal information of 560 million customers. That number is a seller's claim, and sellers are not famous for their honesty. Australian authorities said they were assisting Live Nation, and on 31 May, Ticketek told customers that names, emails and dates of birth held on a cloud platform run by a "reputable, global third-party supplier" may have been accessed. Ticketek did not name the supplier, so neither will I.

30 May to 3 June. Snowflake published its notice. A security firm, Hudson Rock, claimed that Snowflake itself had been breached through an employee's account; Snowflake denied it, and by 3 June Hudson Rock had pulled its report after legal pressure from Snowflake.1 On 1 June, the ASD's Australian Cyber Security Centre issued its high alert. On 2 June, Snowflake, CrowdStrike and Mandiant published a joint statement of preliminary findings. On 3 June, the US Cybersecurity and Infrastructure Security Agency asked users to hunt for malicious activity and report it.

10 June. Mandiant published its report. It named the attacker UNC5537, a financially motivated group it has tracked since May, which it believes has members in North America and works with one more in Turkey.2

17 June. Mandiant followed up with a 65-page threat hunting guide: queries a Snowflake customer can run to look for the same activity in their own logs.

This week. Neiman Marcus said an unauthorised party accessed its Snowflake-hosted database "between April and May 2024", and filed notices in Maine and Vermont. The data includes names, contact details, dates of birth and gift card numbers. A seller called Sp1d3r had offered it for $150,000 before the post came down.

Notice the shape. The data was taken in April and May. Most victims learned of it in late May or June, often because a stranger was selling it. The forum post arrived first, the explanation last. When people say "Snowflake breach", they mean all of this as one event. It was a few hundred small break-ins, done the same way, one account at a time.

Part 2: Infostealers, or the Notebook in the Glovebox

A Bosch-style tavern where a young man plays on a laptop while a rat-like quill creature crawls out and copies his keys into a notebook, and in the background a hooded merchant sells stacks of identical notebooks to a queue of buyers.
Fig. 1 — The game was free. The keys were not. Generated by an AI image model.

Start with something you do every day without thinking about it. You log in to a website, and your browser offers to save the password. You say yes, because you have 140 accounts and a finite number of brain cells. Next time, the box fills itself in. Lovely.

Where does the browser keep that password? In a file on your computer. It is encrypted, but the browser has to be able to decrypt it without asking you anything, because otherwise it could not fill in the box. So any program running as you, on your computer, can usually ask for the same thing. The lock on that file is built to keep out other people's accounts and stolen hard drives. It is not built to keep out software you ran yourself.

That gap has a whole industry living in it. Infostealer malware (or just "stealers") is software that runs on an infected computer, collects everything it can that looks like a credential, and sends it to its operator. In its 2023 report on the trade, the Secureworks Counter Threat Unit says many stealers finish "within several seconds to a minute of total runtime". There is no slow, creepy takeover with the mouse moving by itself. The thing runs, empties the glovebox and leaves before the kettle has boiled.

What goes in the notebook

The output of one infection is called a log. What is in it depends on the stealer. Secureworks describes the Raccoon stealer fetching a configuration file that tells it what to take: "password files, credit card information, web browser data, email messages, cryptocurrency wallets, browser cookies". It says Vidar also collects the machine's details, a list of installed software, "session data, and screenshots". So a single log can hold every saved password in the browser, the cookies that keep you logged in, and a picture of your screen at the moment of infection. Each saved password comes with the site's address and the username next to it, because that is how the browser stores them. To a buyer, that line reads as: this login, for this exact service, at this company.

How does it get onto a computer in the first place? The usual way: someone runs something they shouldn't. Secureworks says RedLine, one of the most common stealers, spreads through "cracked games, applications, and services", phishing and malicious ads, and even YouTube videos that promise free software. The free version of the expensive program, the game mod, the "activator". It installs the thing you wanted, and it also installs the thing you didn't.

Where the notebooks get sold

Logs are a product, and the product has a market. Stealers themselves are rented like any other software service: Secureworks found prices from $50 to over $1,000 a month for access to an operator's control server. The logs then go to marketplaces. In its May 2023 report, Secureworks counted over five million logs for sale on a single marketplace called Russian Market at the end of February 2023, about ten times more than its nearest rival.3 Mandiant, describing this campaign, says lists of stolen credentials exist "both for free and for purchase inside and outside of the dark web".

You do not need to infect anyone yourself. You need a search box.

What Mandiant found in this campaign

Now the specifics. Mandiant says the credentials UNC5537 used came mostly from infostealer infections on systems Snowflake does not own, and it names six stealer families: VIDAR, RISEPRO, REDLINE, RACOON STEALER, LUMMA and METASTEALER. According to Mandiant and Snowflake's analysis, at least 79.7% of the accounts the attacker used had prior credential exposure. The earliest infection linked to a credential the attacker used dates to November 2020. In total, Mandiant identified "hundreds of customer Snowflake credentials exposed via infostealers since 2020".

Since 2020. Some of those passwords had been sitting in a criminal's spreadsheet through four Christmases, a change of government and several tea strainers, and nobody had changed them.4

The contractor's laptop

One detail in the report deserves more attention than it has had. In several of these investigations, Mandiant found the first infection "occurred on contractor systems that were also used for personal activities, including gaming and downloads of pirated software". A contractor who helps several companies with their data platform may use one laptop for all of them, and for their evenings. If that laptop runs a stealer, Mandiant notes, a single machine "can facilitate threat actor access across multiple organizations, often with IT and administrator-level privileges".

Back to the storage facility. This is the locksmith who keeps everyone's spare keys on one ring in his ute, and also lends the ute to his nephew on weekends.

This was not credential stuffing

One distinction before we move on, because the term is going to be in your head. Credential stuffing is when an attacker takes passwords leaked from one site and tries them on many others, betting that people reuse passwords. It works because of reuse, and a password manager that makes a unique password for every site largely defeats it.

This was different. A stealer takes the password for the exact service, straight out of the browser that logs in to it. A long, unique, randomly generated password, saved in the browser, is stolen just as easily as "Password1". That is why the controls that worked here, as we will see, are the ones that do not depend on the password staying secret.5

Part 3: Walking In the Front Door

So the attacker has a username and password for a Snowflake account. What happens next is the part the "massive breach" headlines make sound dramatic. It is mostly administration.

Getting in

Mandiant says initial access often came through Snowflake's own web interface, Snowsight, or its official command-line tool, SnowSQL, run from a Windows Server 2022 machine. In other words, the front door, with the tenant's key. The attackers also used their own utility, which Mandiant tracks as FROSTBITE, in .NET and Java versions, to run reconnaissance queries: list the users, the current roles, the IP addresses and the organisation names.6 And they used DBeaver Ultimate, an ordinary database management tool that thousands of developers use every day.

They connected mostly from commercial VPN services, Mullvad and Private Internet Access, and moved data out through rented servers from a Moldovan hosting provider, storing it with several hosting companies and the cloud storage service MEGA.

Put the pieces together and you get the path in the diagram below. Every step uses a feature that exists for legitimate customers. The only thing that is stolen is the first key.

Five steps from top to bottom. Malware on a laptop copies saved passwords. The logs are sold. The attacker logs in with only a password, because the account had no MFA, the password had not changed since it was stolen and any network could connect. Any one of those three checks would have stopped the login. The attacker then copies the data out with four commands and extorts the owner.
Fig. 2 — Three missing checks, any one of them enough. Diagram based on Mandiant, "UNC5537 Targets Snowflake Customer Instances for Data Theft and Extortion", 10 June 2024.

The commands

Mandiant published the exact sequence it saw "repeatedly" across many customer accounts. It reads like an ordinary data engineer's afternoon, which is the point.

  1. SHOW TABLES lists every database and table the account can see. This is the attacker reading the labels on the boxes.
  2. SELECT * FROM a table of interest downloads it directly. Fine for small tables.
  3. LIST checks what stages already exist. A stage, in Snowflake, is a named place that holds data files on their way in or out of database tables. Think of it as the loading dock.
  4. CREATE TEMPORARY STAGE makes a new loading dock that Snowflake deletes when the session ends. Tidy. Mandiant notes the temporary stage vanishes with the session, which is a nuisance for anyone investigating later.
  5. COPY INTO that stage copies whole tables into files. The attacker set COMPRESSION=GZIP to shrink them, and a maximum file size of 5,368,709,120 bytes, which is five gibibytes per file. These were not people who expected small tables.
  6. GET downloads the files from the stage to the attacker's own machine.

And that is it. Read, package, download. Then, according to Mandiant, the extortion: contact the victim, demand payment, and advertise the data on criminal forums.

Why "not sophisticated" is the scary part

It is tempting to read "not sophisticated" as reassuring. Read it the other way. If a breach needs a clever exploit, you can patch the exploit. If a breach needs only a valid password and a feature that every customer uses, the only things between your data and a removalist truck are the controls around the login.

Mandiant lists three reasons the campaign succeeded, and they are worth quoting in full, because this is the core of the whole story:

The impacted accounts were not configured with multi-factor authentication enabled, meaning successful authentication only required a valid username and password.

Credentials identified in infostealer malware output were still valid, in some cases years after they were stolen, and had not been rotated or updated.

The impacted Snowflake customer instances did not have network allow lists in place to only allow access from trusted locations.

Three gates. All open. And here is the part that matters: any one of them, closed, would have stopped the login. A second factor the attacker did not have. A password changed at any point in the last four years. A rule that only lets in connections from the office or the company VPN. Mandiant says that in many cases the credentials had not been rotated "for as long as four years".

Part 4: Whose Lock Is It?

A Flemish counting house split by a painted line, where the owner polishes strong walls on one side, a tenant hangs his key by an open door on the other, and a clerk holds up a contract pointing at the line while both ignore him.
Fig. 3 — The contract was very clear about the line. Generated by an AI image model.

Every cloud contract has a line drawn through it. On one side is what the provider secures. On the other is what you, the customer, are responsible for. The industry calls this the shared responsibility model, which is a formal way of saying "we look after the building, you look after your key".

The US National Security Agency put it plainly in its Top Ten Cloud Security Mitigation Strategies in March, three months before UNC5537 had a public name. Strategy number one: "Security gaps arise when customers assume that the cloud service provider (CSP) is securing something that is actually the customer's responsibility." The exact line moves depending on the type of service. For software you just log in to (software as a service, or SaaS), the provider runs almost everything. Almost. Who has an account, how they prove who they are and where they can log in from stay with you.

A Bruegel-style snowy village where a thief in a fur hat opens a storehouse door with a brass key from a grubby notebook, porters carry chests to a sledge, a guard waves politely from the gatehouse, and an unused padlock in its wrapping hangs beside the door.
Fig. 4

Snowflake's case

Snowflake's position has been consistent since 30 May. The joint statement it published with CrowdStrike and Mandiant on 2 June lists the preliminary findings:

  • no evidence the activity "was caused by a vulnerability, misconfiguration, or breach of Snowflake's platform";
  • no evidence it was caused by compromised credentials of current or former Snowflake staff;
  • "this appears to be a targeted campaign directed at users with single-factor authentication";
  • the attackers used "credentials previously purchased or obtained through infostealing malware";
  • and one awkward exception, which we will come to.

Mandiant's report agrees on the main point: no evidence that the access "stemmed from a breach of Snowflake's enterprise environment". As far as the people doing the forensics can tell, the building was fine. The tenants' keys were stolen elsewhere and used at the front gate.

So the headline "Snowflake breach" is wrong in a specific way, and if you are briefing a board, say so. Your data was not taken because Snowflake's servers were broken into. It was taken because someone logged in as you.

Now look at the defaults

Here is where I part ways with anyone who stops at "customer's fault". A line in a contract is one thing. The defaults are what most people actually live with.

Snowflake's own documentation, as it stood in June, says: "By default, Snowflake allows users to connect to the service and internal stage from any computer or device." A network policy can restrict that, but only if an administrator creates one. And on MFA: "MFA is enabled on a per-user basis; however, at this time, users are not automatically enrolled in MFA. To use MFA, users must enroll themselves." The same page says Snowflake "strongly" recommends that, at a minimum, users with the top administrator role, ACCOUNTADMIN, be required to use MFA.

Read that as a tenant. The facility hands you a padlock and a fingerprint reader. The padlock is fitted. The fingerprint reader is in a box on the counter, each person who has a key must fit their own, and the gate lets in anyone who has the code, from any road. That is a perfectly defensible design for a product sold to experts. It is also a design where the safe outcome depends on hundreds of separate customers each doing the same three chores, and the campaign found the ones who had not.

Three stacked blocks. The provider secures the platform, its own staff and the controls it offers. The customer decides who has an account, turns on MFA, limits networks, changes passwords and manages contractors' laptops. Between them sit the defaults: any device can connect and each user must enrol in MFA themselves. Every case Mandiant traced began on the customer side.
Fig. 5 — The line was clear on paper. The defaults left it to each customer to draw. Diagram based on NSA, "Top Ten Cloud Security Mitigation Strategies", March 2024, and Snowflake's documentation as captured in June 2024.

Snowflake's own demo account

And then the awkward exception. In the same joint statement, Snowflake said it found evidence that the attacker had obtained personal credentials to, and accessed, demo accounts belonging to a former Snowflake employee. Snowflake says the accounts held no sensitive data and were not connected to its production or corporate systems. But it also says why the access was possible: "the demo account was not behind Okta or Multi-Factor Authentication (MFA), unlike Snowflake's corporate and production systems".

I do not raise this to score a point. Snowflake was open about it, and a demo account with no real data is not a breach of anything that matters. I raise it because it shows exactly how this happens. Snowflake's own production systems had single sign-on and MFA. A side account, created for a narrow purpose and outside the main controls, did not. Even the company that wrote the guidance had one account that slipped past it. If it can happen there, it can happen in your account list, too.

Where that leaves us

So, my position. The customers whose data was taken left three gates open, and closing any one of them was within their power and their contract. That is their part. Snowflake shipped a platform holding some of the most valuable data its customers own with the gates open by default and MFA left to each user, and as of 7 June it says it is "developing a plan to require our customers to implement advanced security controls, like multi-factor authentication (MFA) or network policies, especially for privileged Snowflake customer accounts". That is their part. Both can be true. The useful lesson for you is the same either way: do not wait for your vendor's defaults to change. Check the gates on every SaaS account that holds data you would hate to see on a forum. Mandiant expects UNC5537 to move on to "additional SaaS platforms in the near future".

Part 5: The Three Locks, Down to the Setting

Each of the three missing controls stops a different part of the attack. It is worth knowing exactly what each one does, because "turn on MFA" is advice everyone has heard and fewer people have checked.

Lock one: a second factor

You already use the idea behind MFA. Your bank card needs the card and the PIN. Someone who finds the card in the car park still cannot use it at the ATM. Two things of different kinds: something you have and something you know.

Multi-factor authentication applies that to a login. The password is something you know. The second factor is usually something you have (an app on your phone, a hardware key) or something you are (a fingerprint). I wrote about the basics a couple of years ago, and the reasoning has not changed. What a stealer takes from a browser is the password. It does not take your phone.

The evidence for how much this helps is unusually strong. In a 2023 study of Microsoft's Azure Active Directory accounts that showed suspicious activity, Microsoft researchers found that MFA reduces the risk of compromise by 99.22% across the whole population, and by 98.56% where the credentials had already leaked. That second number is the one for this story. These were accounts where the attacker already had the password, and a second factor still stopped almost all of them.7

In Snowflake, as of June 2024, MFA uses Duo. A user enrols through the web interface and installs the Duo Mobile app, and from then on a login needs the password and a confirmation from the Duo app on the phone. Because MFA is per user, "turning it on" is really a list: every human user, enrolled, checked.

There is a limit worth knowing. MFA protects the login. It does not protect a session that is already logged in. If a stealer takes the browser's session cookies, the small tokens that say "this person has already logged in", an attacker can sometimes replay them and skip the login entirely. That is session hijacking, and it is why the next two locks still matter after MFA is on. Phishing-resistant methods, such as the hardware keys and passkeys, close more of the remaining gaps, and the NSA guidance names "phishing-resistant multifactor authentication" specifically.

Lock two: change the key

The second gate is time. A stolen password is only useful while it still works. Mandiant found credentials that were still valid years after the infection, and in many cases not rotated "for as long as four years".

Credential rotation means replacing a password or key on a schedule, and immediately when there is any reason to suspect it. It is the dullest control in security, and it would have closed the door on the November 2020 credential at any point in the three and a half years that followed. Australia's ACSC put it first in its alert: "reset credentials for active accounts, disable non-active accounts". The second half matters just as much. An account for a contractor who left in 2022 is a key nobody is watching.

For machines and scripts, which cannot answer a phone prompt, the better answer is to stop using passwords at all. Snowflake supports key-pair authentication as an alternative to a username and password, using at least a 2048-bit RSA key pair. The client keeps the private key, and Snowflake stores only the public key, assigned to the user. Snowflake also supports rotating the public key. A pipeline that logs in with a key pair does not leave a password in someone's browser for a stealer to find.

Lock three: only let in the roads you know

The third gate is location. Your company's people and systems connect from a known set of places: the office network, the corporate VPN, the cloud servers that run your pipelines. UNC5537 connected from commercial VPN services and rented servers. Different roads entirely.

A network policy in Snowflake is a list of where connections may come from. Its allowed list says which requests may reach the service; its blocked list says which must not. Snowflake now recommends building those lists with network rules, which group addresses into named sets that a policy then allows or blocks. You can apply a network policy to the whole account, to a security integration, or to a single user, and the most specific policy wins: a policy on a user overrides the account's.

That precedence rule is useful and a little dangerous. Useful, because a service account that only ever runs from one cloud server can be pinned to that server's address, much tighter than the account. Dangerous, because a user-level policy that someone set up as a quick exception ("just let me in from home for a week") quietly overrides the account policy you thought protected everyone. When you check your network policies, check the user-level ones too.

The joint statement's recommended steps line up with these three locks almost exactly: "Enforce Multi-Factor Authentication on all accounts; Set up Network Policy Rules to only allow authorized users or only allow traffic from trusted locations (VPN, Cloud workload NAT, etc.); and Impacted organizations should reset and rotate Snowflake credentials."

Part 6: Marguerite's Laptop

Abstract steps are hard to picture, so meet one person. Marguerite is invented. She is not a client, and she is not anyone from the real investigations. She is a composite, built from the pattern Mandiant describes, so we can watch what each system records.

Marguerite is a freelance data engineer in Devonport. She helps three companies with their Snowflake accounts. At one of them, a homewares retailer, she has a user called MARGUERITE_ETL with a role that can read the customer tables. She saves the password in her browser, because she logs in most days.

A Saturday in 2021. Her teenage nephew borrows the laptop and installs a free copy of a game that normally costs $90. It works. It also runs a stealer, which spends about forty seconds copying every saved password and cookie in the browser and sending them away. Nothing visible happens. Marguerite never finds out.

2021 to 2024. Her log sits on a marketplace, then in a bigger compiled list, alongside hundreds of thousands of others. Her Snowflake password never changes. The retailer has no network policy, and MARGUERITE_ETL was never enrolled in MFA, because the job was only supposed to take a month and nobody got around to it.

An afternoon in April 2024. Someone searches the logs for snowflakecomputing.com, the address every Snowflake account shares. Marguerite's line comes up. They connect through a commercial VPN and log in. It works on the first try.

The same afternoon, an hour later. SHOW TABLES. A temporary stage. COPY INTO, compressed. GET. The customer table, the orders table, the loyalty table. The session ends, and the temporary stage disappears with it.

Late May. The retailer's head of IT gets an email from a stranger with a sample of its own customer table attached and a price.

What the logs recorded

Here is the part I want you to take away, because the evidence was there all along. Snowflake keeps an Account Usage schema of views that record activity in the account. Three of them tell this story.

  • LOGIN_HISTORY records every login attempt for the last 365 days. For that April afternoon, it shows MARGUERITE_ETL logging in successfully (IS_SUCCESS), from a CLIENT_IP that belongs to a VPN provider, not Devonport. FIRST_AUTHENTICATION_FACTOR shows a password. SECOND_AUTHENTICATION_FACTOR is NULL, which, in Snowflake's own words, is what you see without MFA. The view also records the client software, though the documentation warns that the reported client type "is not authenticated": the client says what it is, and Snowflake writes it down.
  • SESSIONS records each session, including a CLIENT_ENVIRONMENT field. Mandiant's hunting guide pulls the APPLICATION name out of it, which is where unfamiliar client software stands out. In one of Mandiant's example log entries, that field reads DBeaver_DBeaverUltimate.
  • QUERY_HISTORY records the queries themselves: the SHOW TABLES, the CREATE TEMPORARY STAGE, the COPY INTO with GZIP, and the GET.

Mandiant's hunting guide joins those three views together and then "stacks" the results: counts by user, by IP address and by application, so the odd one stands out. A user who has logged in from one Devonport address for two years and then appears from a VPN, running a client they have never used, copying whole tables to a new stage, stands out a long way.8

Nobody at the retailer looked. Not because they were careless people, but because nobody had been given the job. That is the gap that the last section is about.

What to Do This Week

A Bruegel-style winter market where villagers fit second padlocks, burn old keys and forge new ones, build a short fence to the village road, while a clerk logs every visitor beside a steaming cup of tea and a laptop.
Fig. 6 — Three locks and a ledger. Generated by an AI image model.

If you use Snowflake, start there. If you don't, do the same checks on whichever cloud service holds your customer data: the data warehouse, the CRM, the analytics platform, the support desk. The pattern is the same, and Mandiant expects it to spread.

  1. List every account that can log in. Every human, every service account, every contractor. In Snowflake, the USERS view in Account Usage is the starting point, and Mandiant's guide includes a full export query. Look for people who have left and projects that ended.
  2. Disable what nobody uses. If an account has not logged in for 90 days and nobody can say why it exists, turn it off. You can turn it back on if someone shouts.
  3. Check MFA per user, not per account. In Snowflake, query LOGIN_HISTORY for successful human logins where SECOND_AUTHENTICATION_FACTOR is NULL. Every name on that list is a single-lock door. Start with anyone holding ACCOUNTADMIN or another administrator role.
  4. Move service accounts off passwords. Use key-pair authentication (at least 2048-bit RSA) for pipelines, BI tools and scripts. Store the private key in a secrets manager, not in a config file on someone's laptop.
  5. Rotate every password that is older than your memory of setting it. Reset all passwords for remaining accounts now if you cannot prove they have not leaked. That is ACSC's advice too.
  6. Create a network policy. List the places your people and systems connect from (office, VPN, cloud workloads), build network rules for them, and apply a policy to the account. Then pin each service account to its own source with a user-level policy.
  7. Audit user-level network policies. Because the most specific policy wins, look for exceptions that quietly loosen the account policy.
  8. Run the hunting queries. Mandiant's guide covers the last 365 days. Look for logins from VPN providers, client applications you do not use, and CREATE TEMPORARY STAGE or COPY INTO followed by GET from users who do not normally do that. Snowflake's notice also links indicators of compromise to check against.
  9. Ask your contractors one question. "Is the laptop you use for our systems used for anything else?" If the answer is yes, give them a managed device, or a virtual desktop that never stores our passwords on their machine.
  10. Stop saving work passwords in the browser. Secureworks' advice is blunt: use strong, unique passwords "that must not be stored in web browsers". A dedicated password manager is a much harder target for a stealer than a browser's autofill, and it helps against credential stuffing too.
  11. Give the logs an owner. Pick a person, a schedule and an alert. A login from a new country, a new client or a new IP range for a privileged user should reach a human the same day.

None of this needs a new product. It needs someone to spend a day on settings that already exist.

Final Thoughts

Go back to the storage facility, and to Terry waving the truck through. Terry did his job. The gate checked the code, the camera recorded the truck, and the fence was never touched. The facility can say, accurately, that it was not broken into. The tenants can say, also accurately, that the second lock was sitting in a box on the counter the whole time. Both statements are true, and the rowing machine is still gone.

That is the Snowflake story as of the end of June. The platform held. The logins were real logins, with real passwords, stolen years ago by malware on laptops that had nothing to do with Snowflake, and used against accounts with one lock, an unchanged key and a gate open to every road. The one thing to take away is that a password on its own is no longer a lock. It is a key that has very probably been copied, and your other controls should assume so.

So this week, pick the cloud account that would hurt most on a forum. Find every user without MFA, every password older than your current phone, and every account that can log in from anywhere. Close one gate today. Then the other two.

Now, if you'll excuse me, I'm going to check whether my browser has been keeping any secrets from me. Then I'll make a cup of tea, because the kettle is the one appliance in this house with no saved passwords.

Notes

  1. The Register's report of 31 May carried Hudson Rock's claim that Snowflake had been breached through an employee's stolen login, and Snowflake's denial. Its 3 June update says Hudson Rock took the report offline following legal pressure from Snowflake. Mandiant's 10 June report found no evidence of a breach of Snowflake's enterprise environment. ↩

  2. "UNC" stands for "uncategorised". Mandiant uses it for a group it can track as one cluster of activity but has not yet tied to a known, named group. UNC5537 is, in effect, a case file number. ↩

  3. Secureworks also describes a preorder system that Russian Market added in October 2022. A buyer deposits $1,000 into the site's escrow, then requests credentials for a particular organisation's domain name or a particular application. Secureworks warned that this could turn opportunistic theft into targeting. ↩

  4. For scale, about 6,500 cups of tea in my house since November 2020, at roughly 1,820 a year. Each one was brewed with more care than those passwords received. ↩

  5. Mandiant's wider incident data puts stolen credentials as the fourth most common way into a network in 2023, the start of 10% of the intrusions it investigated. Not the most common way in, then, but one of the quietest. ↩

  6. Tools get two names in incident reports: the one the attackers give them, and the one the investigators use to track them. The attackers' name for this one is not worth printing, so here it is only FROSTBITE. ↩

  7. Microsoft's 2023 study of Azure Active Directory accounts found that dedicated authenticator apps did better than SMS codes, though both were far better than no second factor at all. ↩

  8. Snowflake's documentation says the LOGIN_HISTORY view can lag behind real time by up to 120 minutes. It is excellent for hunting. For an alert that wakes someone up, a two-hour delay is part of the design. ↩

end of article · 6,181 words · 27 June 2024

Jimmy Nixx, smiling, in round tortoiseshell glasses and a white T-shirt.

Jimmy Nixx

Jimmy is a technology consultant in Hobart, Tasmania, and runs NEOBADGER. He works where technology, regulation and the people organisations serve meet: AI harnesses, development, data and compliance. He writes one or two of these a month, powered by about 1,820 cups of tea a year.

The story

Further reading

3 more articles on Security.