So here’s how we learned that 500 million tiny files will humble every “proper” engineering approach you can think of.

It started innocently enough. We needed to backfill our data warehouse as part of our new Ingestion Pipeline. Migrate all our historical consumption data from the old S3 bucket format to our shiny new warehouse ready structure. How hard could it be?

We had two buckets to deal with:

• datamover bucket: 50M files (11M actually had data)

• raw-consumptions bucket: 500M files across ~1.7M users

The datamover bucket was going to be our practice round. Get the approach right on 11M files, then scale it up to 500M. Classic engineering, start small, iterate, then scale.

Spoiler alert: sometimes “scale it up” breaks everything you thought you knew.

Act I: The Proper Engineering Approach

“Let’s do this right. Database indexes, sequential processing, proper state management.”

We built what any engineer would build for a data migration:

create table migration_file_index
(
    id              bigserial primary key,
    s3_key          text not null unique,
    platform        text not null,
    job_enqueued_at timestamp with time zone not null,
    file_size       bigint not null,
    processed       boolean default false,
    created_at      timestamp with time zone default now()
);

With proper indexes, of course, we are professionals:

create index idx_migration_sort
    on migration_file_index (platform, job_enqueued_at)
    where (NOT processed);

create index idx_migration_resume
    on migration_file_index (processed, id);

create index idx_migration_platform_stats
    on migration_file_index (platform, processed);

The architecture was clean:

• DatabaseIndexer: Catalog all S3 files that had data into Postgres

• SequentialProcessor: Process files in chronological order

• PartitionBuffer: Smart batching for optimal warehouse file sizes

It felt good. It felt right. This is how you handle a large-scale data migration.

Then we ran it on our practice dataset.

11 million files took one week.

For the datamover migration, a week seemed totally reasonable. Less than a sprint! Exactly what product wants to hear. “Yeah, we’ll have the historical data migrated by next Friday.” Perfect.

Let me repeat that for emphasis: 11 million files. One week.

I remember staring at the progress logs, doing the math in my head. If 11M files = 1 week, then 500M files = ... reaches for calculator ... 45 weeks.

Forty-five weeks. Almost a year. To backfill the raw-consumptions bucket historical data.

That’s when we realized that all our careful database design, our proper indexes, our sequential processing - it was all overhead. We were spending more time managing the migration state than actually migrating data.

Act II: The Cloud-Native Pivot

“Surely AWS can handle what we can’t, right?”

Time for a complete rethink. Why are we trying to manage 500M individual files when AWS has services built exactly for this kind of massive parallel processing?

Enter the Athena approach:

1. Create an external table mapping to our raw-consumptions bucket

2. Let Athena discover and process all 500M files in a single massive query

3. Output everything to a staging bucket in warehouse format

4. Do a controlled cross-bucket migration to production

The promise was beautiful:

• 2-6 hours instead of months

• ~$100-200 in Athena costs vs. massive compute bills

• Leverage cloud services for what they’re actually built for

// Instead of this:
for (const file of 500_000_000_files) {
  await processFileSequentially(file); // 🤮
}

// We’d do this:
CREATE TABLE warehouse_data AS
SELECT 
  platform,
  user_id,
  payload,
  ingested_at,
  date_partition,
  hour_partition
FROM external_consumptions_table;
// Let Athena figure it out in parallel!

We built comprehensive tooling around it:

• Quality validation (95% success rate required)

• Cross-bucket migration with configurable batch sizes

• Progress monitoring and error recovery

• Staging-to-production promotion pipeline

It was elegant. It was cloud-native. It was exactly the kind of solution you’d present at a conference lol.

And Athena choked.

Turns out even AWS services have limits when you point them at half a billion tiny files. Queries would time out consistently. Tuns out 500M files is just too many god damn files!!

The cloud-native dream tried hard, but it died.

Act III: The Big Machine Epiphany

“You know what? Let’s just use a really big machine.”

Sometimes the most sophisticated solution is admitting when sophistication isn’t working.

We spun up an i4i.metal instance:

• 128 vCPUs

• 1TB of RAM

• 30TB of NVMe storage

• 75 Gbps network

The plan was beautifully simple:

1. Download everything to local NVMe storage

2. Process it with maximum parallelism

3. Upload the results

No databases to maintain. No cloud service limits to hit. No complex state management. Just raw compute power and storage bandwidth.

Step 1: Setup the Beast

# Create RAID 0 across all NVMe drives
NVME_DEVICES=$(lsblk -nd -o NAME | grep nvme | grep -v nvme0 | sed ‘s/^/\\/dev\\//’ | tr ‘\\n’ ‘ ‘)
sudo mdadm --create /dev/md0 --level=0 --raid-devices=$NVME_COUNT $NVME_DEVICES
sudo mkfs.ext4 -F /dev/md0
sudo mount /dev/md0 /mnt/nvme

Step 2: Download Everything

# s5cmd with 4096 workers to max out the 75 Gbps network
s5cmd --numworkers 4096 cp “s3://consumptions-raw/*” /mnt/nvme/raw-data/

Enter s5cmd - if you haven’t used it, it’s like AWS CLI’s faster, more parallel cousin. While aws s3 cp might give you decent performance, s5cmd is built specifically for high-throughput S3 operations. It can spin up thousands of concurrent workers and actually saturate high-bandwidth connections.

With our 75 Gbps network and s5cmd cranked up to 4096 workers, we could download at speeds that would make your standard S3 sync weep.

Step 3: The Organize Breakthrough

This is where it got interesting. We needed to transform the data from:

platform/user123/2024-01-15T14:30:15.json
platform/user123/2024-01-15T14:45:22.json  
platform/user456/2024-01-15T14:32:18.json

To warehouse partitions:

platform=spotify/date=2024-01-15/hour=14/part1.gz

Our first approach was to process files sequentially, organizing by timestamp. Still too slow.

Then came the insight: “Wait, we have ~1.7M users here. What if we parallelize by user instead of by file?”

# Instead of processing 500M files sequentially:
for file in $(find . -name “*.json”); do
    organize_by_timestamp $file  # Still 500M operations!
done

# Process ~1.7M users in parallel:
find raw-data/ -maxdepth 2 -type d -name “user_*” | \\
    parallel -j 128 ./[organize-single-user.sh](<http://organize-single-user.sh>) {} /mnt/nvme/partitions

Each worker would:

1. Take a user directory

2. Parse all their timestamp.json files

3. Route each file to the correct date/hour partition

4. Coordinate at the partition level to merge everything

This reframed the problem completely. Instead of “how do we process 500M files?” it became “how do we process 1.7M users in parallel?”

One is hard. The other is Tuesday.

Step 4: Process Partitions

Now each partition could be processed independently:

# Each partition just worries about:
# 1. Collecting all files for that date/hour
# 2. Gzipping them efficiently
# 3. Creating warehouse-format records
find /mnt/nvme/partitions -name “file_list.txt” | \\
    parallel -j 128 --progress ./[process-single-partition.sh](<http://process-single-partition.sh>)

Step 5: Upload and Snowpipe Magic

# Upload to partitioned S3 structure
s5cmd --numworkers 4096 cp /mnt/nvme/output/* s3://raw-extractions-prod/data/

And here’s the beautiful part: once the files land in S3 with the correct partitioned structure (platform=spotify/date=2024-01-15/hour=14/), S3 notifications automatically trigger Snowpipe to ingest them into our data warehouse. No manual intervention. No complex orchestration. Just files appearing in the right place, and the rest happens automatically.

It worked. Fast.

And here’s the kicker: the entire 500M file migration took about a week. The same time it took our initial database approach to handle 11M files.

Let that sink in for a moment. We went from processing 11M files in a week to processing 500M files in a week. That’s a 45x improvement in efficiency, not just from throwing more hardware at the problem, but from completely reframing how we thought about it.

What We Actually Learned

About Scale

500 million files isn’t just “a lot of files.” It’s a scalability problem that broke our initial assumptions and iterations.

The overhead of “doing it right” - indexes, state management, proper abstractions - can exceed the actual work when you hit extreme scale. Sometimes the most elegant solution is the one that avoids the problem entirely.

About Problem Reframing

The breakthrough wasn’t better algorithms or more sophisticated architecture. It was changing the unit of work:

• ❌ “How do we process 500M files efficiently?”

• ✅ “How do we process 1.7M users in parallel?”

Sometimes the bottleneck isn’t compute power, it’s how you think about the problem.

About Cloud Services

AWS services are incredibly powerful, but they have limits. You just have to test them!

“Cloud-native” isn’t automatically better than “local processing with a big machine.” The right tool depends on your specific constraints and scale.

About Engineering Pragmatism

The best architecture is the one that ships. There’s real elegance in recognizing when your careful abstractions are getting in the way and just solving the actual problem.

Sometimes “just use a bigger machine” isn’t lazy engineering, sometimes it’s the sophisticated approach.

The Real Lesson

This isn’t a story about finding the “right” solution. It’s about recognizing when your assumptions break down and being willing to completely change approaches.

• Approach 1 failed because we tried to apply normal database patterns to abnormal scale

• Approach 2 failed because we assumed cloud services could handle anything

• Approach 3 succeeded because we stopped trying to be clever and just solved the problem

The most elegant engineering isn’t always in the architecture - sometimes it’s in knowing when to stop being architectural.

Now our data warehouse hums along, ingesting new files automatically via Snowpipe, and our 500 million historical files are exactly where they need to be. Sometimes brute force is the sophisticated solution.

Want to work on problems like this? We’re hiring widely at Koodos. If this kind of scale, reframing, and pragmatism excites you, drop us a line at team@koodos.com.

In this post we will dive deeper and demystify how apps actually implement authentication. Do it right, and you barely notice it. But do it wrong, and you lock users out or open major security holes.

We'll see how apps authenticate you, from old school passwords to slick new standards like WebAuthn, and we’ll get our hands dirty discussing security and usability trade-offs, encryption, individual sovereignty, and more!

By the end, you'll have a solid grasp of how real-world apps tackle authentication using the methods we've covered.

Let's get started!

Authentication 🪪

The first thing you do when you log into an app is identify yourself.

This might be using some kind of identifier — typically an email address or phone number — or logging in through Google, Facebook, or some other third party service, and using their knowledge of your identity instead.

Taking that identification and ensuring that it is yours is the process of authentication1. Once you’re authenticated, you can use the app!

In the earlier post we explained that authentication can be achieved in one of three different ways. You can demonstrate that you:

• know something

• own something, or

• are something

Only the person you're identifying as would know or own this information, or have that identity.

All modern apps use a technique that boils down to one of these three methods to authenticate you.

So let’s look at each in turn!

Knowledge-based authentication 🧠

Password-based authentication uses the first of these authentication methods — it verifies that you know something only the user you're logging in as would know: their password, pin, passphrase, or any other secret shared between the user and the service.

To prevent attackers being able to access your account in case an app’s authentication data is leaked through a hack, the app doesn’t actually store the password, but a hashed2 version of it.

Roughly speaking, the hash of your password is like an encrypted version of your password, except that given your password’s hash, it’s very hard (near-impossible, if hashed correctly) to retrieve the original password3.

Limitations of passwords

Implementing password-based authentication yourself is fraught with difficulty.

Some of the “fun” challenges you can anticipate are:

• Properly implementing passwords — You need to manage salting, hashing, encryption and then store them securely in your database.

• Password recovery flows — You’ll also need to implement email-based password recovery. But what if your password-recovery emails go to spam? Having angry users contact customer support because they can't log in is not fun.

• Handle "password1@3" — You'll quickly realise that people like to use weak and easy-to-guess passwords.

  This could allow hackers to find a password through a dictionary attack4 — something they can do if they get access to a leaked version of the database, for example — before later gaining access to the running system.

• Block bots — Users also really like to reuse passwords5, so you might have hackers attacking your system because of a data breach on a completely different website. Blocking these hackers when they’re attacking your site from many different IP addresses is a major challenge6.

• Additional friction - There will be quite a few users who won't have the patience to go through the password-recovery flows after forgetting their password. Instead, they just give up and go do something else. Forcing your user to remember something to log-in can add additional friction and increase churn.

• And much more7

Each of these steps is hard and requires scarce engineering resources, which is why authentication providers such as Supabase, Auth0 and Clerk offer managed versions, at a cost.

But most significantly, users may forget their password and never even use your site in the first place!

To summarise: passwords are a poor authentication mechanism. They’re hard to implement, unsafe, and lead to a poor user experience!

So: is there a way to authenticate users without relying on something that only they would know?

I wouldn’t still be writing if there wasn’t, so let’s get into that!

Ownership-based authentication 🔒

Ownership-based authentication is our second authentication method — it involves verifying that you own something that only the user you are attempting to log in as would own, such as their email inbox or phone number.

One-time passwords

For instance, when you log in, an app might send you a one-time password (OTP8) via email. This is typically a random short, server-generated, alphanumeric code like ZXY1, but it can also be a magic link, which is a URL containing the OTP as part of it.

The OTP is a temporary secret that is only shared between the app and the user's email inbox, is valid for that specific interaction, and is invalidated once used.

If you log in using this OTP, then you’ve successfully demonstrated ownership of the email address connected to the user account, and so you must be that user — you’re successfully authenticated!

SMS-based logins can also be used, but SMS are not encrypted and can easily be discovered by anyone on the network by sniffing for text messages9.

SMS-based logins can also be gamed for profit — a big reason why many apps no longer solely rely on SMS for authentication, or even only offer SMS based authentication for premium or paying users.

But worst of all, SMS and emails may take minutes to be delivered, or may never be delivered at all10!

Is there a way for the user and service to coordinate on a shared, time-based secret, without sending it over an insecure, slow and unreliable communication channel?

I haven’t stopped writing, so yes!

Time-based one-time passwords

Authentication apps, such as Google Authenticator, provide time-based OTPs (TOTP11).

These authentication apps are set up well in advance of logging in, typically during the first onboarding session. Instead of using SMS or email to send the OTP to the user, an app on the user’s device generates a TOTP using the current time and a secret shared between the user and the web app.

This is much safer than SMS or email OTP delivery, as it does not rely on the security of external communication channels. The TOTP is generated locally on a device the user possesses and controls, making phishing and interception attacks much more difficult.

Hardware authenticators

Finally, hardware security keys, such as a YubiKey12, are a physical authentication device that a user can carry around with them. These are often used in high-security environments, such as in an enterprise, and can also create OTPs. These are really powerful devices, and we’ll dig into them more soon.

Whilst ownership-based authentication methods are often more secure, easier to use (no need to remember a password!), and easier to implement than passwords, they still require the user to do something cumbersome to authenticate.

This is an opportunity to switch device, get distracted, churn, and never use your app at all!

For this reason, authentication methods that require the user to switch device are used sparingly, only for sensitive operations such as when logging in on a new device for the first time.

So — can we authenticate a user without requiring them to use two devices? Absolutely!

WebAuthn 🕸️

WebAuthn13 is an awesome, new(ish) protocol that allows users to authenticate themselves whilst browsing the web.

Instead of using emails, passwords, YubiKeys, or OTPs, the information needed to authenticate a user is stored in an authenticator — typically the browser or device.

Public-key cryptography

The method used for authentication is public key cryptography. It’s the same technology used in Transport Layer Security (TLS) — a networking protocol which is used for nearly all traffic on the internet. It’s fundamental to security, so let’s take a quick aside to dig in:

The idea behind public-key cryptography, or asymmetric encryption, is pretty simple: each person has two keys — a public and private key. The public key is shared with everyone, and the private key is kept to themselves.

Seems pretty simple so far. But we can now do two magical things:

• Anyone can send the user a message by encrypting it with the user’s public key. Only the owner of the private key — the user — is able to decrypt that message.

• The user can use their private key to sign a message. Anyone with the public key can verify that that user signed the message, and no one other than the user is able to sign it as that user.

Public-key cryptography in WebAuthn

The browser or device, acting as an authenticator, generates a new public/private key pair as the user onboards onto a web service. The private key is kept hidden, potentially stored securely in a hardware security module (HSM14) on the device, and the public key is shared with the web service.

When the user later wishes to log in, the website asks the browser to sign a specific message. The browser or device does so using the private key, the website verifies it is indeed the same user using the previously shared public key, and the user is now authenticated!

https://webauthn.io has an awesome demo of this — try it out!

With WebAuthn, the user doesn’t need to type their username or remember a password and can login with a single click — very impressive!

And excitingly, the prompt to log in with WebAuthn will only appear on the correct site, thereby preventing the phishing attacks that were possible with passwords 🎉.

But logging in with a single click also makes it easy for friends, enemies, and attackers to log in with a single click, if they got close enough to your machine.

So, can we ensure it’s you actually doing the clicking? Surely!

Multi-factor authentication

The HSM can actually prompt the browser for something before it allows access to cryptographic operations, such as generating a public and private key or signing a message. And the browser can prompt the user for this data in turn.

The prompt can ask for a password — another example of knowledge-based authentication — or, even better, ask for biometrics, such as your fingerprint or face. Laptops and mobile phones commonly support fingerprint readers, so this is a fast, easy, and secure15 way for users to authenticate.

Clicking a button to sign in and using your fingerprint is multi-factor authentication, since it requires two forms of authentication for you to log in: ownership of the device with the HSM and your inherent identity and ownership of your fingerprint.

And because you attest to your own identity, this is a form of self-sovereign identity — we’ll talk about this more in the next post.

Cross-device authentication

WebAuthn does pose a problem, though.

Users typically use a website on multiple devices: their phone, tablet, and laptop. It’s easy to share a password between devices — does WebAuthn offer something similar?

Yes! Well, sort of.

You can delegate the authentication to a roaming device — which could be your phone, laptop, or YubiKey — anything that is accessible via bluetooth, NFC, or USB. In turn, that device can sign the message on behalf of the main device.

This is an additional layer of protection, since malware on a site would need to infect an entirely different device than the main one you’re using to steal your private key — neat!

However, this means that we’d always be reliant on that first device for logging into that service, or we would need to set up a private key on each device we want to log in from — hardly a good user experience, and potentially worse than multi-factor login using passwords and OTPs.

Limitations of WebAuthn

WebAuthn isn’t a panacea, and does come with some notable downsides:

• Single point of failure — If you lose a device, you’d lose access to all of your logins that that device stores the private keys for. Recreating those logins would take ages, and would be a massive pain!

  Passkeys16 are an exciting new feature to help plug this gap and to share private keys across devices, allowing for ease-of-access and easy recovery. The fun part is that Google17 and Apple’s iCloud would sync these private keys for you — it’s built into the platform!

  As excited as I am about Passkeys, they are new and controversial1819, and it’s unclear if they’ll achieve mass adoption.

• Inflexibility — Passwords are flexible enough to support most flows that users need20. In contrast, WebAuthn only supports a subset of flows.

  For example, sharing access to a single account (eg to access the bills of an energy provider) with a friend is straight-forward with passwords, but much tricker with WebAuthn.

• Uncertain adoption — Perhaps most important is that users aren’t familiar with the passwordless world.

  Will they onboard? Will they log in again on a new device? Will they understand the user flow? We don’t know yet — and it’s a risk to try out something as innovative as this.

Whilst I really want a passwordless future, and think WebAuthn and Passkeys could be our saviour, there’s one more form of authentication that is widely used and worth learning about.

Identity-based authentication 🙋

The third way we can authenticate is to use something we are — our identity. Biometric authentication is one example of this, and we briefly covered it in the last section.

But we can also implement this by outsourcing identity verification to a third party that users already have an account with — an identity provider21.

When you see “Login with Google” or “Login with Facebook”, that's identity-based authentication in action.

The identity provider tells the app that you are indeed the person you claim to be by providing it with a signed ID token.

When the app you’re signing into gets this token, it uses the signature to verify that the identity provider did indeed create that token — and now you’re fully authenticated!

OpenID Connect

OpenID Connect (OIDC22) is the underlying protocol used for this flow. OIDC is a layer on top of OAuth 2.0 — a protocol used for access delegation23 — that standardises identity data like your name and email address.

Big tech giants like Google, Facebook, Twitter, and Microsoft all offer identity services that implement OIDC. It's easy and convenient for users since they can leverage their existing accounts.

And it saves a ton of headaches for developers since they no longer need to build secure authentication systems. Instead of authenticating a user yourself, just plug into Google, and you're done!

This is an example of federated identity24, where a user’s identity is used across multiple systems — Google, Facebook, and all the apps that rely on these identity providers — instead of just one.

Limitations of identity federation

However, identity federation has a fundamental flaw: the identity provider can revoke access at any time, instantly locking you out of all sites. You don’t own your identity — it’s stored on Google’s servers, not yours.

You may think it’s rare, but it’s surprisingly common2526 to find posts on Hacker News of users being denied access to their Google or Facebook accounts due to one reason or another, and totally unable to access any service that they used Google or Facebook as an identity provider for.

The quest for true user-owned identity is an active area of innovation which I'll cover in an upcoming post! Subscribe now to be updated when it’s released 👇

I would love to hear folks thoughts on this — feel free to comment below or DM me!

Have a great week all — stay tuned for Part 3!