Doomscroll M3: My Job Search Pipeline Now Emails Me Hopium

My Job Search Pipeline Emails Me Hopium Every Morning (and Roasts Me)

On October 12, my job tracker found five new jobs. Three of them were the same DevSecOps role at Vantor, posted three separate times. All three required an active TS/SCI clearance, which I don't have.

The pipeline was working perfectly. It was just perfectly useless.

That's where milestone 3 of Doomscroll started. Doomscroll is my serverless job tracker on AWS. Every morning at 6 AM Eastern, a Lambda pulls postings from USAJobs, Greenhouse, Lever, and Adzuna, filters them, and saves the new ones to DynamoDB. Everything is Terraform, and everything deploys through GitHub Actions with OIDC, so there's not a single long lived AWS key anywhere.

Milestones 1 and 2 built the plumbing. Milestone 3 had one job: close the loop. A tracker you have to go query isn't a tracker. It's a database you forget exists.

The funnel: 2,725 jobs in, 8 out

Before the new stuff, here's what the pipeline actually does every day. One run scans about 3,600 postings. SpaceX alone posts 2,725.

Of those 2,725 SpaceX jobs, 219 match one of my keywords. 85 get cut for level (senior, manager, principal). 126 get cut for location. 8 survive. That's 0.3%.

That's the whole point of the project. I'm not reading 3,600 postings a day. I'm reading 8.

Feature 1: A digest that only shows up when it has something to say

The digest is an SNS topic with my email subscribed. The fetch Lambda publishes only when it actually inserted new jobs, so there is never an email that says "0 new jobs." Nobody needs that kind of negativity at 6 AM.

The subject line is Your daily dose of hopium: 5 new matches (Oct 13). Duplicates get grouped by title and company, so Vantor shows up once as "(3 listings)" instead of three times.

One design decision I want to call out: if the SNS publish fails, the jobs are already saved. The next run won't think they're new, so that digest is gone forever. Instead of losing it silently, the handler records the failure and raises at the end of the run. That trips a CloudWatch alarm, which emails a separate alerts topic. Digests and alerts never share an inbox lane.

The SNS topics aren't encrypted, and that's on purpose. CloudWatch alarms can't publish to a topic encrypted with the AWS managed SNS key, and a customer managed KMS key costs $1 a month. On a $5 budget, that's 20% of my spend to protect public job listings. The checkov skip says exactly that, in writing.

Feature 2: A clearance filter that knows "active" from "able to obtain"

Here's the problem with defense and aerospace jobs. "Active TS/SCI required" means you need to already be cleared. "Must be able to obtain a TS/SCI" means they'll sponsor you. I want the second kind. A naive "TS/SCI" in description check throws away both.

So the filter reads every sentence that mentions TS/SCI:

USAJobs gets different treatment. Federal postings have a structured SecurityClearance field, and on a fed posting "Top Secret/SCI" usually means the agency adjudicates you after the offer. So USAJobs jobs never get dropped. They get tagged [TS/SCI sponsored] in the digest instead.

Confession: I shipped clearance.py to Lambda before anything imported it. It rode along in the zip for a while doing absolutely nothing. Then I found out my parsers were throwing away job descriptions, so even once it was wired in, it could only see titles. Fixing that meant teaching every parser to keep the description (Greenhouse only sends it with ?content=true), then stripping it before the DynamoDB write so I'm not paying to store essays.

First real run: 5 postings dropped for active TS/SCI across SpaceX, Palantir, and Adzuna. 21 test cases built from real posting language back it up.

Feature 3: An API with seven statuses and zero API keys

Updating a job's status is a PATCH /jobs/{job_id} on an API Gateway HTTP API. Auth is IAM, which means every request has to be signed with SigV4. There's no API key to leak. I sign requests with awscurl using my SSO session.

The statuses are named for how job hunting actually feels:

The README has the plain English column too, so a recruiter reading the repo knows Curved means rejected and not some kind of bug.

Every path got tested for real after deploy. An unsigned request gets a 403 from API Gateway, so the Lambda never even wakes up. A made up status gets a 400 that lists the valid ones. A job that doesn't exist gets a 404, because the update is conditional and can never create a job by accident. And a real update comes back 200 with the item and a fresh status_updated_at.

The AccessDenied that proved the setup works

The first deploy of the API failed:

ci-doomscroll-apply is not authorized to perform: apigateway:TagResource

My Terraform provider stamps default tags on every resource, and API Gateway stages need an explicit tagging permission to accept them. My CI apply role didn't have it, because I never gave it anything it didn't already need.

That's the point. The apply role started with only what milestone 1 needed. Every new AWS service hits a denial, and the fix goes into the bootstrap stack through a reviewed PR, applied from my laptop, because CI can't grant itself permissions. Least privilege isn't something you guess right on day one. It's a process, and an AccessDenied is that process working.

Testing an email you can't trigger

The digest only sends when there are new jobs. Every job was already in the table. So to test it, I had to delete a job and let the next run find it again.

First try: an Adzuna job. Deleted. Reran. Nothing. Adzuna only searches the last 7 days and the first few pages, so the job just didn't come back.

Second try: a USAJobs posting. Before deleting it, I checked. Posted October 6, outside the 7 day window. Also, "SUPERVISORY" was in the title, which my filters now exclude. It was a fossil from an older version of my filters.

Third try: SpaceX on Greenhouse. Greenhouse returns every open posting on the board, every time, no date window. Deleted it, reran, and got "inserted": 1. A minute later, the hopium landed.

Lesson: when you need a test to be repeatable, pick the source with stable output.

Then the pipeline roasted me

The job in that first real digest email was Structural Engineer, Launch Infrastructure (Starship).

It matched on infrastructure, which is in my keyword list for cloud infrastructure jobs. To SpaceX, infrastructure means launch pads.

And because I used that job ID to test the status API, the first application ever recorded in my tracker says I shot my shot at pouring concrete for Starship.

The fix was four exclude keywords: structural, mechanical, civil, facilities. "Cloud Infrastructure Engineer" still gets through, and Starship concrete doesn't. Then I marked the job Mid with a note: "Filter false positive."

I kept the story in the README on purpose. Keyword filters will always have edges. Pretending they don't would be the junior move.

One gotcha worth stealing

If aws sts get-caller-identity works but Terraform says InvalidGrantException, you're not crazy. The AWS CLI keeps its own cache of temporary role credentials, but Terraform's GMy Terraform provider stamps default tags on every resource, and API Gateway stages need an explicit tagging permission to accept them. My CI apply role didn't have it, because I never gave it anything it didn't already need.o SDK reads the SSO token cache directly. Your CLI can be coasting on leftover credentials while the actual SSO session is dead. aws sso logout, then aws sso login, fixes both.

What M3 actually shipped

• A daily digest that only emails when there's something new, with duplicates grouped
• A TS/SCI filter that tells "must already have" from "can get sponsored for"
• An IAM authenticated status API, tested on every response path
• A CloudWatch alarm on any fetch error and a $5 account budget
• 106 passing tests and checkov in CI, with a written reason for every skip

Next up is milestone 4: the architecture diagram, the final README, and seven straight days of clean 6 AM runs to prove it holds.

The code is on GitHub: [repo link]. I post a weekly build update on LinkedIn and YouTube. If you're building something similar, let's talk.