I think the clear reason is even with their unreliability, the cost of migrating off of GitHub for _most_ places is not worth it, and so companies don't / won't (yet)

This bugs me so much. Management decides what is reasonable cost of service and the employees have to deal with all bullshit coming from that product. And you can't really escape it. Maybe the real moat was enterprise deals we made along the way.

Could be - I’d also argue the time cost to the engineers to switch is pretty significant too, regardless of the monetary cost

As a bit of a counterpoint, I help maintain SDKs for Hatchet in multiple languages (largely Python and TS, but also contribute to Go a bit too) that benefit heavily from generics. It's especially useful for things where users of the SDK provide e.g. return types from functions they register, and we want those to be strongly-typed elsewhere in the codebase. A simple Python example is:

  @hatchet.task()
  async def my_task(...) -> SomeOutputType:
    return SomeOutputType(...)

  ## imagine this is an API handler:
  @api_handler("/some/path")
  async def handle() -> ...:
    result = await my_task.run(...)

    # we now know `result` is of type `SomeOutputType` without any sort of type assertion, etc.

Admittedly, I'm not a Go expert, nor am I a programming languages expert. But I do feel that this type of behavior is really only possible (with nice ergonomics) with generics, and it's always been upsetting to me that somehow Python's type system feels more complete than Go's in this arena, or at least it has until more recently.

Maybe this falls into the 1% of cases, but I'd suspect this sort of thing is more common than that.

Edit: I should have mentioned - in the Python example above, `@hatchet.task` is generic with the output type of the task it wraps.

And I'm not a python expert, so I might be wrong here, but:

My understanding is that this decorator generates some class in the background that wraps the function into some remotely executable container thing, and handles the networking?

Since python is a dynamic language, and go is not, this would be impossible without codegen, generics or not, but go does have codegen facilities.

And the more immediate implication, that many OOP languages do (and seems to be going on in here) is that they handle asynchrony via some generic Awaitable[T] pattern.

Go does not do this, generally the way you handle asynchrony and abstract typed results is by using channels. You don't await on 'smart' objects, you read from channels (which are 'generic' in a way, but they are the few exceptions where go used to allow this behavior).

Ah yep! I should’ve given a Go example. In Python we use decorators because it’s what’s in fashion, but in Go and TypeScript you just create a `NewTask()` e.g. where you pass a function (as well as other args), and it returns something with e.g. a `Result` method. And that result method, which is generic, is the thing that’s really nice to have typed.

Agreed codegen works fine here too by the way, but it feels kind of clunky to me (it’s something I don’t love about Go, although I know it’s also an important part of the ecosystem and is popular).

I'm just saying that this Task[T] pattern is completely alien to Go. If you want to have one-off ansynchrony, then just write it synchronously and start your work on a goroutine.

If you wanna do batch processing, use channels.

For all intents and purposes, I would say 90% of IRL usage of generics is either this Task[T] pattern, collections or map() style array processing functions.

Go had a solution for all 3 of these, that didn't involve generics. I'm in a fortunate position that I get to pick the language I work in for a lot of my work (from a reasonable selection). So if I want to write Go, I'll rather do it idiomatically, otherwise I'd pick something else. Which should be the case for everyone, and language designers should take heed. The 'we want the Java/Python/Go/JS audience' sentiment has ruined many a language, as they've turned themselves into the same mediocre language that has all the features everyone else has.

For what it's worth, Work at a Startup ostensibly does give you something like this on the hiring side, although it's been far from perfect in my experience. I suspect other "platforms" for job searching might have similar, but it'd be tougher in a dedicated ATS like Greenhouse or Lever or some such, since I imagine there are data privacy laws that limit their ability to "enrich" candidate data across companies, which LinkedIn, WaaS, etc. can bypass by having you make a profile

+1 to this - I've griped pretty often that FastAPI's documentation implicitly recommends this (https://fastapi.tiangolo.com/tutorial/sql-databases/#create-...) by suggesting using dependency injection to manage database connections, only to start seeing connection pool exhausted errors as soon as the number of concurrent requests exceeds the number of allowed connections.

FastAPI pattern works very well with Pgbouncer, when it is in transaction pool mode.

Your Python application maintains a connection to Pgbouncer during the lifecycle of the request, but the physical Postgres connection is allocated only during the DB transaction. You will need open/close transactions in your code though.

This is why I said PgBouncer is a sign of something being wrong. Devs aren't managing connections right, they try to paper over it with PgBouncer, it's not really easier cause they now need to be conscious of xacts instead, and now there's an extra moving part in the DB that most of the team doesn't really understand. PgBouncer has its other uses, but I really don't like this one.

I also get it, xact should be 1:1 with connection in a lot of these backend applications. Sometimes I have a few little helpers for that, like pool.sql() will take conn, open xact, execute, close xact, return conn. If the DB driver doesn't already have that.

Oh wow. Dep injection for DB connections is nasty.

I might be outing myself as a noob here, but... what is the (better) alternative?

You inject the pool itself.

Sorry, I am being dense... how does that solve the problem?

I still have to get a connection from the pool, I just do it inside the function body now, right?

So this

    @app.get("/users")
    def get_users(conn = Depends[get_db_conn]):
        users = conn.execute("SELECT * FROM users")
        return users

would become that instead:

    @app.get("/users")
    def get_users(pool = Depends[get_db_pool]):
        with pool.get_conn() as conn:
            users = conn.execute("SELECT * FROM users")
        return users

But I still need enough connections in the pool to handle all concurrent requests, no?

The idea is you only take a connection from the pool when you need to touch the DB, then you give it back immediately. It's very possible that's only a small fraction of the time spent in some handlers. If you inject the connection, you always hold it through the entire request.

No - you do not always give it back immediately in many cases as you have a transaction, which cannot "change hands". If a write connection makes consecutive updates to the DB, you must see it through before closing.

I meant you give it back immediately when you're done with it. So usually after you commit, unless you want to hold it longer for some special reasons.

ahh, gotcha! thanks!

(Matt from Hatchet - Hi Ulysse :wave:)

I, at least, don't know of a perfect fix here. Re: the original comment - Postgres will also error on deadlocks after it detects them without setting your isolation level to Serializable, but I agree with you that often retrying doesn't help, and could even cause cascading / snowballing failures if you have a backlog of retries piling up because of deadlocks.

I don't know if there's a good solution, really. We've fixed deadlocks incrementally over time as we've found them, which has worked pretty well, but of course that means also needing to deal with the "finding" part, which has generally come in the form of lots of `deadlock detected` log lines and errors (and retries accompanying those).

One thing that might be worth auditing is why there are two different bits of application code that are updating the same rows in two different tables in different orders. I know it's a contrived example, but it seems like it could be a code smell to me. Maybe this is the kind of thing that arises when two different subteams are working on the same database and are largely siloed.

Alexander will likely have more thoughts here as well, just my two cents!

(Matt from Hatchet)

One small addendum here is we've had a lot of success performing joins in memory in a few very specific situations where the alternative is a single, often overcomplicated query. I've heard / seen advice many times in the past about performing fewer round trips to the database being something to optimize for (often good advice!). Sometimes this is taken too far, resulting in overly-complex queries requiring complicated JOIN or UNION logic, CASE logic, and so on.

We have a couple of places in our codebase where we perform two or more simpler queries independently instead, and then loop through their results and use maps to match the relevant rows. Conventional wisdom often suggests this path will hurt performance because of the extra database round trip in addition to the loops needed to perform the join, but it is actually beneficial in these cases because of more predictable query planning behavior. We use this trick sparingly, but it can be helpful in a pinch.

Note that some ORMs will also do this for you in the background, which we don't necessarily endorse, and we try to use this sparingly when writing a single query on its own is not realistic.

This kind of advice is very dependent on the scenario.

If you are doing some kind of full cross product where the join creates a much larger set of rows, it could optimize the DB load and network traffic to fetch the source sets and then generate the permuted set locally.

But, many inner join patterns are selective. They produce a much smaller output than the source records. The traffic to pull all the records and then intersect and filter locally is much worse than having the DB do it.

And that's before you even consider indexed joins, where the query plann is able to make good use of indexes to avoid doing brute-force table scans, sorting, and filtering.

Thanks! I should have clarified - we haven't been using this pattern for selective joins. Strongly agreed that pulling down extra data into memory and then doing the filtering doesn't make much sense. We've found it useful in the case where it's hard to write a query where the planner _does_ make good decisions because of the complexity of the join conditions (e.g. joins using cases, a boolean "or", or something similar).

Also, to re-emphasize: we do this rarely, but it's been helpful the times we've done it

Noted that you only use this sparingly but you could try :-

* Materialised views (especially if the computation/joins are particularly nasty).

* Left outer joins are a good alternative to 'joins using case' and more likely to use the index.

Indeed! Materialized views won't work here as PG doesn't support "always updated" / "auto-refreshing" materialized views natively (although you can get something similar with extensions like TimescaleDB). Left joins are the thing we're often trying to avoid though, especially when the join conditions are involved, as we've seen the planner just make poor decisions in the past at unpredictable times. That's exactly the sort of situation where you might reach for this sort of trick

If left joins are causing issues, maybe for a quick win try increasing the sample size on the column(s) involved e.g. ALTER TABLE tablename ALTER COLUMN columnname SET STATISTICS 1000 - (default I think is 100), remember to run 'ANALYZE'.

I feel like i've heard of people using views for this as well. Like setting up two views and then joining across them because of the complexity of doing it all in one query. I could be wrong though.

Gift link, if you'd like to use it: https://www.nytimes.com/2026/07/13/style/backyard-baseball-v...

Thank you, that worked. What a massively wholesome story.

agreed :)

I was just going to post this, and searched first and landed here. Not much to say beyond that reading this piece made me so happy, and brought back tons of nostalgia for days I'd long forgotten.

Hello! Hatchet engineer here - just thought I'd drop a link to some discussion on this topic from last year, which is here: https://news.ycombinator.com/item?id=43574767

Happy to answer more questions beyond this!

I've been using Elixir, which has been wonderful, mostly because of how amazing the built in `Enum` library is for working on lists and maps (since the majority of AoC problems are list / map processing problems, at least for the first while)

Enum really does feel like a superpower sometimes. I’ll knock out some loop and then spend a few mins with h Enum.<tab> and realise it could’ve been one or two Enum functions.