Amusingly that links to https://peps.python.org/pep-0206/ from 14th July 2000 (25 years ago!) which, even back then, described the cgi package as "designed poorly and are now near-impossible to fix".
That fails pretty hard at providing a rationale. Basically it says that CGI is an inefficient interface because it involves creating a new process! Even if that were true, "You shouldn't want to do such an inefficient thing" is very, very rarely a reasonable answer to a technical question like "How do I write a CGI script in Python?" or "How do I parse a CSV file in Python?"
There are certainly some suboptimal design choices in the cgi module's calling interface, things you did a much better job of in Django, but what made them "near-impossible to fix" was that at the time everyone reading and writing PEPs considered backwards compatibility to be not a bad thing, or even a mildly good thing, but an essential thing that was worth putting up with pain for. Fixing a badly designed interface is easy if you know what it should look like and aren't constrained by backwards compatibility.
It would have been a less strange argument 25 years ago, before the manycore era, when using Python involved less of a performance sacrifice. And there are still cases where Python is acceptably performant. However, the argument is from only 6 years ago, which makes it ridiculous.
Python is still a 10x or more performance sacrifice for anything that's actually CPU throughput limited.
Or, alternatively, your VM hosting cost will be 10x larger on Python, than something top of the line, if your workload is CPU throughput limited.
Whether you're actually CPU limited, and whether VM hosting costs is your largest cost, is a totally different question :-)
Typically, computation you do in Python code costs you about 40× as much CPU as if you did it in C. But with Numpy I usually see only about a 4× single-core slowdown after a little optimization work. Many database-backed web services are bottlenecked on the database or text template instantiation, neither of which are really related to Python's CPU efficiency.
As a side note, the most popular databases are getting only a tiny fraction of the available performance on current hardware. I wrote a couple of comments with more details about this a week ago: https://news.ycombinator.com/item?id=44408654
In the manycore world, Python's GIL makes some approaches to scaling across cores unavailable, though that is changing. But I don't think those are usually relevant to web server throughput, just (potentially) latency.
I work with Python quite a bit. Basically you either world where something like a web request takes 40ms to process instead of 20 and that just doesn't matter or you are in the situation where the request takes 2000ms instead of 500ms which is not as acceptable (but might be depending on the UI on top of this web request). At that point your first stop is something like numpy or another C or Rust module that will do the brunt of the CPU-intensive work. Past that, yeah you gotta look at different runtimes. But Python is so fast for prototyping that it might not even be worth it.
I haven't tried it yet but I do wonder about the feasibility of writing code in Python and then having an LLM transcode it to something like C, especially since I know C well enough to do what I want in that directly so I could check the resultant code by hand.
We are not yet there, however I do1 see a future where the target language might as well be machine code, and slowly we will leave the current languages behind.
Moving stuff out of the standard library seems like a reason. However, I think this all is a weird mix of arguments. IMHO new process spawning is a feature and not a bug in the use cases where CGI is used. Most of the stuff is low traffic config interfaces or remote invocable scripts. There was this trend to move stuff to fcgi. We had tons of cases of memory leaks in long running but really seldomly used stuff like mailing list servers. To me cgi is the poor man's alternative to serverless. However, I also do not really completely understand why a standard library has to support it. I have bash scripts running using the Apache CGI mod.
I would have phrased it, "serverless is a marketing term for CGI scripts."
I have bash CGI scripts too, though Shellshock and bash's general bug-proneness make me doubt that this was wise.
There are some advantages of having the CGI protocol implemented in a library. There are common input-handling bugs the library can avoid, it means that simple CGI programs can be really simple, and it lets you switch away from CGI when desired.
That said, XSS was a huge problem with really simple CGI programs, and an HTML output library that avoids that by default is more important than the input parsing—another thing absent from Python's standard library but done right by Django.
I used bash cgi scripts all the time. Haven’t used a python cgi module but the main benefit of perl’s cgi module (also removed) is the query parameter parsing.
Right. Isn't that insane? If I hadn't found it by means of having modules removed that my code depended on, I would have thought it was satire.
That policy, and the heinous character assassination the PSF carried out against Tim Peters, mean I can no longer recommend in good conscience that anyone adopt Python.
I really understand your frustration. Everyone developing in Python for a long time has felt it a bit too often when breaking changes (even between minor version updates) once again ruins the day.
But I also understand that the world is not perfect. We all need to prioritize all the time. As they write in the rationale: "The team has limited resources, reduced maintenance cost frees development time for other improvements". And the cgi module is apparently even unmaintained.
I guess a "batteries included" philosophy sooner or later is caught up by reality.
What do you mean by "character assassination" carried out against Tim Peters? Not anything in the linked article I presume?
Alright. Another case when "code of conducts" trumps manners or actually being a grownup. It really is a shame. Happened to a friend of mine on a rather big technical mailing list just for arguing for something that some people disagreed to. It would be nice to get back to a system based on manners and respect. That system worked for years.
Maintenance costs... that only exists because other parts of Python do not keep a stable and backwards compatible API? Same problem as everywhere else, but particularly silly when there are different parts of the same organization that is ruining it for each other internally. Not that I think it is ever defensible. A small cost-saving in one place that is causing more extra work in many other places.
They have limited resources because the inner circle chased away most active people in order to secure their own corporate positions (which hilariously failed since companies caught on and fired some of them anyway).
So the remaining people periodically launch some deprecation PEPs or other bureaucratic things in order to give the appearance of active development.
Python is for everyone, not just the PSF Cabal. Like the Democratic party, there is a huge need for new leadership. We have all seen what a little brigading can do.
> Everyone developing in Python for a long time has felt it a bit too often when breaking changes (even between minor version updates) once again ruins the day
No, not everyone. I've been using Python as my primary language since 2000 (that's 1.5.2 days). It has been the least troublesome language that I work with, and I work with (or have worked with) a bunch (shell, perl, python, ruby, lua, tcl, c, objective-c, swift, java, javascript, groovy, go and probably others I'm forgetting).
Even all the complaints about the Python packaging ecosystem over the years... I just don't get it. Like, have you ever tried working with CPAN or Maven or Gradle or even, FFS, Ruby Gems/bundler? The Python virtual environment concept is easy to understand and pip mostly does its job just fine, and these days, uv makes all that even faster and easier.
Anywho, just dropping a contrarian comment here because maybe I'm part of the generally silent majority that is just able to use Python day in and day out to get their job done.
> There are only two kinds of languages: the ones people complain about and the ones nobody uses. --Bjarne Stroustrup
I've used CPAN, Maven, gem, and bundler, so I'm also always a little puzzled when people complain about Python's packaging system. However, I've also used npm, so I can kind of understand it.
Python was great in 02000, but some of the things that made it great then are gone now. Stability was one of those; simplicity another; reasonable performance a third; but the biggest issue is really social rather than technical. And I feel like we have alternatives now that we didn't have then.
I wouldn't recommend Python for new projects regardless. New scripts maybe, sure, but not new projects. Python has a lot of problems and it's just not really worth it because the experience it provides is not unique.
Performance is a big one. GIL is still a thorn in Python's side, although somewhat ironically CGI side-steps that. Then there's the environment configuration, which is just one big footgun. And then there's error handling. I find python scripts with ticking time bombs all the time.
The maintenance burden of Python projects is just so much higher than it has any right to be. The language is neat, but not that neat. I think too often we think of performance as a sort of "tradeoff" for having a bad, unergonomic language, but that's not necessarily true. Plenty of languages have poor performance and are also a pain in the ass. We no longer live in a world where our options are C++ or scripting languages. We have mature environments with fantastic tooling. We have fast compilers with amazing error messages. We have great runtimes with more than adequate performance.
Go is a great choice these days pretty much exclusively due to the tooling. Turns out having a good compiler that's really, really fast is a big deal.
My main issue with prototyping as a concept is that it doesn't exist in most workplaces. Prototypes quickly devolve into applications. Discarding code is risky. Your best bet IMO is choosing a language that's ergonomic in the long run, because odds are you're in for the long run.
dotnet is another great choice because of the tooling and batteries included, although you do have to deal with a fairly slow compiler. Java is okay too, but Java is very restrictive and high-friction, which might not lend itself to prototyping.
In the world of scripting languages, ironically PHP is a decent choice. It has better progressive typing than Python and it's reasonably safe these days. We've sort of come full circle on PHP. The downside is that PHP programmers tend to throw everything in an array, especially when going fast. That hurts readability and the IDE a lot.
And then, of course, typescript and node. I don't like typescript. There's something about scripting languages with build steps that pisses me off. But, it's got a wide developer pool and it's not the worst language ever. Although there's a bit too much teeth-pulling IMO with typescript.
I don't even use Python, and even I've read Tim Peters' works and think highly of him! To have him so unceremoniously booted for upsetting a committee is absolutely insane.
This is a bit like Apple firing Steve Jobs for wearing sneakers to work because it violates some dress code.
Idk about the internal affairs, I just really don't like Python for web backend kind of things. It's taking them way too long to sort out parallelism and packaging, while NodeJS got both right from the start and gracefully upgraded (no 2->3 mess).
Also I used Python way before JS, and I still like JS's syntax better. Especially not using whitespace for scope, which makes even less sense in a scripting language since it's hard to type that into a REPL.
Node.js actually had no parallelism at the start, other than the ability to manually spawn new processes. Worker threads were only added in 2018 with v10.5.0, and only stabilized in 2019 with v12.
What Node.js had from the start was concurrency via asynchronous IO. And before Node.js was around to be JavaScript's async IO framework, there was a robust async IO framework for Python called Twisted. Node.js was influenced by Twisted[0], and this is particularly evident in the design of its Promise abstraction (which you had to use directly when it was first released, because JavaScript didn't add the async/await keywords until years later).
I was referring to the async io from the start, not worker threads. Other langs had their own frameworks for this, including Twisted for Python, but it really makes a difference having that stuff built-in and default.
Async IO is concurrency, not parallelism. And Node.JS is simply a framework for JavaScript, like Twisted is a framework for Python. If you compare a framework to a language, then of course the framework has more stuff built in; but that's hardly a fair comparison.
NodeJS is a separate runtime. It's not really the same language either since import syntax differs from web JS and the standard libs are different. Or in practical terms, you can't copy a lot of browser JS code and expect it to work in NodeJS as-is.
But that's beside the point. Performant web backends are way easier to deal with in NodeJS than in Python. I'm not comparing to Twisted because, even though it looks good, every Python backend I've ever seen was either plain Python or Django, which was also a mess compared to Express.
The amount of breakage in the Node land when doing major package upgrades far exceeds anything seen in Python. And it happens more often, too, because the stdlib is so thin you need way more packages to do anything interesting.
Not saying that Python is great, but Node is even worse.
Yeah, generally I feel like the indentation sensitivity was the right idea (the alternative evidently being worse compiler error messages, bugs like the `goto fail` vulnerability, and greater verbosity) but it causes real difficulties with the REPL, as well as with shell one-liners.
Jupyter fixes the REPL problem, and it's a major advance in REPLs in a number of other ways, but it has real problems of its own.
I agree async is a mess, but for web backends what is wrong with multi-process?
I do not think JS got it right. Node did, by doing async, but the reason for that was that JS did not do threads! It was making a virtue of a deficiency.
JS didn't do threads for a reason, though. It's not that the people working on JS had never heard of threads. Java, which JS was named after, was pervasively multithreaded from the beginning. The Microsoft IE folks lived and breathed threads. Opera even had multithreaded JS in 02000 before they took it out.
JS didn't do threads because threads are an error-prone way to write concurrent software. Crockford was a huge influence on its development in the early 02000s, and he had been at Electric Communities; he was part of the capabilities cabal, centered around Mark S. Miller, who literally wrote his dissertation on why you shouldn't use threads and how to structure async code comprehensibly. Promises came from his work, by way of Twisted. Unfortunately, a lot of that work didn't get into JS until well after Node had defined a lot of its APIs.
But this wasn't "making a virtue of a deficiency". JS was intentionally exploring a different approach to structuring concurrent software. You might argue that that approach didn't pay off, but it wasn't some kind of an accident.
I would say that it would be better to offer both options, depending on what you are doing. Not so much for JS's original role in the browser, but threads can be the right approach for a lot of backend tasks.
TCL, which promoted the same approach for the same reasons long before Node, eventually added threading.
That's an excellent point about Tcl; I'm sure it was a strong influence on Brendan's design for JS, though maybe not as strong as Perl, Java, and Scheme.
We clearly need some way to take advantage of manycore, but
I'm still not convinced that threading with implicitly shared mutable state is the right default. It isn't even what the hardware implements! Every core has its own cache! It's a better fit to the hardware than a single giant single-threaded event loop is, and I think that accounts for its curent dominance, but there are a lot of other possibilities out there, like transactional memory, explicit asynchronous message-passing interfaces, or (similarly) lots of tiny single-threaded event loops like Erlang (or, maybe, like Web Workers).
Web backends tend to have lots of concurrent connections doing IO-bound things. Processes or OS threads have too much overhead for this; they're more for when you want CPU parallelism with some communication between threads. Thread pools are still only a compromise. So JS with event loops made a lot of sense.
Greenthreading like in Golang is even better cause you get the advantages of OS threads, but that requires a runtime, which is why Rust didn't do that. And I guess it's hard to implement, cause Java didn't have it until very recently.
An early Java version had greenthreads, but they were soon removed in like the year 2000. That's pretty much the reason all our Java code at work uses some kind of cooperative multitasking with that painful promises syntax (foo().then(...))
It seems Python has firmly reached its "Wikipedia notability" era, with busybodies that code little but discuss much dominating and ruining all progress. They make up stuff that reads insane to anyone doing actual work, like the "maintenance burden" of the cgi module:
Amusingly that links to https://peps.python.org/pep-0206/ from 14th July 2000 (25 years ago!) which, even back then, described the cgi package as "designed poorly and are now near-impossible to fix".
Looks like the https://github.com/jackrosenthal/legacy-cgi package provides a drop-in replacement for the standard library module.