The author has run into the same problem that anyone who wants to do analysis on the NPM registry runs into, there's just no good first party API for this stuff anymore.
It seems this was their first time going down this rabbit hole, so for them and anyone else, I'd urge you to use the deps.dev Google BigQuery dataset [0] for this kind of analysis. It does indeed include NPM and would have made the author's work trivial.
The package with most versions still listed on PyPI is spanishconjugator [2], which consistently published ~240 releases per month between 2020 and 2024.
Regarding spanishconjugator, commit ec4cb98 has description "Remove automatic bumping of version".
Prior to that commit, a cronjob would run the 'bumpVersion.yml' workflow four times a day, which in turn executes the bump2version python module to increase the patch level. [0]
I decided my life could not possibly go on until I knew what "elvisgogo" does, so I downloaded the tarball and poked around. it's a pretty ordinary numpy + pandas + matplotlib project that makes graphs from csv. one line jumped out at me:
str_0 = ['refractive_index','Na','Mg','Al','Si','K','Ca','Ba','Fe','Type']
the university of st. andrews has a laser named "elvis" that goes on a remote controlled submarine: https://www.st-andrews.ac.uk/~bds2/elvislaser.htm
I was hoping it'd be about go-go dancing to elvis music, but physics experiments on light in seawater is pretty cool too.
Tangential, but I've only heard about BigQuery from people being surprised with gargantuan bills for running one query on a public dataset. Is there a "safe" way to use it with a cost limit, for example?
Yes you can set price caps. The cost of a query is understandable ahead of time with the default pricing model ($6 per TB of data processed in a query). People usually get caught out by running expensive queries recursively. BigQuery is very cost effective and can be used safely.
We really need to go back to on-premise. We have surrendered our autonomy to these megacorps and now are paying for it - quite literally in many cases.
My 3TB, 41 billion row table costs pennies to query day to day. The billing is based on the data processed by the query, not the table size. I pay more for storage.
> spanishconjugator [2], which consistently published ~240 releases per month between 2020 and 2024
They also stopped updating major and minor versions after hitting 2.3 in Sept 2020. Would be interesting to hear the rationale behind the versioning strategy. Feels like you might as well use a datetimestamp for the version.
I hate to deride the entire community, but many of the collective community decisions are smells. I think that the low barrier to entry means that the community has many inexperienced influential people.
A lot of these decisions were made after Javascript went "enterprise" to make it seem more like a "serious" programming language to SV entrepreneurs by a small number of corporations, not necessarily the community.
The bar for entry was always low with javascript, but it also used to be a lot more sane when it was a publicly-driven language.
The Julia General registry is locally stored as a tar.gz and has version info for all registered packages, so I tried this out for Julia packages. The top 5 are:
So, no crazy numbers or random unknown packages, all are major packages that have just had a lot of work and history to them. Out of the top 10, pretty much half were from the SciML ecosystem.
Caveats/constraints: Like the post, this ignores non-SemVer packages (which mostly used date-based versions) and also jll (binary wrapper) packages which just use their underlying C libraries' versions. Among jlls, the largest that isn't a date afaict is NEO_jll with 25.31.34666+0 as its version.
Incidentally I once ran into a mature package that had lived in the 0.0.x lane forever and treated every release as a patch, racking up a huge version number, and I had to remind the maintainer that users depending with caret ranges won't get those updates automatically. (In semver caret ranges never change the leftmost non-zero digit; in 0.0.x that digit is the patch version, so ^0.0.123 is just a hard pin to 0.0.123). There may occasionally be valid reasons to stay on 0.0.x though (e.g. @types/web).
It's the type definitions for developing chrome extensions. They'd been incrementing in the 0.0.x lane for almost a decade and bumped it to 0.1.0 after I raised the issue, so I doubt it was intentional:
This is part of the DefinitelyTyped project. DT tends to get a lot of one-off contributions just for fixing the one error a dev is experiencing. So maybe they all just copied the version incrementing that previous commits had done, and no one in particular ever took the responsibility to say "this is ready now".
Anthony Fu’s epoch versioning scheme (to differentiate breaking change majors from "marketing" majors) could yield easy winners here, at least on the raw version number alone (not the number of sequential versions released):
I don't know if this is the origin, but the semver spec says 0.x.y is unstable. Sure, not everybody uses semver, but it is popular enough for people to make incorrect assumptions.
It's not the origin. Using 0.x for stuff like this was already a thing long before semver. For example, the very first release of Linux in 1991 was v0.01.
The "winner" just had its 3000th release on GitHub, already a few patch versions past the version referenced in this article (which was published today): https://github.com/wppconnect-team/wa-version
I made a fairly significant (dumb) mistake in the logic for extracting valid semver versions. I was doing a falsy check, so if any of major/minor/patch in the version was a 0, the whole package was ignored.
Hmm yeah, I decided that one counts because the new packages have (slightly) different content, although it might be the case that the changes are junk/pointless anyway.
Brief reminder/clarification that these tools are used to circumvent WhatsApp ToS, and that they are used to:
1- Spam
2- Scam
3- Avoid paying for Whatsapp API (which is the only form of monetization)
And that the reason this thing gets so many updates is probably because of a mouse and cat game where Meta updates their software continuously to avoid these types of hacks and the maintainers do so as well, whether in automated or manual fashion.
Considering the 18 billions price tag and the current mixing of user data between meta and WhatsApp I believe that meta has now revenue streams in mind than just the API pricing
> Time to fetch version data for each one of those packages: ~12 hours (yikes)
The author could improve the batching in fetchAllPackageData by not waiting for all 50 (BATCH_SIZE) promises to resolve at once. I just published a package for proper promise batching last week: https://www.npmjs.com/package/promises-batched
Just spin up a loop of 50 call chains. When one completes you just do the next on next tick. It's like 3 lines of code. No libraries needed. Then you're always doing 50 at a time. You can still use await.
async work() { await thing(); nextTick(work); }
for(to 50) { work(); }
then maybe a separate timer to check how many tasks are active I guess.
Promise.all waits for all 50 promises to resolve, so if one of these promises takes 3s, while the other 49 are taking 0.5s, you're waisting 2.5s awaiting each batch.
Why not write all of our applications on one file? Why bother using (language specific) modules? To take your argument to the logical extreme, DRY is a fanatical doomsday computer science cult.
If it's baked into the tool (can run offline) then it would be unavoidable, need a new version to get a new release on the package manager.
1.2.3 -> 1.2.3+1 (or +anything, date, whatever) could arguably be idiomatic semver though - that's what you do for packaging changes, like updating the description or categories to file it under etc. without actually changing the program.
And yet you need internet access when running the package with npx, even though the package is already locally installed.
At least, I can't use npx on so e of my VMs which do not have access to the internet. It just takes forever to atart (and I get annoyed after some minutes).
The particular problem here is if you started out doing it wrong then changing your update behavior would break everyone's scripting around it. By changing the 'code version' everyones CI/CD system just keeps working the same way as any other package.
There are plenty of larger ones and plenty of ones that used the date as the version, but I was mainly curious about packages that followed semver.
Any package version that didn't follow the x.y.z format was excluded, and any package that had less published versions than their largest version number was excluded (e.g. a package at version 1.123.0 should have at least 123 published versions)
Well, we are looking at npm packages, where every package is supposed to follow semantic versioning. The fact that we don't have date as version number means everyone is a good citizen.
Off the top of my head, CloudFlare uses a somewhat date based method of typing for their Workers types package, but it makes sense in context as you define compatibility dates for a Worker when you set it up, which automatically enables/disables potentially breaking features in the API.
`latentflip-test` is from the same fellow who did the "What the heck is the event loop anyway?" JSConf EU talk that many have seen. https://youtu.be/8aGhZQkoFbQ
> I was recently working on a project that uses the AWS SDK for JavaScript. When updating the dependencies in said project, I noticed that the version of that dependency was v3.888.0. Eight hundred eighty eight. That’s a big number as far as versions go.
It also isn’t the first AWS SDK. A few of us in… 2012 IIRC… wrote the first one because AWS didn’t think node was worth an SDK.
Haha, good luck finding a real project that holds that title. It's always some squatted name, a dependency confusion experiment, or a troll publishing a package with version 99999.99999.99999 just to see what breaks. The "king" of that hill changes all the time. Just another day in the NPM circus.
It seems this was their first time going down this rabbit hole, so for them and anyone else, I'd urge you to use the deps.dev Google BigQuery dataset [0] for this kind of analysis. It does indeed include NPM and would have made the author's work trivial.
Here's a gist with the query and the results https://gist.github.com/jonchurch/9f9283e77b4937c8879448582b...
[0] - https://docs.deps.dev/bigquery/v1/
The package with most versions still listed on PyPI is spanishconjugator [2], which consistently published ~240 releases per month between 2020 and 2024.
[1] https://console.cloud.google.com/bigquery?p=bigquery-public-...
[2] https://pypi.org/project/spanishconjugator/#history
Prior to that commit, a cronjob would run the 'bumpVersion.yml' workflow four times a day, which in turn executes the bump2version python module to increase the patch level. [0]
Edit: discussed here: https://github.com/Benedict-Carling/spanish-conjugator/issue...
[0] https://github.com/Benedict-Carling/spanish-conjugator/commi...
The underlying dataset is hosted at sql.clickhouse.com e.g. https://sql.clickhouse.com/?query=U0VMRUNUIGNvdW50KCkgICBGUk...
disclaimer: built this a a while ago but we maintain this at clickhouse
oh and rubygems data is also there.
[0] https://sql.clickhouse.com?query=U0VMRUNUIHByb2plY3QsIE1BWCh...
[1] Quota read limit exceeded. Results may be incomplete.
They also stopped updating major and minor versions after hitting 2.3 in Sept 2020. Would be interesting to hear the rationale behind the versioning strategy. Feels like you might as well use a datetimestamp for the version.
This is insane
I hate to deride the entire community, but many of the collective community decisions are smells. I think that the low barrier to entry means that the community has many inexperienced influential people.
The bar for entry was always low with javascript, but it also used to be a lot more sane when it was a publicly-driven language.
Caveats/constraints: Like the post, this ignores non-SemVer packages (which mostly used date-based versions) and also jll (binary wrapper) packages which just use their underlying C libraries' versions. Among jlls, the largest that isn't a date afaict is NEO_jll with 25.31.34666+0 as its version.
Edit: nevermind, I misunderstood your point
https://www.npmjs.com/package/@types/chrome?activeTab=versio...
https://antfu.me/posts/epoch-semver
I wonder why. Conventions that are being broken, maybe.
https://semver.org/#spec-item-4
If the guy writing and maintaining the software is stating "this software is not stable yet" then who am I to disagree?
I made a fairly significant (dumb) mistake in the logic for extracting valid semver versions. I was doing a falsy check, so if any of major/minor/patch in the version was a 0, the whole package was ignored.
The post has been updated to reflect this.
1- Spam 2- Scam 3- Avoid paying for Whatsapp API (which is the only form of monetization)
And that the reason this thing gets so many updates is probably because of a mouse and cat game where Meta updates their software continuously to avoid these types of hacks and the maintainers do so as well, whether in automated or manual fashion.
The author could improve the batching in fetchAllPackageData by not waiting for all 50 (BATCH_SIZE) promises to resolve at once. I just published a package for proper promise batching last week: https://www.npmjs.com/package/promises-batched
Just spin up a loop of 50 call chains. When one completes you just do the next on next tick. It's like 3 lines of code. No libraries needed. Then you're always doing 50 at a time. You can still use await.
async work() { await thing(); nextTick(work); }
for(to 50) { work(); }
then maybe a separate timer to check how many tasks are active I guess.
The implementation is rather simple, but more than 3 LoC: https://github.com/whilenot-dev/promises-batched/blob/main/s...
Couldn't find any specific rate limit numbers besides the one mentioned here[0] from 2019:
> Up to five million requests to the registry per month are considered acceptable at this time
[0]: https://blog.npmjs.org/post/187698412060/acceptible-use.html
> carrot-scan -> 27708 total versions
> Command-line tool for detecting vulnerabilities in files and directories.
I can't help but feel there is something absurd about this.
1.2.3 -> 1.2.3+1 (or +anything, date, whatever) could arguably be idiomatic semver though - that's what you do for packaging changes, like updating the description or categories to file it under etc. without actually changing the program.
And yet you need internet access when running the package with npx, even though the package is already locally installed.
At least, I can't use npx on so e of my VMs which do not have access to the internet. It just takes forever to atart (and I get annoyed after some minutes).
Any package version that didn't follow the x.y.z format was excluded, and any package that had less published versions than their largest version number was excluded (e.g. a package at version 1.123.0 should have at least 123 published versions)
https://docs.npmjs.com/about-semantic-versioning
https://www.npmjs.com/package/@cloudflare/workers-types
AWS still made the top 50
If this was an actual measurement of productivity that bot deserves a raise!
Bigliest, boomiest version is 3735928560 from https://metacpan.org/dist/Acme-Boom
It also isn’t the first AWS SDK. A few of us in… 2012 IIRC… wrote the first one because AWS didn’t think node was worth an SDK.