I’m eager to test this out. I have agent instructions to try to limit the worst of this already, but patterns still sneak through. I have a review agent run after every single edit looking for all of the following if you need more ideas for checks:
- DRY principle violations, multiple definitions of the same helpers or utilities.
- Changes that deviate from existing patterns and architecture already in the code, especially in nearby and related code
- Comments that add no context or simply restate the field name.
- Naming violations (enterprise factoryfactoryabstraction stuff, excessively long names, overly technical names, banned words like “seam”, “durable”, and no-value-qualifiers like “SaveGame” -> “Save”).
- Tests that check implementations instead of correct business behavior.
- Overly backwards-compatible unless asked for (this one is incredibly hard to keep under control, as AI loves to guard everything even if the previous code was never deployed and thus there is no contract break)
- Un-necessary guard code (this is hard to control, most common case is the AI not relying on the serializer error handler and instead adding guards that the library already handles)
- Changing public API contracts without express permission to do so (depends on the code, eg a library JAR or versioned REST service)
- Meta references to previous code versions, to tasks or todos, or to instructions and other non-code context (e.g you tell the AI the adder should ignore negative numbers and that meta fact enters the comments or code)
I usually hand review all changes myself but it’s incredibly tedious so I try to first pass with the review agent until it comes back clean. I hate wasting tokens on it though.
Oh my gosh, the guard code drives me crazy. In try so hard to get Kimi to put in asserts instead of silently swallowing corrupt values, but it keeps handling bad values poorly instead of crashing. I've even explicitly put in CLAUDE.md that correctness is more important then continuing to run, but it still keeps defensively programming when it should loudly crash.
This is a great idea. Even if you're one of those developers squarely focused on getting the final result working, code quality still matters (to people and LLMs).
Everyone should be doing regular code reviews and this helps a lot.
A linter with rules for AI-specific weirdness is absolutely a great idea, thank you! Are there any plans to support other languages besides javascript?
I was about to write what advantage it has over linters but then saw the built on section. Good work. We use megalinter with our flavour of go and vite rules, plus extensive e2e testing after each agent run. Quality of the spec driven agentic PRs are significantly better than the baseline. Megalinter is quite resource heavy and slow, so will definitely check this out
Gave it a try but there were a lot of false positives. SQLModel's exec method for example gets flagged every time thinking it's python's exec() function.
I don’t see if this is one of the covered cases, but one of the more common and nefarious patterns I run into is what you might call "sweeping exceptions under the rug." I think the agent’s motivation to get things running encourages these antipatterns of designing routines that are fault tolerant in a sort of maladaptive way: e.g. catching an error, logging a warning that something didn’t work, and continuing, but with now potentially missing/broken state.
This has bitten me a couple of times, and it’s surprisingly annoying to nudge agents into good/resilient patterns or identify situations that should fail loudly, at least in my experience. The retry mechanisms they come up with on their own are often pretty terrible as well.
I’ll note, though, that I have seen this from human engineers plenty of times, and at least the AI usually adds some logs rather than just totally silently absorbing an exception!
I think a lot of the telltale signs of AI can be found in the comments. Besides the slop writing style, I've found AI comments to 1. be overly verbose, 2. unnecessarily describe before/after code state (# This function used to do foo, but now it does bar), and 3. reference its own internal "plan" (# This function is part of Stage 3 of the implementation of Use Case X from the requirements doc) WTF is Stage 3? - says code reader 2 years from now. Although I bet you can probably prompt these behaviors away.
I tried it on my codebase. There's a lot of overlap with tools like Oxlint / ESLint, I'm not sure that's too valuable vs. a more focused tool that actually focuses solely on 'slop' signals. These lint rules tend to get very opinionated which is why those tools expose so many configuration options.
One real bug tho:
> [ERROR] Imports "mdast" but it's not declared in package.json
A type-only import like `import type { Blockquote, RootContent } from 'mdast';` is actually acceptable if `"@types/mdast": "^4.0.4"` is included in the package.json.
One funny thing I see it doing is deleting seemingly random comments lines, for example if a file has a comment that spans multiple lines but doesn’t use a multi line comment syntax. It just chooses one at random transforming the once useful comment into slop.
The intent is good, but frankly, credibility is lost by using "slop" words imo, OP you might seriously want to re-evaluate who is the target market, probably users that leverage high automation 24/7, startups and so-on, they don't want to incorporate products that talks about the modern way (and somehow only way to compete) of development as "slop" imo because soon enough, it's not true anymore (it's already not true with the right tooling).
Yeah, to be honest, it really is a bit weird to incorporate a tool called slop into a corporate CICD flow. Anubis fixed a similar problem by creating a corp-safe version without the anime mascot.
Thanks, I actually concentrated on improving code quality, the patterns I flagged are poor design choices that humans wouldn’t write. Examples are duplicated functions doing same thing, dead or redundant codes etc. These builds up and degrade the codebase over time.
I'm interpreting this not as a "catch ai submissions gotcha" tool, but as a "last pass in review catch mistakes AI made that i may have missed" tool. Having more linters is a good thing IMO (I say this as someone who doesn't use AI to generate code, but works with people who do and has to review a lot of AI generated code)
- DRY principle violations, multiple definitions of the same helpers or utilities.
- Changes that deviate from existing patterns and architecture already in the code, especially in nearby and related code
- Comments that add no context or simply restate the field name.
- Naming violations (enterprise factoryfactoryabstraction stuff, excessively long names, overly technical names, banned words like “seam”, “durable”, and no-value-qualifiers like “SaveGame” -> “Save”).
- Tests that check implementations instead of correct business behavior.
- Overly backwards-compatible unless asked for (this one is incredibly hard to keep under control, as AI loves to guard everything even if the previous code was never deployed and thus there is no contract break)
- Un-necessary guard code (this is hard to control, most common case is the AI not relying on the serializer error handler and instead adding guards that the library already handles)
- Changing public API contracts without express permission to do so (depends on the code, eg a library JAR or versioned REST service)
- Meta references to previous code versions, to tasks or todos, or to instructions and other non-code context (e.g you tell the AI the adder should ignore negative numbers and that meta fact enters the comments or code)
I usually hand review all changes myself but it’s incredibly tedious so I try to first pass with the review agent until it comes back clean. I hate wasting tokens on it though.
(Not that I think you didn't probably already know that.)
Everyone should be doing regular code reviews and this helps a lot.
This has bitten me a couple of times, and it’s surprisingly annoying to nudge agents into good/resilient patterns or identify situations that should fail loudly, at least in my experience. The retry mechanisms they come up with on their own are often pretty terrible as well.
I’ll note, though, that I have seen this from human engineers plenty of times, and at least the AI usually adds some logs rather than just totally silently absorbing an exception!
This made me chuckle.
Cool tool, the dead code checks can be very helpful. I’ve seen Claude leave unused functions when iterating which always frustrates me.
> Don't post generated comments or AI-edited comments. HN is for conversation between humans.
One real bug tho:
> [ERROR] Imports "mdast" but it's not declared in package.json
A type-only import like `import type { Blockquote, RootContent } from 'mdast';` is actually acceptable if `"@types/mdast": "^4.0.4"` is included in the package.json.
I'd love for it to have flags for Pi and Crush too :)
One funny thing I see it doing is deleting seemingly random comments lines, for example if a file has a comment that spans multiple lines but doesn’t use a multi line comment syntax. It just chooses one at random transforming the once useful comment into slop.
I wouldn’t say they fixed it. Rather it was an intentional choice to put an anime girl on websites unless you paid for the product.
Concentrate on code quality, and whether it does what it needs to do. Not whether it was written by AI or not.
They certainly do in my experience. Maybe you've been lucky and haven't worked with really messy programmers.