so, i've gone through a bunch of iterations on how to interview in days of LLMs and i came up with a process that is quite good:
first, a coding problem that is solved live (right now, online) -- it's not a hard one and most LLMs can easily one shot it.
BUT: if you are not a senior developer, you need to do it without LLM, and if you are a senior+ and do it via LLM, the bar is super raised and now you need to explain in details everything the LLM wrote as if you wrote it yourself.
the main component here is to know if the candidate understands coding in general, understands problem solving and how to dive into an existing (although quite small) codebase. solving the problem is not really the goal -- in fact, most candidates we hired did not solve it 100%, but were able to understand and explain everything that they put it on code. this usually takes 1h30.
(we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
the second is an architecture interview, where we build a system together BUT instead of doing the whole spec, miro board, whatever, we do it in more of a free way: we start with a really simple, MVP level, spec and we ask how the candidate would build it. then, we add another feature, and we ask how they would go for it. during this whole thing, i ask random questions like "why would you use x instead of y?" or "why do you prefer this approach?".
one important thing: we allow the candidate to say "i don't know" and we tell the candidate that there's no gotcha questions (which is true, i hate these kind of questions). this lets the candidate be super comfortable (feedback we had from real people).
in general, this flow helped us a ton and i really really like running it :)
I feel like git operations are one of the few things that I truly don't care at all about whether someone uses an LLM for and doesn't tell me. PR descriptions and commit messages should be written by humans for humans, but like, if someone doesn't know how to manually configure their git remotes and what exact command to use to amend/squash their commits into the shape needed and instead has an agent prepare all that and use the GitHub CLI to open the PR, why should I care? Beyond the code itself, I care about the things that will continue existing in the long run fitting my team's conventions (the number of commits, the commit messages, the PR description and comments, and then whether there's a merge commit at the end), but the process used to create those things doesn't matter to me.
> but like, if someone doesn't know how to manually configure their git remotes and what exact command to use to amend/squash their commits into the shape needed and instead has an agent prepare all that and use the GitHub CLI to open the PR, why should I care?
Because LLMs aren't a reason to not learn your tools. A determinist CLI command vs using a statistical machine for it. It's just a waste of tokens at the very least.
I rarely use anything beyond checkout, clone, pull, commit, cherry-pick, revert, and push in a job. If I have to, I'm probably asking the AI what to do by giving it my situation. And hopefully the AI can read about whatever git standards the company would like to be enforced so the engineer doesn't have to interrupt someone for that.
This is more about opportunity cost because instead of asking about git, you could ask about other valuable skills in your limited time with the interviewee, but I guess if git is really that important, then it's justified.
At one job I had, we used git extensively, but not GitHub. We had internal systems for review and branch management. I could tell you all the gory details about various workflows and landmines in git. Is the test here really about git, or git+GitHub?
Something similar for senior devs: code reviews. Reading and understanding code has always been important. I find offering simple code reviews very telling. In a way, your LLM approach matches that. It's a way to do a code review. It's a test of understanding, and not just memorization.
This is the third Jane Street-related thing I've seen on here this week after not seeing much about them that I can remember any time recently beforehand I'm aware of their use of OCaml and probably do other things that might be of interest to the HN community, but it seems like a lot in a very short time out of nowhere. Is there some sort of marketing push or something going on?
People like to talk about Jane Street because it is this Very High IQ™ company that pays people astronomical salaries and is extremely hard to get into.
They don't have to do marketing, though. It's not like they sell anything to Hacker News readers, they make their money trading finantial assets
You don't think that companies who want to hire engineers ever try to post stuff to Hacker News as a way to market themselves as a potential employer? Where did you get this impression of "Very High IQ" and "astronomical salaries"? I'm guessing at least part of it is from reading stuff from them or from talking to other people who read stuff from them.
The fact that you seem to adamantly disagree that there's any plausible angle here doesn't mean it's not marketing, but it might mean that it is marketing and that it's working on you.
Such strong wording, you got that from me saying "They don't have to do marketing"?
> where did you get this impression of [...] "astronomical salaries"?
From the real world fact that the base salary is US$300k/yr, and often said that the total compensation for the first year ends up being US$700k+. Even for Manhattan + SWE jobs, surely that merits "astronomical". Other devs making that much are usually working at Anthropic or OpenAI, aren't they?
> where did you get this impression of "Very High IQ"?
You didn't copy my ™ there, which was meant to be snarky.
In any case, what I was trying to say is that they don't have to do marketing in the usual sense. They don't have to get the Jane Street logo in your subconscious so you buy a Jane St. product when you go to Kroger's, and you can't sign a Jane St. subscription plan.
People in this thread seem to be implying that the company is botting/astroturfing, but for what? To maintain the image that they're a highly sought after employer?
Anything Paul Graham posts gets upvoted here instantly, because it's Paul Graham who posted it. For Jane Street there's the same halo effect, because many believe Jane Street's work is impressive. Is it really easier to believe the idea of botting to get an even longer list of candidates to interview, than it is to believe people just like to hear what this company has to say?
I don't ever see anyone saying Paul Graham is botting posts about his "essays"
I had lunch with a couple friends this week, and we each work in different tech environments. Most of the conversation was about the workflows we're each using, and seeing our colleagues use.
It's a really interesting time to work in this field. People who are paying attention to workflows are using a range of tools, and constantly refining their own workflow. For example a whole team might use Claude, but one person collects data about their own usage on an ongoing basis and develops more advanced workflows than everyone else on the team. Their focus isn't just about working faster, but about doing better work.
I don't think the post is vague; I think we're at a time where you need flexibility in your interview process to find candidates who are doing the best work these days.
This sort of announcement has made me very happy I left the software world. I still code as part of my job, but it's not my main part and luckily I don't have to use LLMs are part of my job. I still code 100% by hand.
In some ways, although I detest AI, it has clarified the sort of people and organizations I want to associate with. Any latent characteristics that I abhor have been made obvious with the existence of AI and of course, I would want nothing to do with Jane Street.
first, a coding problem that is solved live (right now, online) -- it's not a hard one and most LLMs can easily one shot it.
BUT: if you are not a senior developer, you need to do it without LLM, and if you are a senior+ and do it via LLM, the bar is super raised and now you need to explain in details everything the LLM wrote as if you wrote it yourself.
the main component here is to know if the candidate understands coding in general, understands problem solving and how to dive into an existing (although quite small) codebase. solving the problem is not really the goal -- in fact, most candidates we hired did not solve it 100%, but were able to understand and explain everything that they put it on code. this usually takes 1h30.
(we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
the second is an architecture interview, where we build a system together BUT instead of doing the whole spec, miro board, whatever, we do it in more of a free way: we start with a really simple, MVP level, spec and we ask how the candidate would build it. then, we add another feature, and we ask how they would go for it. during this whole thing, i ask random questions like "why would you use x instead of y?" or "why do you prefer this approach?".
one important thing: we allow the candidate to say "i don't know" and we tell the candidate that there's no gotcha questions (which is true, i hate these kind of questions). this lets the candidate be super comfortable (feedback we had from real people).
in general, this flow helped us a ton and i really really like running it :)
(edit: spelling)
Because LLMs aren't a reason to not learn your tools. A determinist CLI command vs using a statistical machine for it. It's just a waste of tokens at the very least.
I've heard this multiple times now and it still boggles my mind that git is something you have to test for.
This is more about opportunity cost because instead of asking about git, you could ask about other valuable skills in your limited time with the interviewee, but I guess if git is really that important, then it's justified.
not even joking: i've seen people with like 10y of experience trying to push to main :(
Jane street news made it to the front page a couple times and got a lot of attention.
Now more people are recognizing the name and upvoting other Jane Street posts.
Also one of my absolute favorite programming talks ever was hosted by Jane street:
https://www.janestreet.com/tech-talks/algorithm-for-precisio...
It’s like half talk, half story. Got me super interested in logic programming, and minikanren in particular.
https://hn.algolia.com/?q=jane+street
They don't have to do marketing, though. It's not like they sell anything to Hacker News readers, they make their money trading finantial assets
The fact that you seem to adamantly disagree that there's any plausible angle here doesn't mean it's not marketing, but it might mean that it is marketing and that it's working on you.
Such strong wording, you got that from me saying "They don't have to do marketing"?
> where did you get this impression of [...] "astronomical salaries"?
From the real world fact that the base salary is US$300k/yr, and often said that the total compensation for the first year ends up being US$700k+. Even for Manhattan + SWE jobs, surely that merits "astronomical". Other devs making that much are usually working at Anthropic or OpenAI, aren't they?
> where did you get this impression of "Very High IQ"?
You didn't copy my ™ there, which was meant to be snarky.
In any case, what I was trying to say is that they don't have to do marketing in the usual sense. They don't have to get the Jane Street logo in your subconscious so you buy a Jane St. product when you go to Kroger's, and you can't sign a Jane St. subscription plan.
People in this thread seem to be implying that the company is botting/astroturfing, but for what? To maintain the image that they're a highly sought after employer?
Anything Paul Graham posts gets upvoted here instantly, because it's Paul Graham who posted it. For Jane Street there's the same halo effect, because many believe Jane Street's work is impressive. Is it really easier to believe the idea of botting to get an even longer list of candidates to interview, than it is to believe people just like to hear what this company has to say?
I don't ever see anyone saying Paul Graham is botting posts about his "essays"
ok thanks for this slop then. Dont see why this stupid article is on frontpage.
They seem to belive in 'organic' marketing like doing a fake podcasts on dwarkesh. So maybe they have some bots upvoting their name.
It's a really interesting time to work in this field. People who are paying attention to workflows are using a range of tools, and constantly refining their own workflow. For example a whole team might use Claude, but one person collects data about their own usage on an ongoing basis and develops more advanced workflows than everyone else on the team. Their focus isn't just about working faster, but about doing better work.
I don't think the post is vague; I think we're at a time where you need flexibility in your interview process to find candidates who are doing the best work these days.
In some ways, although I detest AI, it has clarified the sort of people and organizations I want to associate with. Any latent characteristics that I abhor have been made obvious with the existence of AI and of course, I would want nothing to do with Jane Street.