132
points
dakshgupta
Joined 590 karma
Co-founder at Greptile.com.
- Apologies, that is poor wording on our part. It's internal data from engineers that use Greptile, which are tens of thousands of people from a variety of industries. As opposed to external, public data, which is where some of the charts are from.
- Most of our customers are enterprises, so I feel relatively comfortable assuming they have some decent testing and QA in place. Perhaps I am too optimistic?
- Thanks! The first 4 charts as well as Chart 2.3 are all from our data!
- This is a good one, wish we had included it. I'd run some analysis on this a while ago and it was pretty interesting.
An interesting subtrend is that Devin and other full async agents write the highest proportion of code at the largest companies. Ticket-to-PR hasn't worked nearly as well for startups as it has for the F500.
- How would you measure code quality? Would persistence be a good measure?
- This is a great suggestion. I'll note it down for next years. Curious, do you think this would be a good proxy for code quality?
- This is per month, I see now that's not super clear on the chart!
- We're careful not to draw any conclusions from LoC. The fact is LoCs are higher, which by itself is interesting. This could be a good or bad thing depending on code quality, which itself varied wildly person-to-person and agent-to-agent.
- We weren’t able to find a good quality measure. LLM-as-judge dint feel right. You’re correct that without that the data is interesting but not particular insightful.
- We weren’t able to agree on a good way to measure this. Curious - what’s your opinion on code churn as a metric? If code simply persists over some number of months, is that indication it’s good quality code?
- We expressly did not conclude that more lines = better. You could easily argue more lines = worse. All we wanted to show is that there are more lines.
- We were trying not to insinuate that, because we don’t have a good way to measure quality, without which velocity is useless.
- Hi, I'm Daksh, a co-founder of Greptile. We're an AI code review agent used by 2,000 companies from startups like PostHog, Brex, and Partiful, to F500s and F10s.
About a billion lines of code go through Greptile every month, and we're able to do a lot of interesting analysis on that data.
We decided to compile some of the most interesting findings into a report. This is the first time we've done this, so any feedback would be great, especially around what analytics we should include next time.
- Greptile | Software Engineer (junior, senior, staff)| San Francisco ONSITE | https://greptile.com
Greptile is building AI agents that catch bugs in pull requests. Over 2,000 teams including Brex, Whoop, and Substack use Greptile to review nearly 1B lines of code every month.
We're a team of ~20 in San Francisco, working on things like better agent evals and sandbox execution environments.
We've raised $30M to date, including our recent Series A led by Benchmark.
Stack: Typescript
Open roles: greptile.com/careers
Salary ranges: $140k-270k base (depending on seniority) + $40-100k/yr equity
- 5 points
- Greptile | Software Engineer | ONSITE San Francisco (SF) | https://greptile.com
Greptile is working on AI agents that catch bugs and enforce standards in pull requests. Reviewing nearly 1B lines of code a month for 1000+ companies including Brex, Substack, Whoop, as well as multiple F100s.
<20 people, raised ~$30M from Benchmark, YC, Paul Graham, SV Angel and others.
To apply, email daksh [at] greptile.com with subject line "Engineering at Greptile". Include most recent role and company and links to your LinkedIn and GitHub.
- That used to be how we did it, but this method performed better on super large codebases. One of the reasons is that grepping is a highly effective way to trace function calls to understand the full impact of a change. It's also great for finding other examples of similar code (for example the same library being used) to ensure consistency of standards.
- Change in number of revisions made between open and merge before vs. after greptile
- Percentage of greptile's PR comments that cause the developer to change the flagged lines
Assuming the author is will only change their PR for the better, this tells us if we're impacting quality.
We haven't yet found a way to measure absolute quality, beyond that.