1
point
eightnoteight
Joined 75 karma
Founder & CEO @ Opti Owl
https://optiowl.cloud
https://twitter.com/eightnoteight
https://linkedin.com/in/eightnoteight
- eightnoteight parentfor the workflow DAG, what type of backend are you guys using? is it like temporal or self-built durable workflows
- Zenact AI | Founding Engineers & Interns | Full-Time + 6-Month Internships | Onsite Bangalore | Location flexible for internships (India) Tech: Golang • Python • AI Agents
At Zenact AI, we are building AI agents that test apps like real users. I personally faced this problem at Zomato for over 6 years while handling many bugs and incidents.
We launched recently and already got 35+ signups from leading unicorns & soonicorns in India.
Backed by the Zomato mafia.
Team comes with deep expertise from Zomato’s scale journey.
## Roles:
* Founding Engineers.
* Interns (6 months). Must’ve built serious projects or freelanced early in college.
## Tech Stack:
Golang, Python, Java(5%), Appium, AWS, Docker
## You’ll work on:
* Building the platform from the active feedback from the customers, with heavy focus on improving the end to end latency of testing.
* Fine-tuned vision & reasoning models (currently 92% accuracy vs SOTA ~60%)
* AI agents for mobile testing, reasoning flows.
Apply: shoot a mail to sri@zenact.ai or fill form here: https://forms.gle/yWproMowhA4ZF4gv6
- first i thought the website would say how many elephants left in the world (like white rhinos)
- seems obvious in hindsight, but never knew CT scans are used in manufacturing QC
- Zenact AI | Founding Engineers & Interns | Full-Time + 6-Month Internships | Onsite Bangalore | Location flexible for internships (India)
Tech: Golang • Python • AI Agents
At Zenact AI, we are building AI agents that test apps like real users. I personally faced this problem at Zomato for over 6 years while handling many bugs and incidents.
We launched recently and already got 35+ signups from leading unicorns & soonicorns in India.
Backed by the Zomato mafia.
Team comes with deep expertise from Zomato’s scale journey.
Roles:
Founding Engineers.
Interns (6 months). Must’ve built serious projects or freelanced early in college.
You’ll work on:
* Building the platform from the active feedback from the customers, with heavy focus on improving the end to end latency of testing.
* Fine-tuned vision & reasoning models (currently 92% accuracy vs SOTA ~60%)
* AI agents for mobile testing, reasoning flows.
Apply: shoot a mail to sri@zenact.ai or fill form here: https://forms.gle/yWproMowhA4ZF4gv6
- one thing that is getting clear is that the gains from model enhancement is getting saturated
thats why we are starting to see a programming of ai, almost like programming building blocks
if there is a pathway for models to get smart enough to know when to trigger these hooks by themselves from system prompt or by default itself, then it wouldn't make sense to have these hooks
- being on the developer side, the realtime dashboard for admin ui is really useful for end to end testing
so many of the eventually consistent stores reflect so late that its a pain to work with their SDKs
out of curiosity, what kind of metrics or observability do we get for events that got dropped due to some issue
- credit balance api is pretty cool
always liked the v0 feature where they notify to top up when the balance runs low
when do apps invoke the credit balance api? before sending events or they poll asynchronously and keep a local cache about the credit balance?
i guess caching would be a bad pattern too as the app may not function immediately after the top up of credits
- most of the current systems that need a reliable managed service for distributed locking use dynamodb, are there any scenarios where s3 is preferrable than dynamodb for implementing such distributed locking?
- > The problem is that sentinel errors, as typically and idiomatically used, in fact are special, and are more expensive to deal with than other values. My suggestion to use boolean values outperforms them by a lot, 30x in fairly common idiomatic usage.
while I agree to some degree, but when performance comes into picture, what really matters more is normal path vs surprise path rather than happy path vs error path
it's hard to argue what is a happy path in the code, but it's not wrong to say that io.EOF check is a normal path of the code i.e. not a surprise in production. the bad performance of errors.Is is something to be improved upon but it's not a surprise in production when there are a large number of `errors.Is` checks during normal path of the code
now coming to the surprise path of the code, here's where performance gets really important, no one wants their code to suddenly hog 100% CPU because of some special error case - https://blog.cloudflare.com/cloudflare-outage . but such surprise paths often contain a large amount of business logic that weigh much more than how slow errors.Is function is compared to a boolean check
it would be interesting to see where this line of reasoning is valid but IMO performance isn't a good argument against why errors are not normal outcomes of operations in production
but thumbs up for the article, now I know what to reference for backing the below pattern that I often use, when I first saw the errors.Is it was pretty obvious that its going to be slow but just didn't have time to prove it and use below pattern
```
if err != nil && errors.Is(err, x) {
} else if err != nil && errors.Is(err, y) {
} else if err != nil {
}// handling unknown error```
- people who need to monitor a lot of channels are usually in senior/leadership layer, but one technique usually they follow is focus on a specific problem and consequently some set of specific channels for few weeks or a month and shift focus as the project/task changes
how are you thinking about capturing such dynamic decisions to choose focus area, happening outside the communication tool - like zoom or meetings etc,... algorithm can be real-time but even with data points from meetings etc,... can it be made in such a real-time?
instagram feed algo is pretty real-time but the number of unique behaviours or behaviours to people ratio is quite low. but I'm guessing in a work environment that ratio or the unique behaviours would be too high for the algo to react quickly, right?
- controlling the producer is such a hard problem, even with exponential back off and backoff times in the response headers, you still get at minimum 2x throughput increase from the producers during a retry storm
problem is that the most common backpressure techniques like exponential back-off and sending a retry-after time in the response header have constraints on maximum backoff time they can do, in some scenarios that is much much less than the normal.
for example, imagine a scenario where a customer explores 10 items on Amazon, and then finally places an order, so 10rps for the product page and 1 rps for the order page. if order services goes down, slowly the customers get stuck on the order page and even with backpressure, your RPS keeps on growing on the order page. exponential backoff doesn't help as well
while dropping requests is a good idea, but that action is not designed by default every time, systems go into metastable state and you need the ability to control the throughput on producer side
you could solve it by keeping a different layer in between like load balancer or some gateway layer that is resilient against such throughput spikes and will let you control throughput on your service and slowly scale up the throughput as per your requirements (by user or by random)
for frontend devices, it gets exponentially harder to control the throughput. having an independent config API that can control the throughput is the best solution that I came across
- I think of it as different companies helping at different stages of mainstream adoption
for any open source project, the initial adoption during its initial stages would have been that project itself, later on it will be someone with financial incentive to drive adoption but that would still be a niche market and as soon as it starts to show potential aws simply copies and gives it as an offering
all 3 help with the adoption, whether we like it or not, aws has much larger distribution channel and people would rather just use one of the existing vendor than buy a new Nth vendor
- > Nowadays I open Aegis and I have > 20 services there, and trying to look for my code between all the running numbers is a pain.
exactly :(
I wish passkeys get rolled out quickly across all sites, most people use just 2 or 3 trusted devices 99% of the time.
for those edge cases where you are working on an untrusted device, the passkey on your trusted mobile can help with authentication via Bluetooth or some QR code etc,...
- I took a high enough number to showcase the problem, for a fresher it doesn't change much even if that number is as low as 15 or 20, or even if 5 people that they don't know or at higher levels
also I feel like, the number of people that hop on the incident call are almost always related to the category of the incident, sure you can always break out to a separate room, but often the person would have already realised the impact and the weight of the incident
- > “Oscar, do you mind sharing your screen so Deepak and Deanna can see the weird log messages too?”
it seems so obvious from an Incident Commander perspective but so much goes into this workflow during an incident
* what if the person is a fresher, you are asking him to share screen, debug and perform actions in front of 100 people in the incident call and the anxiety that comes with it
* While IC has much more practice with handling fires continuously, for instance, if there is a fire every week in a 50-team organisation, a specific team would only be seeing their first incident once a year
* Self-consciousness/awareness instantly triggers a flight or fight response from even the most experienced folks
I don't know how other industries handle such a thing, I'm pretty sure even in non-tech there would be a hierarchy for the anomaly response and sometimes leaf level teams might be called to answer questions at top level of the incident response (like a forest fire response, might have a state wide response team and they pulling local response team and making them answer questions) probably they get much more time to prepare than in tech where its a matter of minutes
- websockets and sse are a big headache to manage at scale, especially backend, requires special observability, if not implemented really carefully on mobile devices its a nightmare to debug on frontend side
devices switch off network or slow down etc,... for battery conservation, or when you don't explicitly do the I/O using a dedicated API for it.
new connection setup is a costly operation, the server has to store the state somewhere and when this stateful layer faces any issue, clients keep retrying and timing out. forever stuck on performing this costly operation. it's not like there is an easy way to control the throughput and slowly put the load on database
reliability wise long polling is the best one IME, if event based flow is really important, even then its better to have a 2 layer backend, where frontend does long polling on the 1st layer which then subscribes to websockets to the 2nd layer backend. much better control in terms of reliability
- I never focussed much on sleeping postures, but one day I read this article about how acid reflux goes away if you side-sleep on your left hand side i.e stomach is at a lower height than when you sleep on your right hand side
that really changed my life, it was like, how did I waste 28 years of my life without finding this trick :D
- > here they explain why they had to betray their core mission. But they don't refute that they did betray it.
you are assuming that their core mission is to "Build an AGI that can help humanity for free and as a non-profit", the way their thinking seems to be is "Build an AGI that can help humanity for free"
they figured it was impossible to achieve their core mission by doing it in a non-profit way, so they went with the for-profit route but still stayed with the mission to offer it for free once the AGI is achieved
Several non-profits sell products to further increase their non-profit scale, would it be okay for OpenAI non-profits to sell products that came in the process of developing AGI so that they can keep working on building their AGI? museums sell stuff to continue to exist so that they can continue to build on their mission, same for many other non-profits. the OpenAI structure just seems to take a rather new version of that approach by getting venture capital (due to their capital requirements)
- > it keeps the intangible benefits it accrued by being ostensibly non-profit
but there would be no different to a for-profit entity right? i.e even for-profit entities get tax benefits if they convert their profits to intangibles
this is my thinking. Open AI non-profit gets donations, uses those donations to make a profit, converts this profit to intangibles to avoid paying taxes, and pumps these intangibles into the for-profit entity. based on your hypothesis open ai avoided taxes
but the same thing in a for-profit entity also avoids taxes, i.e for-profit entity uses investment to make a profit, converts this profit to intangibles to avoid paying taxes.
so I'm trying to understand how Open AI found a loop hole where if it went via the for-profit then it wouldn't have gotten the tax advantages it got from non-profit route
- I don't believe non-profits can have investors, only donors i.e an investor by definition expects money out of his investment which he can never get out of a non-profit
only the for-profit entity of the OpenAI can have investors, who don't get any tax advantage when they eventually want to cash out
- once it converts into profit-seeking venture, it won't get the tax benefits
one could argue that they did R&D as a non-profit and now converted to for-profit to avoid paying taxes, but until last year R&D already got tax benefits to even for-profit venture
so there really is no tax-advantage of converting a non-profit to for-profit
- > The best tool in this space should eliminate the need of specialized PM.
exactly this, having a project manager drive tasks just erodes the sense of ownership in the team. the sense of ownership that comes from giving the responsibility to devs to manage their tasks directly is immense, especially during the initial years
I think the for fast moving teams, the eventual winner in this space is going to be someone who can transparently bring in product requirements and timeline requirements to devs directly than via some middleman
- * changelog feature is pretty great
* one other feature I have always wanted at my previous company is about transparency of how many tasks a person is doing, that way I don't have to bother him multiple times on whether a certain task is picked or not
* similarly for monthly planning or weekly planning, the stakeholders involved in planning is often the team itself and humans are not that great in remembering what all promises they made over the last week. i wish there was some way for all stakeholders even outside teams to be notified of planning events and add items to the planning event agenda
- > "these systems have to be fully automated because no one could afford to operate manual systems at our scale", what's really being said is more along the lines of "we would not be able to generate as many billions a year in profit if we hired enough competent people to manually review cases our system should flag as ambiguous, so we settle for what we can get without compromising profits".
well anything that comes out of any business has these intentions for sure, but statements could be like multi-dimensional vectors, they could be doing 2 things at once
the ROI for fraud is low for a smaller platform and so it doesn't happen for smaller scale companies as much as it does for bigger scale. but given the interested party who is explicitly looking for the target cohort even the smaller platforms see the fraud
so even if social media as whole were to split into multiple entities, the ROI would still make sense for fraudsters to continue the fraud.
even if you have 1000s of entities, there would be a startup who can build a wrapper out of those 1000s of entities to make the fraud ROI possible. just like how there are businesses that offer scraping competitors
- I don't think it would have reduced the bill much, but generally RDS is 2x costly than EC2, but I'm guessing most of the improvement the article speaks about came from ephemeral disk i.e nvme storage
its a recent feature but I think RDS Optimized Reads should achieve similar improvement in performance and get the cost from 11K to 4.2K
- extra margins for products on top of ec2 are mostly fine, there is a huge R&D effort and continued maintenance, so extra charges to recuperate that spend are totally justifiable.
but 16-20% extra charges for foundational compute tech like fargate just go against the promise of cloud :(