1
point
forgetbook
Joined 20 karma
snip
- To explain better, there's basically 3 jobs avaliable in this kind of work right now, all of which I've interviewed for and been not selected with varying degrees of frusturation.
(1) build a bunch of automation scripts that we can reliably run as part of our product. This is closest to my day-to-day, you're (mis?)using selenium or playwright to make a known set of "click here, fill that, click there" scripts, and then expose some interface that calls them. At a company usually this will be a fastAPI / microservice, and your set of automation scripts is part of what makes the product possible. In my case, I'm currently working on registering my Tkinter/Pyinstaller app with windows so people can run my .exe and click the "do the script" button. There's also a slightly different approach to the selenium/playwright jam that runs curl requests which mimic the network requests of performing some action. You'll be able to see what I mean by clicking into your browser devtools "network" tab and clicking around the web. Imagine having a library of known API calls, you save a users cookie, and run script to achieve result.
(2) build a browser agent that we can use to run all of our (1) scripts. This is stuff that I iterate on as needed, and in my experience is best done by having a bunch of "helper functions" that spawn/kill/season different selenium webdrivers. It will depend on whether your (1) scripts are dealing with JS-heavy websites, bot detection, or just constantly changing UI's, but the deep rabbit holes here end up in places where you're building your own browser agent on top of webkit/chromium, implementing some kind of captcha solver, or trying to automagically discover what buttons exist by fuzzing the API or DOM.
(3) use a (2) to get us every PDF we need for our upcoming RAG chatbot. This is something I do by executing on (1), and I only bother to note the difference because it's a great example of the kind of actualy end product goal that all of this leads to.
Academically, the problems happening in web automation broadly fall into discovery and reproduceability. Discovery, meaning API fuzzing (how do we get that library of known API calls? / the equivalent buttons in a DOM?); Reproduceability, meaning running-without-errors. (How do we wait for the target's server to be ready to send the next command? How do we avoid getting blocked? How do we detect/recover when the target updates their website?) The most interesting opportunity IMO is inserting an LLM to build a self-healing scraper, and the edges of your tes/prod environments will be defined by your product's tolerance for wrong/nondeterministic behavior. I've got a great blogpost draft about a "railroad model of software development", where an LLM is a hammer nondeterministically pounding in railway ties, and an end product is a deterministic piece of code that can have trains run over it all day long. (effectively, LLM as test-environment devtool thesis, I don't think I'm saying anything that hasn't been said before.)
Practically, the problems that are facing me as an engineer are in packaging these tools for sale/distribution. My current state-of-the-art is to wrap up a .exe with Pyinstaller, build a GUI with Tkinter, and register with Windows so I'm not showing scary "This program is made of evil" messages when people try to run it. From there the plan is to give away free trials and after a month of people clicking their magic buttons they all disable and demand you purchase a license to re-enable (like if winRar was evil, but sorry yall I gotta eat). I'm also trying to sell building these tools as a service but that's very word of mouth, I haven't found a viable web/storefront model for that yet.
IM-Practically, most companies with a browser automation component are struggling with the same HR/Onboarding issues everyone else is. In the past ~2 months I've Interviewed at ~4 serious companies that profit from their browser automation, and every process has been unique: 1) firecrawl.dev; a black mirror hourlong AI-chatbot-zoom-interview, followed by a human call scheduled 3 weeks out, only to then be told that really they want someone fully specialized on breaking captchas, with a vaugely condescending suggestion that I'm "customer facing" and no followup when I lean into that 2) atomic.financial; actual-human-zoom-calls with engineers who I get along with great only to have no idea why I'm getting a rejection email the next week 3) sheer.health; a very contentious first call that demands I name a salary, followed by a trivially easy take-home test, followed by my 3rd round, 1/2 hour call with the CEO being cancelled the morning of because they filled the role. 4) Mozilla; where a principal/staff engr cold DM'd me, to schedule a call with an HR rep that told me they're paying 350k base 420k total and another staff eng who's leaving to start a startup, only to then tell me I'm not senior enough but could maybe come on as a contractor, only to then tell me they're using internal resources for the contractors.
Overall, I think the best opportunity in the space is going to look something like https://ui.vision/, which is an open-source tool!
- It's almost 100% secretarial workflow automation. Someone is currently being paid to click, drag, etc. I collapse that into a single button and charge for the button.
- Very cool. I make a consulting business out of packaging selenium scripts into windows apps for small businesses, do you have any desire to turn this into a saleable product?
- Thank you!
- I did your challenge problem, hosted it on my server, and sent you an email to a chorous of crickets.
- Any thoughts on Nix for this?
- I'm still junior, as in I spend more time reading docs than writing code because most code I have to write is stuff I haven't written before.
IMO, first impression? This is just a straight-up better way to show docs to me. To quote the landing page: "Often times, I’ll want to refer to different pages at the same time. So I’ll CMD + click “a couple times” while browsing around and before I know it, I have 12 new tabs open – all indistinguishable from each other because they share the same favicon."
Wow. They fixed it. First of it's kind, at least in my career so far. If you're got an example from DOS then yeah, I missed out, and agree that something important was lost along the way.
- Yep, this is pretty much exactly what I'm looking for, thanks!
As a though experiment, how could NixOS be packaged with this for consumer users?
- I see both of your points here. I also have lost muscle memory on nix-rebuild-shell, and it's a technical onboarding to get that back. What I'm hoping for is a graphical installer within NixOS that abstracts it away for nontechncial users
- This is a great note for AI developers, but my use case around NixOS is targetted to consumer users. The value is in self-hosted services (clouds, VPNs, etc.) without needing to be technical, and supporting the standard suite of stuff the average consumer needs/uses. (Word processor, email, internet, PDF, zoom)
My biggest concern here is not feature parity with the latest in AI; but in usability, or maybe irrelevance (what happened to thin clients?). My hope is that what stopped thin client adoption was just paying cloud providers forever, and that the average consumer that has a home computer can use that computer as a NAS to actually be their own iCloud / OneDrive, with the ability to deploy their 'home machine' on any laptop.
- You're misunderstanding, and it's a great opportunity for me to clarify:
I want a graphical installer for applications within NixOS.
Currently NixOS applications are added by editing the configuration.nix, with more specific tooling avaliable via flakes and home-manager. We agree that this is cool and good.
What I want for consumer use is NixOS, with an interal graphical installer that handles updating configurations in the background; ideally forming a no-terminal-necessary UX for consumer users.
- Def post the link when up; from your website what I'm getting is somehow docker images + kitchen sink, which may just be my lack of knowledge but in any case means I as a consumer / target user don't get it
- This is super cool, and between what you're up to and [Coding Without a Laptop - Two Weeks with AR Glasses and Linux on Android](https://www.hackerneue.com/item?id=43985513) I'm super excited for the pocket-computer, no-laptop future
- 1 point
How would you reccomend someone discover/decide is faster to learn/learn/implement existing solutions?
I'm asking because I'm currently in my "cambrian explosion" phase of homelabbing, so implementing that loop for myself personally will pay dividends.