I definitely agree with this. Our early prototype of Synnax actually sat on top of a combined Redis/S3/SQL stack and focused on those high level features. We found that it was challenging to deploy, manage, and synchronize data across these services, especially when you're running everything on prem.
We've come to believe that a re-architecture of the underlying infrastructure can actually unlock the high level workflows. For example, to compare a real-time telemetry stream with a historical data set you'd need to query across tools like Kafka and Influx at the same time. For an experienced software engineer this isn't too hard of a task, but they don't tend to be the people who understand the physics/mechanics of the hardware. We want it to be possible for say, a Turbo machinery expert, to translate a Python script they wrote for post-processing a CSV into something Synnax compatible without a huge amount of work.
In short, we're working on finding a way for subject matter experts in hardware to implement the anomaly detection mechanisms they already have in their head, but don't have the software expertise to implement.
> The thing I wanted was one solution for something that was always two: a properties/metadata database, and a separate time series database.
What do you think about TimeScale for this sort of use case? Haven't run it in production myself, but having your time-series data in the same place as a SQL table seems pretty nice.
> We found that visualization layers tended to reach down just far enough into the data intake world that it was really hard to sell just another tsdb.
This is a good point. We think that focusing exclusively on the DB is probably not the right approach. Most of our focus nowadays is on building out higher level workflows on top of the new database.
I found TimescaleDB after I wrote this — it does look like the answer to my problems from a decade ago. I don’t do that anymore but I’m glad someone brought it to market.
If you can describe with clarity how a scientist/hardware engineer using your tool is going to implement their anomaly detection, or whether your software will somehow shadow and assist/learn from what they try to do, I think that would be a much more compelling pitch.
> We used old control software that spit out data in massive 10 GB CSV or TDMS files. After a long day and night of testing, no one wanted to go through all the work to review the data.
> We think Synnax is unique in that it provides a bridge between <lab/automation DAQ systems>
On the surface it seems like anomaly detection is still the hard problem, but you’re not setting out to solve it?
Time series databases are state of the art generally in finance, not in industrial/InfluxDB, so I don’t think saying you’re 5x influxSB on writes is going to persuade too many people, especially given the cost now for a terabyte of RAM. I’ll just move all of it to an in-memory database before I’ll take on the switching costs.
The thing I wanted was one solution for something that was always two: a properties/metadata database, and a separate time series database.
It seems to me like you are maybe building a level too low and could get a lot more value working on the problem that you say motivated you in the first place. It is hard because of all the context required to automatically detect anomalies, but I think that is why it is valuable to solve.
The value we had was we rolled in the data/cellular connection all the way down to the endpoint, so they could avoid IT integration, which was a big hurdle at the time. I don’t know if IT integration is still a hang up for your customers.
We found that visualization layers tended to reach down just far enough into the data intake world that it was really hard to sell just another tsdb.