Hacker Neue

Preferences
132 points
Show HN: Synthesize TikZ Graphics Programs for Scientific Figures and Sketches
12 comments potamides github.com
Creating high-quality scientific figures can be time-consuming and challenging, even though sketching ideas on paper is relatively easy. Furthermore, recreating existing figures that are not stored in formats preserving semantic information is equally complex. To tackle this problem, we introduce DeTikZify, a novel multimodal language model that automatically synthesizes scientific figures as semantics-preserving TikZ graphics programs based on sketches and existing figures. We also introduce a Monte Carlo Tree Search-based inference algorithm that enables DeTikZify to iteratively refine its outputs without the need for additional training.
nequo
This is brilliant work, OP! A great use case for machine learning.

Why does the arXiv non-exclusive license preclude the inclusion of arXiv figures in the published data set? I don't see how the conditions set by the license imply that:

    I grant arXiv.org a perpetual, non-exclusive license to distribute this article.
    I certify that I have the right to grant this license.
    I understand that submissions cannot be completely removed once accepted.
    I understand that arXiv.org reserves the right to reclassify or reject any submission.
https://arxiv.org/licenses/nonexclusive-distrib/1.0/license....
wrs
Uploading to arXiv gives distribution rights to arXiv, not the users of arXiv.
wrs
Shout out to an open source model that includes the dataset crawling and training code, not just the weights and inference.
tmaly
I am a little confused by the use cases.

Would someone feed the model a hand drawn scientific figure and it would output the TeX to create the figure digitally?

potamides OP
The use cases we see are (i) assisting researchers in creating scientific figures from scratch (sketching them om paper is usually easy but actually developing them can be daunting), and (ii) enabling semantic edits to existing figures stored in lower-level formats by synthesizing high-level graphics programs that generate them.
abdullahkhalids
When I am teaching online, I write on a digital board. Lots of math, figures here and there. I have attempted to convert some of my lectures into written notes in the past. Something like this would help quite a lot with the conversion.

Would be insane to have an end to end product. One that transcribes my audio, at the same time annotating it with the equations and figures I have drawn on the whiteboard.

Xerox9213
As a teacher who regularly rips off large expressions from textbooks and test generators with MathPix, this would allow me to also capture diagrams into my exercises.
dimatura
Yeah, I think that's one of the primary use cases. As a researcher I've looked at using Tikz in the past but it's a nontrivial learning curve. Something like this seems useful to at least get a starting point to tweak.
a_e_k
As someone who's used TikZ for paper figures recently, this is brilliant! I could see this saving a lot of time spent both on studying the documentation and on trial-and-error to coax TikZ to render the figures the way I want. I would absolutely use this.
taopai
Awesome! Really awesome.

I hope that German secondary physics and chemistry teacher who has an amazing free pdf book about tikz sees this.

This was such a great and didactic book to real tkiz. I can't find it right now, but must be somewhere.

abdullahkhalids
Excellent work. I see it transcribes basic text well. Do you have any plans of creating a model for handwritten math to latex math?
potamides OP
We have some rough ideas to support multiple "backends" depending on the input, so that could definitely happen. That being said, tools like LaTeX-OCR [1] already claim to (at least partially) support this.

[1] https://github.com/lukas-blecher/LaTeX-OCR

This item has no comments currently.

Keyboard Shortcuts

Story Lists

j
Next story
k
Previous story
Shift+j
Last story
Shift+k
First story
o Enter
Go to story URL
c
Go to comments
u
Go to author

Navigation

Shift+t
Go to top stories
Shift+n
Go to new stories
Shift+b
Go to best stories
Shift+a
Go to Ask HN
Shift+s
Go to Show HN

Miscellaneous

?
Show this modal