shuzchen parent
I would much prefer a library such as this be done in C/C++ so it could be packaged up as a library that could be called from other languages. Pretty much every major language has some form of FFI to call out to C/C++ code. This way, you can get consistent behavior if you need to parse robots.txt in python vs ruby vs java vs etc.
I know it's a meme to say "C is not C++" but in this context C is really not C++. Calling into C through FFI is significantly easier than calling into C++. Very few languages have decent FFI with C++, while many have great support for C.