I'm sorry, but not having a mathematical formulation makes it extremely tedious to look into the protocol and get a feeling if it is secure or not (or if what you are doing makes even sense).
If you plan to do some additional work, focus on clearly defining what you are doing for a (maths-)cryptography audience. This would definitely help!
But, I can try and give you my 2-cents after skimming the code:
as far as I understand, the primitive is intrinsically sequential and encrypts chunk by chunk.
Depending on the "type", you either use a stream-cipher or some OTP-like (with pad related to the hash of part of the message).
You have a public way to decide (from the current chunk ciphertext?) which encryption method you use for the next chunk.
Am I getting it correctly?
If this is the case, I have to admit that the OTP-like part looks weird and definitely would be the first place where to look into.
Especially how the secret key is effectively expanded for the different "rounds", and if there might be some weird property for when the encryption scheme selects twice the same path.
I'm sorry, but not having a mathematical formulation makes it extremely tedious to look into the protocol and get a feeling if it is secure or not (or if what you are doing makes even sense). If you plan to do some additional work, focus on clearly defining what you are doing for a (maths-)cryptography audience. This would definitely help!
But, I can try and give you my 2-cents after skimming the code: as far as I understand, the primitive is intrinsically sequential and encrypts chunk by chunk. Depending on the "type", you either use a stream-cipher or some OTP-like (with pad related to the hash of part of the message). You have a public way to decide (from the current chunk ciphertext?) which encryption method you use for the next chunk. Am I getting it correctly?
If this is the case, I have to admit that the OTP-like part looks weird and definitely would be the first place where to look into. Especially how the secret key is effectively expanded for the different "rounds", and if there might be some weird property for when the encryption scheme selects twice the same path.